When the CPU controls a land it gets population growth bonus like no other. When your castles are struggling with "castle", the CPU may have 9000 pop by then. For castles that are in low-pop areas like innsbruck where its locatoin is perfect but has low pop, leave it to the CPU.