I dont see anything wrong with your style, but you have to spend the first 20 odd turns fighting with militia. Your castles simply cant raise population that fast.

I have 3 castles, cavalry, infantry and ranged. In each one i make farms first, then the unit producing structure then armoury. Then whatever until i can upgrade it to the next tech and repeat. I am presuming thats where your problem is coming from, the low growth in your castles?