It has happend to me though mostly in the viking era and when i had just 2 peasant units..seems like it happens only when you dont got alot of troops in the castle.