Crusades gain men from units with high piety that come across and peasants from high zeal provinces. On the other hand every unit in a crusade looses a few men each turn unless when in enemy territory. So it gets bigger if there are great stacks in the provinces it passes by but can loose many men when crossing low zeal or non-catholic provinces. The lower the zeal, the more men will desert, but dropping from 1000 to 170 in two turns without a battle seems odd to me.

Also it seems not to get bigger than about 2000 men unless you actively add units.

I may be wrong about those things but thats my observations so far.