This is normal - its a game mechanic that simulates the fact that crusades that took too long to reach their destinations or passed from areas of low catholic zeal faced desertion.

The best thing to do is to prepare armies of your own and insert them manually in the Crusade. These armies do not desert as fast as armies that the crusade picks up. They stay in for ages and you get the benefit of not paying upkeep for them.

Noir