Many crusaders stuck around and many went back home. Those that stayed started life anew in the Holy Lands. Some went onto take more land from the muslims. Some started trade and other businesses. Like mentioned above if you want your crusading armies to reflect them leaving just disband them. For the AI crusading armies not much you can do about those stacks unfortunately. I've seen them stay and I've seen them go in my experiences.

I do think your idea about the pilgrims and fanatics a good one. After a crusade concludes they should auto disband from a stack. The rest should be allowed to stay if the player wants to hire them out like mercenaries.