It takes a stack of 8 units to start a crusade (at least 1 general and then 7 other units, general or other). If you remove a normal unit from a crusade it will desert. I've never seen the AI remove normal units from a crusade. However, if you remove a general from a crusade (i.e. you have 2 or more generals in a crusade army and remove one of them) then you end up with a 1 unit crusade army composed of the general. Generals will not desert and you can add units to his crusade army.