Luckily, I had a savegame where Im in a position to see what happens.

I abandoned the crusade and noticed that the little cross icon in the units cards now had a red bar across it. There was no immediate loss in relations to the Pope. However, over the following turns my men still deserted.
Now I never made it to the end of the Crusade but I imagine that you would suffer a loss of relations with the Pope at it's end just like you do when you never bothered joining at all. Perhaps not as severe. Someone else will need to test that.