the biggest thing you should be afraid of is if the Pope calls a crusade on your capital, you have more than 3 enemies, those 3 enemies and more will join, and if they actually do come (with full stack armies) its quite dangerous. economy will be very hard to get back up if they capture you capital- unless you have a lot of settlements and have another city worthy of being your capital.
didnt happen to me, but thats what i predict.