The starting period is the hardest part, depending on your faction. Once you have established decent border armies, a couple of fleets and have built a working economic infrastructure about the only thing that can really threaten you is a Civil War, or the Golden Horde invasion, if you´re playing with one of the eastern factions.
I suppose if you´re playing as a Muslim faction Crusades directed at you can provide a challenge, too, as they come with a lot and quite high-quality troops.