Not necessarily the hardest, but the most challenging: the Turks.

After all the initial rebel settlements are hoovered up, you have borders with Byzantium, Russia and Egypt, all of whom are powerful. You have excellent and interesting units to fight with, and varied terrain (from steppe to mediterraenean to desert). Plus, being muslim means you are likely to have crusades declared against you, meaning a lot of catholic bashing. Then come the Mongols and the Timurids, at which point your empire will be tested to breaking point, as your troops will most likely be on your frontiers rather than at the heart where the Mongols invade.