Yeah, the Empire may have a lot of territory but it has a lot of enemies.

To name a few :

Orc Waaagh's attack the Empire frequently

Chaos Warhosts invade the Empire once in a while

Chaos warbands roam across the empire's borders

The forrests are being inhabited by herds of Beastmen

The Empire is also attacked from within, it has several chaos cults

There are Skaven living under cities of the Empire that can attack the Empire whenever they want