I haven't had this and I only play on normal. On the defense the enemy only come and get me if I'm hilariously outnumbered AND their army has a half decent core of real units instead of militia and citizenry. They refuse to give battle otherwise and will simply wait me out, a tactic rather more lethal for me since I play without a battle timer; the game will literally wait until I, personally, die.

Six units of 24-pounder howitzers though, that wakes them up. Doesn't leave them much choice but to attack.....