I once (in vanilla) had a single enemy soldier run into my square while I had all my men defending the walls. The timer ran out.
I share your pain....