It happens as well with town gates. When the AI is attacking your town and destroys the gate, it's like the whole army tries to get inside at once. Last time i was waiting for them near the entrance with spearmen and archers and i had 1 FPS for at least one minute. Very annoying indeed.