Yep, you've an infestation of enemy spies holding the door open for their compatriots. Nasty, init?

Counter by leaving at one spy in your cities/castles purely for counter-espionage purposes. If you've got your guilds and generals' trait boosting your spies' skills, then you'll find each turn that you're capturing enemy infiltrators and that your intelligence forces are steadily acquiring retinues like beguiling bard, and exotic dancer.

Also, remember that the AI will find itself in your position when your spies do the same, and when you get the gates are open signal, rush your army in before they can effectively deploy.

What a game!