Siege battle eh? Quite a few people have the same problem (including me)

From own experience, the gateway towers, normal towers (all stone that is), settlement buildings that use cobble tiles (city type) cause the slowdown.
For me, it happens in:
Any city type (minor, large, huge)
Southern and eastern castle (stone castle, fortress)
All citadels

Siege towers also tend to bog performance quite a bit for some reason. I'm guessing for all of the above, its the level of detail of the tiles which is causing the problems. CA should do something about letting the player choose what performance tiles should be used (for example low - no cobbles visible, plain walls; medium - some larger sized pieces; high - as is at the moment)