I'd recommend you have a look through the bug thread, you may be experiencing one of those problems. I seem to recall there's one that has something to do with ladders that can horribly bog the game down during a siege, and likewise some buildings in settlements apparently can cause massive slowdown (I don't know if anyone figured out exactly which they are, I've just noticed it is a sometimes-present thing). For that at least you can try conducting the sieges viewing from inside the city outward, it seems to alleviate the problem. Also I'm not sure if patch 1.1 resolved any of these, but if I did I haven't heard that it was the case.

For general performance, I've found disabling the shadows to have a huge impact, with minimal actual graphical impact: i.e. it gains a lot of performance with very little loss of eye candy. In fact I think it's a bit silly that shadows default on no matter what graphic level, as they put a very noticeable drain on things without nearly as much to show as bumping up some other graphical settings would. I've also heard it suggested that you can disable grass to pick up some more performance, but I haven't tried that particular one myself.