Yeah, that's what I ended up doing. I totally forgot to repost here :)

Basically used agent limit 0 at town hall level and 1 at city chambers and above.