yeah, i think it would be nice to implement that you can never take a town without a fight.

you can
-siege
-assault
-ask them to surrender

this way if there are no trained troops around, the villagers will defend and a peasant unit will spawn, and more if the city is bigger. after the siege/assault they'll leeve again
but they will not contribute to the happiness in the city, cuz they are the city