Destroying them will keep you on the safe side.
If you want to roleplay that there was just a short take-over, you can use process_cq SETTLEMENT to speed up your building in this specific town.
For other CTD's, read the frequent issues.


XSamatan