I suppose it may be possible to exterminate the hell out of it when you cap the place, trying to blast it back into being a town. You can't destroy the wall levels, but if extermination can set the walls back a level or more then maybe this would work out for you. I don't really know, I exterminate pretty infrequently.

If it can affect the city walls, you can of course allow the city to rebel and then retake it and exterminate, as many times as is necessary to make it the right level to flip to a castle.