Erm other than simply not building them or demolishing them before sallying, there is no way to control them. No on/off switch that is.
Unless you want to cheat and give/remove money each time to build/demolish them and use process_cq to give them to a settlement.