Yep, this will happen some time and there's nothing that can be done about it since it is hardcoded rebellion programming. If you think a town is just going to go back to a faction, you can just gift it to them when you leave. This may spawn an army, but it is usually not as strong as a rebellion army spawn.