I think the problem is tied to either the founding faction (the town doesn't know who to revert to), or the town can't train any units to garrison the newly rebel town. Probably the later. Either way I don't think it matters who you are or who you took if from. I believe that this same problem is tied to a CTD when a AI faction that holds one of these towns loses all its family members. (I failed to test this theory fully.)