Personally I never do, except to get rid of an ancillary if I want a character to get a governorship or satrapy and they already have eight. I have never had, or heard of, any problems related to swapping retinue.