Ulstan in the entrance hall says:

"Also, I have read somewhere that switching heirs gives you a hit to your influence. Does anyone know how much influence this costs?"

When I switched my heir from one brother to another, the first brother got the 'disinherited' vice, which said he used to be the faction heir but was passed over, etc, and he lost one influence. Not much of a big deal really.