You shouldn't worry too much. Visimar already have less Influence than his heir so there is not much to lose!