No, I always got the full effect of the governor, even though they were elsewhere. I even noticed that when I've stacked my governors with their ruler in one province, some of them still got VnV's, even though they were not in their own province or in command of the stack. BTW, dread only works until the balance of the population is of your religion, then the piety of the governor takes over.
I have partially filled units anyway, so I just restack them to my advantage. My administrators don't fight and my fighters don't sit around doing nothing. This method keeps great governor's safe so that they don't get assassinated, burned at the stake or lost in combat, because they are stacked with their ruler in one safe place.