I haven't noticed it happening, but if you're stuck in a place with hostle natives everywhere you would start to get a bit annoyed at the people who put you there! I don't put governers away from my capitol much for this reason, except if it's an important city (enemy capitol etc.).