It's their experience rank. The little stars next to their portrait. So, if a 7-star guy sees a 5-star guy get appointed into an office, he gets cranky.

The problem is, often these high-experience/rank guys, especially if they're army generals and haven't fought for a while, don't have enough influence for an office. The solution would be to swap characters regularly back and forth between field command and governorships, so that they're continuously earning influence and always have enough for an office every 8 turns.

The good news is that the loyalty hit isn't that big, and can be countered politically by one of the wives.