Diplomatic relations seem to have an impact. Early in the game, I often see small Kings go from 4 to 5 influence without expansion. I don't know the numbers on this, but it seems to need quite a bit of talking with others to gain that extra point.

Also the ingame assassination messages seems to imply that getting your emissaries/agents killed will lower your influence. My own testing seems to support this, but again, it's small scale.

Sorry if this is a bit vague, but we are dealing with something that's hard to accurately meassure.