There is a way to do this (although of course a diplomatic option would be more handy) : park a military stack in the "red zone" around his troublesome stack (or a fleet next to his fleet). This will send a "I'm watching you, you little git !" message to the AI and will make it reconsider it's invasion plans.

Then it'll either retreat, or declare war, depending on both your overall strengths+production possibilities compared to his, and said stack (a single militia unit will usually entice him to attack outright if you are both equally powerful, while a huge army will send him back wimpering to try and invade someone/somewhere else).

Either way, your reputation is safe.