The strangest part of it was that his port raids didn't count on the screen that shows who's at war with whom

I don't check that screen every turn so several turns later when I decided to return the "favor" I got the message about 'do you wish to attack a neutral faction'? Hmmmm....I thought we were already at war.........

...........well if we weren't, we are now