AI naval ineptitude

[Ars Moriendi]

I found the AI being more competent at sea than on land - they have a chance to actually kick my ass when fighting against approximately same-power fleets. But, that could be just me not being very good at it.
In my experience, in a situation like the one you described, a smaller&slower fleet would be quickly surrounded and rout/surrender without much of a fight - my sailors are quite a cowardly bunch.

[Zarky]

AI sure does some bad decisions in naval battles, but as said earlier: with equal fleets they are a threat. I often solve this problem by concentrating all fire on their weakest/closest ship, rout/capture/make it surrender and then move on to the next one.
Often my plan involves having a Galley that sails at the biggest enemy ships max firing range, they can't hit it and it effectively nullifies that ship if you sail smartly.