I think having ships "disengage" off of the map edge, just like land battles, would make most sense, as long as the map was big enough. If the fleeing ships were faster than their pursuers, they should be able to run away.