My system (which seems to work) is much simpler.
I just form my fleet into a double line ahead (en exchequer) with ships in the second line covering the intervals in the first.
I then position this double line to windward (up wind) and angle ready to run down on the enemy fleet (with the wind behind them).
When the battle opens I head for the lead ship of the enemy fleet, and try to cross ahead of it.
If I don't manage that then I basically manoeuvre my fleet so that my lead ship constantly has their lead ship in its target zone. Everything else I leave to the AI and 'fire at will'.
I only use 'dismantling shot' and 'grape' if I heavily outnumber the enemy and plan to board the enemy and go for some prize money. In any sort of challenging battle the only thing worth firing at is their hull.
[I second Lobo's comment - in naval battles weight is more important than numbers. You idea of swamping a 5th rate with brigs is a long shot and very expensive to boot. The first thing I would do is get a militia cavalry unit into that port and force the enemy to put back to sea. If you are lucky someone else will sink them, if not you need to research 5th rates before trying to take them on, unless you have fluyts or galleons.]
Bookmarks