You select the target by clicking on the enemy ship. But if you want the formation to make it a priority target you alt click.

The other way of capturing a ship is forcing it to strike its colors. That is best done by shooting its sails and reducing its crew.