You don't have to merge stacks to transfer troops, it can be done between adjacent stacks.

To free your ships, do the following:

- First transfer off all agents and troops, either offloading them onto land or to a neighboring fleet
- Create a rally point at the nearest convenient port & drag it to a sea square immediately adjacent to the stuck fleet
- Drag the stuck fleet to the rally point... your ships are now free