2 things happen in retraining. . .
1) Re-building unit strength (replenishing)
2) Adding armory (blacksmith, etc) bonuses.

You can only replenish if you can recruit the troops in that town.
Adding blacksmith & experience bonuses can be done to any troops that don't already have that bonus (even mercs).