When you create a rally point, troops recruited at that city (or ships at that port) will start outside the city and move to the rally point on their first move(s). I don't generally use them, but they are handy if you are using a city to supply a constant stream of reinforcements for a front-line war. That way you don't forget to go back to that city and order the next batch of reinforcements to move to the battle area.