There are some gameplay differences. Having larger units not only means that training each unit is a bigger drain on a city's population, which might have adverse effects on city growth, but also in the way they maneuvre on the battle map. Larger units seem to respond more slowly to orders, simply because a unit of 80 men doesn't have the tactical mobility as a unit of 40. That bigger unit is going to swing more slowly on the march, in battle, when flanking, etc.