Maybe it's because there's less room to manuever it seems like units move faster?

You should also remember larger unit sizes place a heavier drain on population.