Well, it is quite easy: Always 2x as much as before:

Hastatii e.g.:
Small - 20 men
Normal - 40 men
Large - 80 men
Huge - 160 men

Please take in mind that especially HUGE unit size settings of video options influence your campaign. Your upkeep e.g. will be much higher.

I personally agree with another poster that LARGE is usually a fine choice, and saving a bit performance, too.