The size of units doesn't change that much to performance. At least if you turn shadows off.

On a second note, don't play the smallest size as it will prevent your infantry to form squares.