the number of soldiers is based on the number of riders.

If you put 12 invisible riders per mount, and make the unit size 6, then even on large unit sizes the 1 mount will be 1 unit I believe.