Perhaps make it only possible for units where there is evidence of the conquerors using units of the conquered.

Alexander filling his army with Persians comes to mind.