You're right, this argument can be used to defend organized crime, regardless if a majority of people desire power over others. There only need to be a small number of people who are unchecked in their desire for power over others, and who will stop at nothing to get it. There are some important differences. Organized crime either ends up ruling the state literally, or is kept in check by a police force. There is no police force on the world scale (especially at that time), and thus vast Empires are totally unchecked. Regardless, I think we return to human nature. You're probably familiar with this quote: "Stop quoting law, we carry swords."
Furthermore, even if an Empire does impoverish it's own people, they are likely better off than if another Empire was subjugating them. That is an important point.
So the logic is this: Aggressive states that conquer others and form an empire are often more powerful than independent states, and thus assimilate or conquer independent states. The people of conquered states are generally treated worse than the people of victorious states. Thus it is best to the conqueror.
Bookmarks