I think most of the time it was bisexuality, with someone having both male and female lovers. Certaint Ancient Greeks practiced this, for example.

But I think most of the time it was of the nobility or royalty, as you said...