As a boxer and a full-contact martial artist, I prefer fighting out of my weight class. And believe me for every disadvantage a light-weight has against a super-heavy weight the super heavy weight has just as many disadvantages (which are ripe for the light weight to take advantage of).

How often do you see Super-Heavy Weight bouts go past the 5th round? Often. how many bouts between ultra-lights do you see that go past the 5th round? I haven't seen many.

In fencing, I was at a severe disadvantage fighting my friend Rachel who was 5 foot 2 and a hundred pounds even, bear in mind that I am 6'2" and weigh 180 pounds. In sharp contrast, my other sparring partner Billy is 6'5" and weighs 290, and because of that size he has many disadvantages. Granted these disadvantages can be trained down, but they still exist.

I was 180 pounds and loved fighting Wookies (people bigger then me). I loved fighting southpaws, you rarely see them and are a challenge because of that.
And I absolutely loved to fight martial artists who weren't of my styles (except for TKD fighters, TKD suck and any practicioner should be shot on sight). I got absolutely sick of fighting against my same school...