Well, not all knights were rich. Quite the contrary really. Most noble families would tend to spawn more than one son, and as estates were normally inherited by the primogeniture principle (ie. the firstborn son pretty much gets everything) the younger brothers by and large either had to take the holy orders or take their weapons and armour and go sell their military skills to the highest bidder (or, as also happened, their eldest brother - no reason why they couldn't become *his* household knights, as such) to support themselves.

AFAIK many sergeants could well be quite markedly better off financially than most landless wandering knights were; after all, the sergeants could own land too in the feudal pyramid and could similarly be employed as parts of some lord's personal military followers.

As a side note, an incredible amount of modern military terminology and rank titles (at least in English) can trace their ancestry directly back to the Middle Ages. "Colonel" was apparently originally a Late Medieval Spanish term for what IIRC seemed to be a big-time condottiere, for example.