I agree that sufferage should be more restricted than it is, and that prisoners might not be the most relevant group, BUT they are the easiest - the ones in jail don't vote until they get out. As you point out, age is arbitrary, and one could make as good an argument against the under 18's as one can against the over 90's.

Other ways of discriminating are... discriminating, which in itself is Evil.
Then there's the methodology of deciding who can vote. That has the potential to blossom into a massive industry. whist possibly being no more fair.