I tend to agree with you. I'm not personally wired for polygamy, but it's really none of my business if other people are. So long as everyone involved consents. There's a lot of room for abuse in this type of situation (I'm thinking about women from poor countries who marry men to move away to a better place).
Another thing that gives me pause is that groups of people could use a legal framework to get tax benefits, etc. Imagine a bunch of grad students living together and decide to get 'married' as a group to get tax benefits, etc. This is abuse of the law to me, and I don't know how the gov't would prevent such a thing from happening.
Bookmarks