OK, I've obviously caused some confusion, and it seems to be my fault. I am talking about those extreme, ideologically-driven versions of libertarianism, where the private and public sphere must be kept seperate. The reasoning behind this being that some individuals private beliefs may offend others when aired in the public sphere, and since this is form of harming others, this cannot be tolerated in a 'pure' libertarian system.

For an example, look at the recent case with the bakini burqas. People said they were offensive (regardless of the private beliefs of those wearing them), and so they wanted them banned from the public sphere, because they were supposedly incompatible with libertarian western society. See how immediately we have created a sliding scale where some private beliefs are deemed unsuitable to enter the public sphere? Now, how far do these libertarians want to take this? If they are sensible, they may stop at hate speech, or anything which could cause physical violence. If they take it further, they might want to tell Islamic women what is suitable dress in a libertarian society where everyone can think for themselves (lol, irony). And now we are on this scale, there's really no barrier to taking it further. What if Richard Dawkins is offended by my preaching on the street? Then my private beliefs have come to harm another in the public sphere, and as such should arguably be banned.

See how a libertarian system can very quickly become incredibly repressive when taken to its natural, ideological extremes?

And when people do not take it to these extremes, then they lose the moral backbone of their argument. They are no longer a clear ideology compatible with all others, instead they become like the rest of us putting a limit on private freedoms when they become a threat to the public sphere.