I used to have exactly this problem, fortunately my palsey has deteriated, so now my limp is more pronounced.

That seems to have fixed it.

Hmmm.... Now I think of it, though, it also stopped around the time I became a serious Christian, which is also when people started routinely asking me for money in the street.

So, perhaps you just exude angst, rather than anything malevolent.