I've read it was a popular good-luck charm among soldiers around the end of WW1 actually. But in any case we were using it in official functions long before those poseur Nazis even existed...
(I've heard they only changed the Order of the White Rose's Grand Cross with Chains from the original pattern after a certain Charles de Gaulle had politely turned down the thing during a state visit... something about it not being appropriate for him to wear something with swastikas.)