Meh. The building has a great, logical layout. Four L-shaped ones, connected at the centre, four semi-courtyards. For a building of it's kind it looks rather pleasant to me.
You can never see any swastika resemblance from ground level either. If the navy feels compelled to make a statement, I'd rather they spend their $682,000 on an anti-racist social program or something.