It seems that they are the first to attack me rather than neutral factions!

Why on earth is that?