I've noticed this too - you should have the option to neutral factions....