That's all very strange. The only trait I can think of off the top of my head that causes permanent star loss is the "Lover of Beauty" one, but that's only 1 star.
It has more levels, two for guarantee but might even be more. Other traits that reduce stars are "drunkard", "bad tactican" and certainly others. Unfortuantly "lover of beauty" and "drunkard" can't be avoided because they are triggered by basic buildings like the tavern.