It drops by a percentage towards Despicable (more, if you are trustworthy, less if already untrustworthy) each turn for being at war for each faction. So if you are at war with a lot of people, your reputation tends to drop very quickly over time.

Alliances raise reputation over time btw.