From my experience, enemy armies staying in your territory adds to the devastation penalty. Also, having a big battle in your territory also adds to it. That penalty decreases over time, however.