Neutral armies don't cause devastation, and devastation does recover. It's just very slow about it, and the bigger the area the longer it takes. Look at the city details. Under income, it'll show "Devastation" in the loss column, and if you hover your mouse pointer over the icons, a tooltip will tell you how much income you lost this turn due to devastation. It drops slowly.