As far as I recall - but might, as usual, be completely wrong - the difficulty settings also have an impact on things like tax income and unrest.