Well, how do you explain then that facing away from the settlement decreases that "lag" dramatically? It kind of does sound graphics related.

What is processor related is the number of units alive (and to a small extent dead) on the battlefield. You can face away and not get any performance improvement for example.