What unit size are you playing at? It is probably because you lost a bunch of experienced bodyguards.