This happens to me as well but only with heavy cavalry and even then quite a lot still die. What cavalry was it and what did they charge into to?