In addition to everything said above, make sure the cavalry isn't blocked or has to change directions during the charge (e.g. an allied infantry unit or trees get in your way during the charge, or you have to turn a corner in a city battle). This will really weaken the charge.