A matter of a horse and man (Presumably with a deal of armour as well) weighing more than just a man. If a horse could charge into a solid formation of lightly equipped soldiers, it would still wreak havoc because of all the force involved. The reason they don't is because they don't want to charge into something that looks dangerous.