No, cavalry will not be harmed by charging over stakes "the wrong way". I think it will disrupt their charge somewhat, however. In addition, they can even cross stakes from the front so long as they slow to a walk to do so.