I find using archers can sometimes get them to move, set your archers on the bridge and fire at them withdrawing as soon as the enemy comes at you. Sometimes you have to run back and forward a lot but it usually works in the end.