Lancers are not much better than any other knights. They were much better before 1.2, when the shield bug was still present.

4 points of the knights' defence value comes from their shields, which work against attacks from the front. But since in a cavalry melee most attacks come from the side/rear, they wouldn't benefit that much from their shields. Body armour is better because it works against hits from any direction.

Furthermore, you can upgrade the knights' armour, while the lancers are already maxed out - giving knights even more of an advantage.

I don't understand why many people go for lancers. In some custom battles they did really well, but it isn't represented in the unit stats. I suppose there could be more benefits to late units and advanced armour than we can know from the unit cards.