Yeah, if you want to remove the Peasants as troops you face from enemies, then remove them as a trainable unit in castles.

That should do it. Meanwhile you will still face Peasants among the rebels, which I think is fairly fitting.