Okay,
so on foot or by foot
but never feet in any combination?

It sounds reasonable.
Now it would be nice to have someone explain it.
Think of it as an idiomatic expression - it is 'on foot' (a la a pies because that is the expression, like 'Get Bent' would never become 'Get bended' or 'Become Bent'

ichi