I resisted when this first came up, but I've ultimately added cavalry to my smithing towns. I think Econ21s guide on city development suggests doing this.

The reality is that I just don't use that many cavalry in my armies regardless. Rothe's point is good about possibly overloading your smithing town's recruitment demands, but I personally only carry around two or three cavalry per army. If a dedicated cavalry town is only producing cavalry six or eight turns out of thirty, I don't feel like that's a good use of town resources or a good reason to reduce my global food supply which ultimately is a significant hit to my Emipre's finances.