I've been doing that and having all non-troop building provinces eventually just have a market, ninja and temple line. I think I'll eventually drop the ninja line past 5, since the food for the castle upgrade is better than what it gives.

The one that's hard to give up is the temple line. I find arts research to be too slow without using as many of these as possible.