It seems that there is really no way to avoid this in the longer run: once a governor does not build farms for a stretch of a time he gets it... Even if all the towns near-by have full farm upgrades...