You already get extra valour with these buildings. These small incomes are nothing compared to the agricultural and trade incomes, therefore, removing them would have no impact.