It would only double if you built a certain building that promotes your faction's culture, its the risk you have to take when conquering new settlements.