Is it possible to give building bonuses only to certain factions and not to any faction occupying the city?

For instance: there's a building called Torture Chamber that would give Law bonus to faction A, but if faction B occupies the city, they get no bonus from it. Or even, faction B would get a penalty instead of a bonus (aside from the obvious culture penalty).