Yes, you can still conquer that province. Just do not tear down that building.