Damaged Agent/Military Buildings

If a gambling hall or other agent's recruiting center is damaged by an army or sabotaged, will the agent be affected in some way?

I know that I would be unable to recruit a new one, and if the province is lost I lose the agent. But is there any adverse affect to a damaged agent building on any existing agents?

For instance, the experience bonus for ninja recruited at infamous district.

If an archery building is damaged, are all the units from their no longer benefiting from reload?

Logic would say there is no affect because they were already recruited, but this is a game after all.