Limits on the number of agents

What determines how many assassins, diplomats, merchants, priests (imams), and spies you can support? How is this calculated?

I believe it is the number of buildings in your empire capable of providing each agent type. Someone correct me if I am wrong.

So for example, if you want to be able to recruit more priests, build more churches.

Some what. Religious agents are 1 per religious building in your settlments. Merchants are 1 per city. I've never goten to a limit on assassins/spies/diplomats yet.