Yep, it is good to keep a priest (or other appropriate religious figure) in each settlement as it will keep heresy in check and stops cluttering up the map...

I only dispatch priests for special task such as accompanying crusades and attacking heretics...