I actually like the challenge a good band of brigands can offer sometimes. It's good for your generals since they will get some battle experience, so I keep it at the standard value. If the brigands appear, I will fight them.