One thing you can do is to go into the text file for brigands and just set it so that brigand armies have no units in them (or very small numbers).