You dont see bandits the way I do. They are excellent for training your generals. They gain superior commander like nothing, and it adds some mix in the game. Sometimes when im playing a big faction I house a few cohorts of troops in a fort along the road. This stops (or seriosly slows down) invasion and when the bandits do come up, you can take them out (and sometimes earn a extra bonus- a new heir).

They shouldnt be too much of a problem, maybe you just hate micro-management (I love it).