Do I really need to go after rebel armies in my territories if their just sitting there not doing much? What's the downside?

Thanks in advance