I usually destroy them.
1) The worst thing they do is block trade on roads, even though you don't see it, they could be causing a LOT of lost trade. This is sort of the historical aspect of them, they are basically robber bandits. I am not sure, but having rebels running around might cause your governors to get "lawless" traits.
2) They are good for building experience of generals/troops. Don't attack 3 rebels with a full stack though, try to use even odds so your general can get some good traits.
3) They tend to make it difficult to move around small groups of units, for example lets say you have 1 Fortress making Superknights, and you desperately need 3 of them at the front line, if rebels are blocking your path, the knights either have to take the long way around and arrive several turns late; or fight them and take losses, arriving at the front understrength; or fight them, go ALL the way back to the Fortress, repair, and head out again 3 turns later. I would rather fight them when I have the time and units free.