If you get a Riot event you will not lose the settlement...

You have to get a Civil Revolt event to actually get a rebel (or other faction) army appear and kick your troops out...