Red Harvest's post implies it, but maybe it is worth saying explicitly - I think 100 men is a magic number. As I understand it, have a garrison less than that and you risk a rebellion, regardless of the loyalty %.