For me, you have to leave them with one settlement and they'll probably counter offer you by asking large amounts of money.and they had no army but a garrison.