If you occupy the settlement and make sure to get your troops out of the province/disband them before giving it back you should be okay....