This a really annoying "quirk". I think the least they could do was feature the settlement on the battle map so you can leave them behind the walls. They show the settlement when you are sallying out with the garrison so I think it could have been done.