Yeah it's cause the garrison is reinforcing. If they wanted to attack you near the city, they would sally out, which they do on occasion depending on how strong they are.