I have had massive armies of gauls walk right by forts and not attack them.

Meanwhile, the peasant/brigand/slave armies (that the forts are there to try to suppress) are quite happy to attack a fort with one town guard in it.