I usually never bother with walls as the enemy never reaches my towns. They are intercepted as soon as they cross my border or try to siege one of the many passage blocking forts.