I suppose CA made them 'undestroyable' so it would be harder for a player to sack / abandon / sack / abandon / ... .

But yeah, you should be able to destroy walls...