No, you didn't do anything wrong. There is only a very slim possibility that you will burn the Rams down, but the Siege towers are fair game.