Two units of bowmen are enough to burn a tower down. Luck is involved. Sometimes it won`t even reach the walls, sometimes it will collapse when the first enemys have already reached the wall.
Don`t worry though, they will start attacking when all men have climbed up the ladder - can take a long time with huge unit sizes.
If the tower collapeses, all men that have climbedt to the top level will be placed on the wall, all others are dead right away.