As far as I'm aware the size of the invading army makes no difference. I think the whole thing about destroying structures is entirely random.