As I recall from Centennial by James Michener, when settlers were moving across the American West, there was a eventually a point where some would turn back. Those who did were described as "having seen the elephant."

According to the "tradition," those who truly lacked the heart to go west would suddenly and randomly see the "image" of an elephant - in a rock formation, or in the clouds - where no one else did and then decide that is was an omen, that they should go back.

It certainly seems possible to me that the expression "seen the elephant" marked the time when some decided to go forward and some decided to go back, as seems to be the case here (he "saw the elephant" but kept on going), and in the American West it was misinterpreted as only applying to those who turned back.