It was popularized from a little bit before that (say, early 1800s), and derives from seeing elephants in travelling circuses, when it just meant "seeing something exotic." You are right, though, that it took on the more common usage of "been in battle" (or, more generally, "a loss of innocence") in the American Civil War.
I wouldn't be surprised if it's even older than that though, at least in some form.
Bookmarks