The best way to take down elephants is to spook them. Once they're routing, send a cavalry or other fast unit after them, and they'll drop like flies or any other routing unit. Killing them when they aren't routing usually takes a Phalanx-type unit from my experience. The only two elephants I've lost on my watch were due to pathfinding problems going through a gate on a city and a Phalanx nearby.

Another oddity is that Elephants seem useless vs routing units. I had a unit of Velites being attacked by some Elephants, so I put them on "withdraw". Despite them being continuously knocked down, they always got back up. After a while they got some distance, so I told them to stop and hurl some javelins (who knows, maybe one of the beasts might fall). As soon as they stopped running, the elephants made their "charge" horn sound, and my guys started dying again. I put them back on "withdraw" and they stopped dying. During all this, my unit was interspersed with the Elephants.

That same unit of elephants was surrounded by my units and spooked. Suddenly the indestructable tanks were dropping like flies as my General's unit chases after them. Strange behavior IMO.