A couple of thoughts.

First, fire. Fire was just as dangerous to the user as to the usee back in the olden days. Several books I've read that cover Greek Fire mention how volatile it was, and how it would often just as well incinerate the attackers due to some kind of mishap or malfunction. Even lesser oil-based items wouldn't have been as feasible. First, having some kind of device to use and throw would have been additional weight to carry to the battlefield. Second you'd have to get uncomfortably close to toss it. Third you'd have to hope like hell the container would break, the most likely device would have been some kind of clay pot. You'd have to do it thin, find some way to light it, then get it the hell out to the front lines to toss before it'd go up in your hands. Not really worth the trouble, in terms of weight, usability, and potential danger to the user, so I can see why this never became popular.

Second, missile-based attacks. This right here is your best bet if forced to take them head on, and it's still not a guarantee. One of the main benefits of the phalanx-type massed spear formation was that it provided excellent defense against arrows and javelin type attacks due to the large number of spears and how they were arrayed. Of course it wasn't perfect, but it was much better when using the sarissa than say your average warspear 6' in length. The non-sarissa non-phalanx formations would generally have been more vulnerable, however this is also dependant on the amount and type of armor worn by the individuals.

Third, non-smooth terrain. Pick your battles, and pick where you have them. Case in point, the well known Battle of Pydna. Phalanxes depended on cohesion for strength, the uneven terrain took away this advantage and the close combat sword and shield proved more versatile.

Fourth, flanking. Best way by far. Get along side or behind 'em. The spear-based phalanx was always strongest to the front, and weak on the sides and back (except in RTW, where you have the butt-spike bug).