Well, Im sure some people can post more ezpert replies than me, but my 2 cents:

1. Wait for them to stop moving before asking them to change formation. Sometimes it looks like theyve stopped but some guys are still wriggling into the right lines. So at that time, press the backspace key to make them STOP and then after a couple of seconds ask them to reform.
2. Its better not to actually ask them to attack something directly while in phalanx mode. Just make them form a line about half a pike length away from what you want them to kill and they will start stabbing it soon enough. Some people suggest that you actually get them to form up and then order them to "move" to a point on the other side of the unit you want them to attack. Personally Ive not found this effective on hard difficulty level, but other people may have different experiences.
3. My main tactic with pahalnxes on defence is just to stand still at the place you want to fight and let the enemy jump on the pikes. I will often change the angle of facing to help this process, but rarely take them out of formation except when I want them to charge someone in the rear with their secondary weapon :)
4. On offence I run them in a line hard into the faces of the enemy. Stop about a javelin thow away, press backspace and then form a phalanx wall, now the enemy is so close they cannot try and run around the phalanx and have to fight you at least partially head on. Yes I know this double timing isnt historically accurate but the AI doesnt seem to respect battle lines once it gets within charging range, plus all that milling around just annoys me where you try and target a unit to move against and it promtly turns right and heads to the end of the line...
5. I only ever target a unit directly if charging with the secondary weapons, otherwise I always initiate combat by drawing a line with the troops at an approppriate distance. In combat if the enemy is attacking me from an angle, I will change the facing of the phalanx to stop that (or if Im fighting another phalanx to give me the advantage...).

That's all I can think of :) You just need to be patient and do everything twice as slowly as you would expect to.