The AI in RTW is a victim of the graphics IMHO. Graphics sell more, impress more if your showing a game to others.

Remember that RTW was the very first of the TW series in 3D - an entirely new engine and it had been used on TV to reinact historical battles in "Time Commanders". With graphical expectations so high, CA had to deliver -and they did, but at the cost of AI and some features that previous TW's had.

I would be more tempted to ask why the AI in MTW2 wasn't significantly better, since it used the modified RTW engine and presumably CA had a little more time for AI and other coding.