It all seems blatantly simple to us, but I would bet the developers are more hamstrung by limitations of technology than we would like to think. There are also the time and cost considerations to be dealt with and just how much of each can be put into any *problem*.
we're not asking for a near-human AI, just an AI that doesn't make completly obvious stupid mistakes. For example in RTW the AI doesn't even use their missile units properly. Give them 19 archers and 1 cavalry unit and they will still charge in that cavalry unit as soon as they can, instead of waiting until the archers have depleted their ammo.
Writing AI is hardly limited by technonogy, it's just a matter of investing enough time figuring out the AI algoritms and putting them in the game. AoE3 had a single developer solely working on water effects. I garantee you TW's AI could be more then acceptable if CA would let a single programmer work on AI and nothing else.