How I Think the Bugs Slipped Through

[Carl]

The Answer:

1. SEGA: According to this post the Dev's weren't totally happy with the balance when it shipped anyway.

2. Auto-Calc: (IMPORTANT NOTE: The following Piece is a Theory only, I may be wrong).

This is the BIG one IMHO, and explains how the game got so unbalanced and how it was so hard to fix.

I believe that for most of the main balancing part they actually used the Auto-Calc engine to run their tests, this would be relatively simple as Auto-Calc is quick to do, can probably be made to give more detailed information than what we are actually given and is simply a cut down version of the main engine, so in theory it should perform pretty much as the full graphical engine does. However we know that isn't always true, the 2-Hander bug for example has no effect on Auto-Calc.

Whilst we don't know exactly how Auto-Calc is cut down too much, we do know for example that combat is pure calculations and animations have no effect.

Thus what I believe is happening is that in both game engines everything runs off calculations in the background with what I call an "animation override" in place in the Graphical engine. So if the animations dictate something different to what the calculations say should happen, the Animations get precedent.

e.g. With 2-handers: The calculations run and dictate X number of hits. It then instructs animations to play out to achieve this result. However because of the Bugged animations none actually happen and nothing happens, no hits are scored by the animations and this overrides the calculation meaning the next stage (determining how many hits actually kill), has zero hits to play with and can't kill anything.

Another point is that if 2 sides fight and one has faster animations they can get more kills in than the calculations might suggest, also anyone cut down in mid swing in the Graphical engine subtracts from the hits, whilst in Auto-Calc it would have no effect.

It also raises another point. Densely packed units, (i.e. situations where 2 or 3 units are one on top of the other), or situations where several units are packed back to front and side to side with each other would all not benefit from arrows hitting anyone other than the target unit as only the calculated number of arrow hits will be scored, and only against the unit that is being targeted.

That in turn raises an interesting point about enfilade fire. In auto-calc, shooting from the flank won't get you any extra hits because the number is calculated according to some accuracy value, thus any extra arrows that would fall on the unit in the Graphical engine are ignored by the Auto-Calc engine.

NOTE TO HA FANS: I'm not say enfilade fire IS over-powered in the Graphical engine, just that it's probably stronger than in the auto-calc engine. Which is the intended level of power, only CA knows.

I can't prove it as theirs no evidence, but I suspect it might be the case that whichever bit of bugged code is causing the Shield Bug is also not used in Auto-Calc, thus the shield bug slipped through.


Overall I suspect they used Auto-Calc to get the majority of the balancing done as apparently the animations COULD have an effect in RTW too, but something about the min attack delay cut these effects to almost nothing. So they probably expected the same in M2TW and didn't worry about the animations and other stuff being cut out.

Thus the only times the balancing team would have used the Graphical engine would doubtless have been right at the end of play-testing when they wanted to look for any IMBA that was happening because the player could out-think the AI. (i.e. player Exploit finding). With the 2-Hander bug being introduced so late on i don't doubt that they had very Little time in which to re-balance the 2-handers relative to the animations (they'd probably realized animations mattered now), and probably didn't have time to pick up on the problems relating to all 2-handers being bugged vs. Cav.

Likewise, we know CA wasn't happy with the balance at release, but we also know that they didn't know about the shield bug, (they sounded surprised to learn of it when we found out about it), so it's equally clear that whilst they knew their was a problem, they had no clue as to what it was.

Don't get me wrong, I'm not trying to say that CA didn't make mistakes, they did. Just that a combination of rushing them at the last minute, (when they had time to discover the animation related bugs), and coupled with them possibly using the auto-calc engine for most balance testing has probably resulted in the bugs getting through. they simply didn't know about them until too late to even figure out what was causing them.


Of course I could be wrong and they could have used the Graphical engine for testing from day one, but using the auto-calc engine makes more sense as you can run more tests in less time.

[dismal]

Well, the game was released with a unit that caused the game to crash (Elephant artillery) so it's hard to believe there was extensive playtesting.

[Carl]

@Dismal the elephant bug dosen't show up in Auto-Calc though.

That my point, by the time they where starting do do ANY testing in the Graphical engine, (the one we use when we fight the battles ourselves), SEGA was IMHO pushing for release so they never got it actually playtested in the Grapical engine, just the Auto-Calc engine where almost none of the games bugs actually show up.

[Puzz3D]

We know they use auto-run to balance the SP campaign because MikeB posted that's how he did it in RTW. That's how the RTW load/save issue slipped through, and it was extremely difficult to get them to fix it after it was discovered which is the other half of the problem. CA makes an assessment on how much work it will take to fix a problem. If it's going to take "too much" work, the problem doesn't get fixed. For instance, I know from the RTW v1.2 beta that redoing a unit's animation is "too much" work, so we couldn't get any speed adjustments made to the units because they are tied to the animations. CA said we had to wait until M2TW for speed adjustments after initially claiming there wasn't any problem with the unit speeds, and then this was turned into a selling point for M2TW when it actually started out as a playbalance issue when I mentioned it back in Aug 2005 and a bunch of veteran players agreed with me.

Another issue is the introduction of new problems in a build. It's so prevalent that patches get released which introduce new problems in the game. In fact, every patch for RTW introduced at least one new problem, and the final version of the game has a civil war bug introduced in the final patch.

The battle engine is separate from the campaign. They may use an auto-run to test that, but I know that LongJohn did the balancing manually in STW and MTW. Palamedes is doing this for M2TW, but he was facing an impossible task with shield bug. Now this clearly demonstrates that CA is willing to release their games with serious imbalance issues, and it's consistent with my impression of the company after serving on four of their beta teams. This is no doubt due to pressure to meet a deadline, but I think there is a chronic problem in that CA doesn't put enough debugging time into the development schedule of their games. I know they introduce new features too late in the schedule for those features to be debugged, and a problem like the shield bug which is probably just a sign reversal in an algorithm would be detected by the standard debugging method of inputting a set of parameters into each algorithm and checking that you get the expected numerical result.

They've used the excuse that they don't have enough time to debug the game, but I would say don't put in features if you don't have the time to make sure they are working. I'd rather have fewer features that work than more features of which only some work. Because of what I know about how CA developes their games, I won't consider purchasing anymore of their games until after the final patch is issued. Mods do me no good because I play multiplayer. The official version has to be working properly.

[JCoyote]

Actually I'm pretty sure autocalc is so streamlined things like enfilade fire simply aren't part of the equation. It doesn't simulate maneuver on the battlefield at all, it simply compares types and relative values against a random seed number. This is why castles are much easier to attack under auto; if you have the means to attack it simply throws the walls out of the equation altogether.


I have an idea though! For the future. For CA and all of us.

You see, I'm a bit miffed because I feel like I bought a beta... but more accurately, that I bought a beta without being TOLD I was buying one. But I realize something; I am a TW fan, and if it had been sold as a beta there are good odds I would have bought it. And a lot of us would have done the same, yes even most "whiners" around here.

How about this: Use a service like Steam or something similar. A direct-download-sales framework with built in updates. Release it this way as a limited release beta and sell it to us, with the game we buy rolling over into the final version when it turns into that. Maybe a discount buying it at this point would be nice, but even if not you'd still get a lot of us in on it. The update part is key, because it allows CA to keep everyone with the game at that point on the same page so to speak.

Because the TW series has an involved enough community, this could really work out well for everyone. I can guarantee all the modders would get in on it. Most of us who post here would. We'd get the game, we'd accept the bugs and incomplete parts because it's not finished and we knew that, and we'd help root them out. The game could undergo some final polishing during this and when it was ready for store shelves it could be complete enough it wouldn't need more than a minor update for people who bought it at final release. That leads to more dedicated TW fans, more community members, etc for the next go around.

I think this would work out fine ethically, because the beta players who are in it would be compensated by A) getting the game early and B) getting the full final game - possibly at a discount over the initial shelf price. CA wins because they can release a better, more reliable product at brick and mortars. The final release buyers win because they don't have to go looking for updates, they can just enjoy the game fully right out of the box.

[IsItStillThere]

Well, the game was released with a unit that caused the game to crash (Elephant artillery)

To be fair, this unit only shows up very late in the game. If they were playtesting by playing grand campaigns its easy to imagine them missing this.

I think the bottom line is, modern games are very complex. It would be hard to imagine even the most dedicated team of devs finding all the bugs, especially considering changes are being made up until the game goes gold (ie. changing one thing breaks something else).

I think the best we can hope for nowadays (from any new game) are timely patches that fix the major bugs. *does best Yoda impersonation* Beta testers are we, forget that not.

[Carl]

Actually I'm pretty sure autocalc is so streamlined things like enfilade fire simply aren't part of the equation. It doesn't simulate maneuver on the battlefield at all, it simply compares types and relative values against a random seed number. This is why castles are much easier to attack under auto; if you have the means to attack it simply throws the walls out of the equation altogether.

Actually we've had CA confirm that it DOES take things like manouvre into account. As I said, it's a cut down version of the main engine, we have been told it takes things like enfidle and manouvre and unit postioning into account, and we've also been told animations have no effect in auto-calc. beyond that we don't know how else it's been cut down, i'll try to find the post.

@Your Idea: i like it, I know I wouldn't mind doing that TBH, at leas if I could get in and mod it.

I think the bottom line is, modern games are very complex. It would be hard to imagine even the most dedicated team of devs finding all the bugs, especially considering changes are being made up until the game goes gold (ie. changing one thing breaks something else).

Exactly, As a member of both the DOW and TW communities and a coder in both I know just how easy it is to make mistakes with the limited amount of code we have to work with. The CA Devs have 1000's of times more to play with, that they make as many mistakes as they do is to be expected and if the mistake is subtle enough you won't notice it until a LOT later in my expiriance, with the complexity of code they have to deal with it will be even worse.

@Puzz3D: hanks for the confirmation, and ACTUALLY, I AGREE with CA when they say an animatio is too much for a small patch, the reality is that that kind of thing takes a lot of time and effort to do if your not after lifelike animations, if your after very lifelike animations as in TW, then honestly it's more bother than it's worth and I understand and respect it, even though I will acceppt that a changed animation might well have been a better soultion to the problems you raised.

As to your rant at Ca, i'm leaving it be or this thread will get locked.

[Carl]

The Post regarding what the Auto-Calc engine uses is here.

[seneschal.the]

Good observations in this thread.

It is obvious the Battle map didn't get much love at all from the testers/bug fixers (doesn't matter). Seriously, they released a game where the AI is completly broken for much of the battle map fighting (any siege, really; units STILL don't move away from fire, units STILL don't even REACT to your troops as they slowly march out, surround them and charge ahead (then they react, but if you withdraw everyone will return to the standard battle line so you can regroup, 30 feet away from them, the AI STILL in many cases just stand around doing nothing.) And lets not go into the whole animation debacle, or the broken combat mechanic system (remember, it's BAD to have a shield!). It would have taken one developer like 15 minutes to retry a battle 10 times to see that yes, the uber peasant exists. Yes, most of the 2H units does something wrong and it needs to be looked at etc.

Much of this isn't even the deep down stuff, but stuff that is obviously bugged. After just a few hours the player notices odd stuff like peasants and artillery crew being very very tough, 2 handers are acting strange, the broken AI etc. It took some time before the shield-bug was properly named, but we all knew SOMETHING was wrong with the combat calculations.

Odd thing is that this wasn't even rushed for christmas, which is usually when producers decide to ship an unfinished product just to get the extra revenue.

[Vladimir]

The question I’ve been thinking about is: “Is it practical to make a perfect 1.0.?” With the size and complexity of modern games, can a company afford to create a near-perfect experience without a massive beta test (read 1.0 version)?

[Satyr]

I think that when you add this theory, and the previous one proposed, ie. that the CA boys just aren't very good gamers, you probably have a fairly true look at the reasons this game works as poorly as it does in battle. It would also be why they just don't seem able to envision a better battle AI. Like other gaming companies that have gone before them, they seriously need a staff of the best TW players to write and test the AI. Until then we will continue to have a pretty face but an empty head.

[Slaists]

Just wanted to chip in that IMHO, the shield bug is the "mother of all bugs" (at least, balancing bugs) for MTW 2. Without any fixes all basic spear units are pretty useless against knightly cavalry (despite what the game advisor says). With shield value zeroed out and half the shield value going into armor, the other half - into defense, armored sergeants suddenly have a chance against cavalry. Even militia spearmen do...

[dismal]

@Dismal the elephant bug dosen't show up in Auto-Calc though.

That my point, by the time they where starting do do ANY testing in the Graphical engine, (the one we use when we fight the battles ourselves), SEGA was IMHO pushing for release so they never got it actually playtested in the Grapical engine, just the Auto-Calc engine where almost none of the games bugs actually show up.

Yes, understood.

I wasn't disagreeing with you. I tend to agree there are signs the game wasn't balanced much by actually playing it as we play it.

"Balancing" with autocalc strikes me as a bit of an oxymoron. A game "balanced" on autocalc is not going to be a balanced game because a decent player tends to achieve significantly better results than autocalc.

This might also help explain the AI's under defending cities and penchant for futile attacks. Those are probably wins on autocalc.

[Lusted]

1. SEGA: According to this post the Dev's weren't totally happy with the balance when it shipped anyway.

And that is mainly due to the shield bug as it messes up combat completely. Which CA did not know about, but now know about and i would have though it would be fixed in the 2nd patch.

Oh, and for those who claim its easy to see, seeing peasants beat other units makes me thinks peasants are overpowered not shields dont work. I've done extensive rebalancing and testing of units in M2TW and i never even thought there might be a problem with shields until the thread about it here at .org.

As for cannon elephants, the crash only seems to happen when you face a unit fo them that has taken casualties from a previous battle. That can only happen very late in the campaign, and the testers likely autoresolved all battles in test campaigns.

My own views:

1. Changes were made to the game code right until release, and with a game as complex as M2Tw is bound to cause some bugs.

2. Autocalc used too much to test battles.

3. Had to release when SEGA said, remember SEGA own CA, they dont just publish the games.

4. Changes made because of community feedback(yes, the 2 handed spin animation which was removed because of community complaints, and replacement animation not added properly) which are then not properly tested.

5. Too much reliance on patches to fix buggy release(common throughout PC gaming now, not just CA).

[Carl]

"Balancing" with autocalc strikes me as a bit of an oxymoron. A game "balanced" on autocalc is not going to be a balanced game because a decent player tends to achieve significantly better results than autocalc.

This might also help explain the AI's under defending cities and penchant for futile attacks. Those are probably wins on autocalc.

Partly, but the Auto-Calc engines AI should be equally stupid on both sides so it's a level playing feild. Thus in theory all the have to do is create a better battle map AI and check their is nothing the player can "exploit" based on the comments about animations having no effect in RTW I suspect that they didn't expect any problems beyond that. Thats still a big job, but it's much less than getting the basic balance right. Using Auto-Calc for that allows you to do it much faster, and with a greater amount of testing as with a simple auto graber program to record the resuls you can program 50 diffrent army combinations and test each a 1000 times in a matter of a couple of hours, somthing that would be nearly impossibile with Humans.

If you want an anolagy:

Somone is making a big metal item, they have 2 ways of making it, eithier do it by hand in a blacksmith, or cast it and then polish it to specification.

The Casting (auto-calc balancing), is reletivlly quick and can be reproduced on-mass. It then needs polishing, (the Graphical testing for exploits and to get a good AI going).

The Hand Crafting in the Blacksmith, (test it with humans throughout), has the advantages of being ore precice than basic casting and polishing, but only slightly and it takes a LOT more time and efort.

It's my belif that SEGA forced CA to rush it out the door so fast they never got the polsihing even started on really.

It would have taken one developer like 15 minutes to retry a battle 10 times to see that yes, the uber peasant exists. Yes, most of the 2H units does something wrong and it needs to be looked at etc.

Much of this isn't even the deep down stuff, but stuff that is obviously bugged. After just a few hours the player notices odd stuff like peasants and artillery crew being very very tough, 2 handers are acting strange, the broken AI etc. It took some time before the shield-bug was properly named, but we all knew SOMETHING was wrong with the combat calculations.

As you note, it took a while to figure out what was wrong, it dosen't matter if you know somthing is wrong, if you don't know what the problem is you can't fix it, and CA's reaction to the sheild bug discoveriy leads me to belive CA didn't know what the problem was even then.

I agree they should have snuck some graphical engine testing in amongst the auto-calc testing to check auto-calc and the graphical engine where matching up, but in the end the real blame, (but not necesserilly full blame), lies with SEGA for deciding to publish the game when they did, CA didn't have any control over that.

P.S. PAY ATTENTION TO LUSTED.

[Lusted]

It's my belif that SEGA forced CA to rush it out the door so fast they never got the polsihing even started on really.

Reminds me of RTw(though that was published by Activision). If anyone dares claim thaty RTw 1.0 was polished i will hit them. It's not, it was at least 1.3 before RTW could be considered polished.

[Foz]

Reminds me of RTw(though that was published by Activision). If anyone dares claim thaty RTw 1.0 was polished i will hit them. It's not, it was at least 1.3 before RTW could be considered polished.

So maybe things have improved a little then: from what I'm hearing, M2TW might resemble polished with the 1.2 release. At the very least I think Lusted has been more than clear that we expect the most game-breaking bugs will be gone, and the game without them is fairly close to polished at that point IMO. I think everyone will admit that it has the look and feel of a polished product, until something absolutely broken smacks you up side the head and forcefully removes you from the happy land of all things polished. The game primarily without those things, then, ought to be an immersing and thoroughly enjoyable experience. Perhaps I'm anticipating a little bit too much, but what can I say, I've got a sunshiny disposition at the moment

[Puzz3D]

As a member of both the DOW and TW communities and a coder in both I know just how easy it is to make mistakes with the limited amount of code we have to work with. The CA Devs have 1000's of times more to play with, that they make as many mistakes as they do is to be expected and if the mistake is subtle enough you won't notice it until a LOT later in my expiriance, with the complexity of code they have to deal with it will be even worse.

That's why it makes sense not to buy the game right now.

[PaulTa]

I will judge CA on the second Patch.

This new revelation means that while it might have been CA's fault, it was a mistake that wasn't entirely intentional. Now that they realize that some big mistakes have slipped through the cracks, I would expect them to remedy the situation.

If CA fails to remedy the situation properly (through patches) and abandons their work before it is "polished", then I will not purchase CA released titles in the future.

I will say though, given the size of the tasks set in front of them, I will be cheering them on if they do decide to polish their product. The amount of work that must go into making and properly implimenting these changes must be pretty big.

[Yun Dog]

My only problem will be IF .... (and this goes for the streamed beta idea as well) SEGA take the position of the past TW series publishers and set a 2 patch hard limit - ala STW, MTW, and RTW (till people threatened legal action over the save bug and big time complaining)

if thats the case it'll be M2TW 1.2 ... long wait

M2TW XP.. long wait

M2TW XP 1.1

I know other games like Stardock and Blizzard, Biowaredo the whole regular(more frequent) patching/updating thing... but that hasnt been the history here or at many other games co's where the patching is WOEFUL (still got Caesar IV in the drawer waiting for a fix to make that game work!!!)

The community finding all the bugs has never been a problem .. the problem has always been convincing CA (or the publisher) to allow CA to spend time/money fixing them.

I would like to see a change in policy on this front and would then have hope that this title and the next will reach their full potential

[Midnight]

Hang on a minute... if the game is 'balanced' in auto-calc, how come auto-calc against cities and castles gives such a tiny number of kills for the enemy compared with what would actually happen if you fought the battle? I was pretty sure that AC didn't take into account fortifications.

[Lusted]

My only problem will be IF .... (and this goes for the streamed beta idea as well) SEGA take the position of the past TW series publishers and set a 2 patch hard limit - ala STW, MTW, and RTW (till people threatened legal action over the save bug and big time complaining)

Well SEGA do have a different policy to patches. Remember we got 2 more patches for RTw once SEGA took over(1.3 and 1.5) and 1.6 for BI. So im guessing we will see perhaps 4 patches for M2Tw, 1.2 being the first big patch.

[GuyofJerusalem]

The only answer you need is that CA is only interested in your money and couldn't give a Sudanese camel's nuts about making a great game.

[Lusted]

The only answer you need is that CA is only interested in your money and couldn't give a Sudanese camel's nuts about making a great game.

Ahahhahahhahahahahhahahahaha.

You couldn't be more wrong.

[Carl]

Agreed, but he's saying a lot of stuff lie that and I think even one of the Moderators has called him a low-grade troll so make what you will of it.

This new revelation means that while it might have been CA's fault,

Remeber, it's just a THEORY, we HAVE NOT had this directly confirmed by CA, allthough it looks very likliy IMHO.

[dismal]

Thats still a big job, but it's much less than getting the basic balance right. Using Auto-Calc for that allows you to do it much faster, and with a greater amount of testing as with a simple auto graber program to record the resuls you can program 50 diffrent army combinations and test each a 1000 times in a matter of a couple of hours, somthing that would be nearly impossibile with Humans.

The presence of the Elephant Artillery crash probably means the whole Timurid invasion received zero playtest as the player would play it. I can't imagine this is "best practice".

In the old days, playtesters would literally be charged with finding creative ways to get the software to produce weird outcomes.

If you aren't playing the game as a user will play it and actively looking for issues, you are inviting the release of a buggy product.

It's my belif that SEGA forced CA to rush it out the door so fast they never got the polsihing even started on really.

Well, there is always going to be pressure of this sort. Sega or no-Sega. It would seem there is some level of balancing that is optimal pre-release. Some amount that gets the game good enough to not cost sales while getting it to the market on a timely basis.

I think the game is a lot of fun as it is, but there's no question it got out the door with some big issues.

[Carl]

The presence of the Elephant Artillery crash probably means the whole Timurid invasion received zero playtest as the player would play it. I can't imagine this is "best practice".

In the old days, playtesters would literally be charged with finding creative ways to get the software to produce weird outcomes.

If you aren't playing the game as a user will play it and actively looking for issues, you are inviting the release of a buggy product.
.

Thats my point Dismal though. They should have and probably DID intend to do that kind of playtesting before release, but never got that far before SEGA decided to publish the game. Had SEGA waited 3 months i'm sure this kind of thing would have been caught, the trouble was it got rushed out before they got more than the most basic testing done with real players.

Remeber, they obviouslly belived that the Auto-Calc engine was accuratlly representing what was going on in game. All they had left to do, (or so they thought), by the time they finish auto-calc testing is to check that their are no exploits like the Pike one, (where you order your cav to run through the pikes instead of charging them so that Cav now beat Pikes), Other than that i think they belived Auto-Calc and The player usage would produce nearly the same result.

Naive perhaps, but tottally understandable at the same time.

[Brighdaasa]

They should have and probably DID intend to do that kind of playtesting before release, but never got that far before SEGA decided to publish the game. Had SEGA waited 3 months i'm sure this kind of thing would have been caught, the trouble was it got rushed out before they got more than the most basic testing done with real players.

To me this sounds like BS. If CA wanted that level of playtesting and debugging they would have done so. CA knew the deadline SEGA set for them when they started on the project. That's when is decided how much time implementing, testing, debugging, playtesting, ... gets. If they needed 3 more months to get the playtesting done, they should hire a new projectmanager imho.
Edit: to calrify: SEGA didn't bring the release date any closer at any point so that there was no time to (play)test, CA let the development stage run too long and thus didn't have enough time for testing.

The shield bug also suggests to me that CA's internal or Sega's QA didn't do sufficient "unit testing". By this i mean take a finished piece of code and input all sorts of regular and irregular numbers and see if they produce the right result. It's usually a basic part of (internal) testing. Things like the shield bug shouldn't be found in playtesting but while testing the code.

On the other hand, they must have so much code that it may be impossible to find all those little coding bugs, no matter how good their testing routines are. Only someone with enough experience in testing such a huge project could give a definite answer on that, and i have to admit that's not me.

[Carl]

To me this sounds like BS. If CA wanted that level of playtesting and debugging they would have done so. CA knew the deadline SEGA set for them when they started on the project. That's when is decided how much time implementing, testing, debugging, playtesting, ... gets. If they needed 3 more months to get the playtesting done, they should hire a new projectmanager imho.

Dosen't matter, If SEGA didn't give them enough time to get the game done in, (and if they had they would have released a game with all those bugs taken out)., then it would have been done. SEGA didn't, it wasn't done and they still decided to release anyway. If SEGA actualy cared about it's custopmer they wouldn't have released it till testing was complete.

As to managment, we don't know why they didn't hit the deadline, based on what i know about bussiness practises, it's my belief SEGA gave CA the absolute minimum time necessery, which means even 1 minor delay would mean they'd never make the target. And in my expiriance NOTHING ever goes without some delays.

And thats my point, regardles of what CA should or should not have done, SEGA shoul;d never have released the game when even CA belived their was somthing wrong with the balance and that their where bugs.

On the other hand, they must have so much code that it may be impossible to find all those little coding bugs, no matter how good their testing routines are. Only someone with enough experience in testing such a huge project could give a definite answer on that, and i have to admit that's not me.

At least your being fair here (IMHO).

Intresting Data here for you guys, i did some sums to figure out the time and cost diffrances between Auto-Calc Testing and Human Beta testing to get a given amount of testing done.

To do a weeks worth of continus Auto-Calc testing by 1 PC, (8.6 million tests), would take 100 Beta Testers worjking 24 hours a day, 1 YEAR to complete. At the the British natinal minimum Wage it would cost CA approximetly 0.4 million pounds in beta tester wages too. Add in costs for electricity for the PC's, the PC's themselves, the cost of extra managment to recruit and manage Beta testers and the cost of employing the rest of the CA team for a year, and your probably looking at over 10 million pounds more in development costs. That would Jack up the price of games, (assuming they assumed 1 million copies would be sold), by some 10 pounds per copy.

They probably did a lot more, i'd bet at least 10 PC's doing a months worth of testing. hat would be impossibile with even 10,000 Beta Testers (42 million to employ that many too).

[Puzz3D]

SEGA didn't bring the release date any closer at any point so that there was no time to (play)test, CA let the development stage run too long and thus didn't have enough time for testing.

That's exactly the problem. It's easy to get carried away with adding features especially since there are plenty of customers who don't care if a game's features aren't working properly or if the gameplay is unbalanced. There are certainly plenty of people posting to this forum who don't care, so it would appear CA is right to give playtesting a low priority. This problem of inadequate playtesting extends into the patching phase, and I know that because I was on four Total War beta teams and it happened in all of them. You wouldn't believe how long we had to test the final build of RTW v1.2 if I told you, and that was going to be the final patch for it.

The shield bug also suggests to me that CA's internal or Sega's QA didn't do sufficient "unit testing". By this i mean take a finished piece of code and input all sorts of regular and irregular numbers and see if they produce the right result. It's usually a basic part of (internal) testing. Things like the shield bug shouldn't be found in playtesting but while testing the code.

Same thing happend in RTW with the phalanx not reversing the cav's charge bonus the way it was supposed to. To SEGA's credit they allowed two more patches to RTW after v1.2, and that charge reversal was fixed although now cav charge is reversed even when you charge into the rear of the phalanx which is still not quite right.

On the other hand, they must have so much code that it may be impossible to find all those little coding bugs, no matter how good their testing routines are.

That's why it becomes counterproductive to put too much pressure on the programmers since that increases the number of mistakes they make.