Dear CA: Beta-Testing, Beta-Testing, Beta Testing

[ToranagaSama]

I think whinning for a patch, rightly, needs to evolve into a community SCREAM for CA/Activision to implement a "beta-test" program, with beta-testers drawn from the community.

In the long run, it s/b more cost-efficient, in terms of "multiple" patches with better results. JMO.

Better start screaming NOW, if we want beta-testing for Rome.

The guys from CA have previously stated, that they don't control decisions regarding Patching, Activision does; and that Patches MUST go through "testing" which is done by Activision. This is a BIG part of why so few patches.

I think the COMMUNITY will attest that Activision's "Testing" procedure is proving, increasingly, inadequate to OUR standards; or, at least, that improvements need to be made regarding "Testing".

---

Additionally, below is a model for an appropriate Patch "Policy". Most of it I have experienced with other games:

After "Release:

1) Quick Patch to address OBVIOUS bugs;

2) 2nd Patch to address LESS obvious bugs;

3) 3rd Patch to address *improvements* and additions to game "elements". In TW's case that would be Strategic and Tactical elements. "Elements" consist of *things* that the developers ran out of time to input by release date, but were intended to be included. Additionally, Elements include "suggestions" and "insights" provided by the Community.

Speak NOW, or forever hold your peace Its time for the Community to rally for its cause.

~ ToranagaSama

[ToranagaSama]

Oh, I was reading Old Bald Guy's thread and it remined me I forgot to add as a SIGNIFICANT benefit to beta-testing:

I MUST state that The Creative Assembly is one of the absolute best developers when it comes to listening to its game community and addressing the community's concerns. The problem is that the community will comment regarding "Release 1", but the serious comments don't get implemented until "Release 2".

Community Beta-Testing SHOULD allow that much of the community input, at least the beta-testers input, to make into the "developing" release.

[Mount Suribachi]

I have thought for a while that CA should do some beta-testing using the vets on this forum. I don't mean every tom, dick and harry, but the respected guys - we all know who they are.

Say 10 to test the MP, 10 to test the campaign and give feedback to CA, and feed the rest of us tidbits - not that that includes me right now as I am not buying RTW till there is a VI patch. But semi-public beta test of RTW would be a massive step in the right direction.

[rasoforos]

i wonder what will happen if another gaming company starts producing DECENT TW like games. i guess we wont have to scream or pay £20 for 'patches' that offer a whole range of exciting new bugs , anymore. http://www.totalwar.org/forum/non-cgi/emoticons/biggrin.gif

[Gregoshi]

The best players aren't necessarily the best testers, or I should say, they aren't the only good testers. You need inexperienced players (new to the game system) and imaginative testers with strong curiosity drive: "I wonder what would happen if I did this..."

[Teutonic Knight]

I'm a screamin' I'm a screamin'

I've been E-mailing Activision letting them know that they are going to start losing loyal customers if they don't straighten up. I have given them links to the above sticky...

still no response...

[ToranagaSama]

Quote[/b] (Gregoshi @ July 09 2003,15:12)]The best players aren't necessarily the best testers, or I should say, they aren't the only good testers. You need inexperienced players (new to the game system) and imaginative testers with strong curiosity drive: "I wonder what would happen if I did this..."
I think what you're talking about Gregoshi are "Usability" testers.

Beta-Testers would be those with deep intimate knowledge of the game's functioning as players (not coders). Such people would need to have the time and inclinition to play the game like mad for a great many weeks. They also need good communication and documentation skills. A good broadband connection would be helpful too, for downloading huge files on a daily or weekly basis.

Newbie's wouldn't, normally, fit the bill. Though, its always prudent to run your game code by such uninitiated players in order identify areas of the Interface and/or functions that might be overcomplicated and intimidating to someone new to the game.

[Brutal DLX]

I'm pretty sure CA would consider doing that if they had a say in that matter. They often visit the boards and as far as I know, some MP players worked on testing the 1.2 patch for STW... Also respected guys like Erado San were mentioned in the credits of STW MI...
Generally it is a great idea to let community beta testers have a go at the unfinished versions of the game, rather than to let AI play the SP campaign by itself in countless cycles. After all, the interaction between the player and the AI is the most important thing and if 10 testers play 20 SP campaigns each, I'm sure you could identify a lot more little issues that would get overlooked otherwise.

[Rocket_Boy]

Totally agree ToranagaSama

[Longasc]

Wonder, I am

WILLING to help patching VI

-> besides bitching about the still non-existing patch here everyday... http://www.totalwar.org/forum/non-cgi/emoticons/cool.gif

[ToranagaSama]

Thanks for the comments guys. 142 views, you people need to speak up

If the powers that be are listening, the least that can be expected would be:

Unofficial Patches beta-tested by the community.

Is that too much to ask?

[hrvojej]

You might want to check this thread (http://www.totalwar.org/cgi-bin/forum/ikonboard.cgi?act=ST;f=5;t=8243), and Gil's response to Wellington.

[Mount Suribachi]

Hey, I work in QA, pick me

(testing pharmaceuticals, not computer games tho http://www.totalwar.org/forum/non-cgi/emoticons/frown.gif )

[WesW]

Quote[/b] (Gregoshi @ July 09 2003,13:12)]The best players aren't necessarily the best testers, or I should say, they aren't the only good testers. You need ...imaginative testers with strong curiosity drive: "I wonder what would happen if I did this..."
Like, say, mod-makers? http://www.totalwar.org/forum/non-cgi/emoticons/biggrin.gif http://www.totalwar.org/forum/non-cgi/emoticons/biggrin.gif http://www.totalwar.org/forum/non-cgi/emoticons/biggrin.gif

I also won't be back in school until at least next January, so I will have a lot of time to spend on the game, too.

I just hope this beta-testing goes better than what happened to us in Call-to-Power II. The poor designer there, where they were producing the game in-house for Activision, couldn't even get the mail room to send us our copies of the beta until weeks had passed.

I hope all of Activision's testing for CA's games involves hardware compatibility, since their in-house people aren't going to know anything about the game itself.

[ToranagaSama]

Quote[/b] (hrvojej @ July 10 2003,14:36)]You might want to check this thread (http://www.totalwar.org/cgi-bin/forum/ikonboard.cgi?act=ST;f=5;t=8243), and Gil's response to Wellington.
Thanks hrvojej.

Totally forgot about that thread and GJS' response.

Agitate, agitate, agitate....

BTW, WesW, if you get the job, then....ahhh....what will happen to future mods of the MedMod? http://www.totalwar.org/forum/non-cgi/emoticons/shock.gif

Good Luck

[Obex]

Quote[/b] ]Thanks for the comments guys. 142 views, you people need to speak up

I completely agree with ToranagaSama. I just wanted to throw my vote in here. I hope that GilJaysmith's faction wins out.

[khurjan]

Quote[/b] (Gregoshi @ July 09 2003,06:12)]The best players aren't necessarily the best testers, or I should say, they aren't the only good testers. You need inexperienced players (new to the game system) and imaginative testers with strong curiosity drive: "I wonder what would happen if I did this..."
hehehe i know what you saying the especially the one
what would happen if i did this lol...no wonder i been doing so much beta testing and hair pulling when an interesting game went kaput

[Kraellin]

having been in on both the beta tests for this game, i can say that you need both experienced and inexperienced testers. some of the best technical manuals are written by folks that have never used the program before and just sit down and figure stuff out and then write about what they find. folks that have been using a program, game or otherwise, for a long period of time, tend to get into a mode of doing what's expected. it's the unexpected sometimes that can produce the biggest bugs.

i'd also say that you do tend to want experienced players if the beta test is for game balancing and not bug finding. and there's a vast difference in these two types of betas. in the we/mi patch, we were balancing mostly and not bug finding, thus we spent a LOT of time doing one unit vs one other unit to try and balance the game. in the other beta we were doing bug testing and NOT balancing. you have to pay attention to the purpose of the test cause it's not always the same. therefore, you want testers that fit that particular purpose the best.

a beta test for Rome, would prolly involve a mix of the two, bug testing and balancing and even a third, game features, so you'd want a rounded test group there. if mtw were to receive a patch, from what i've seen, the issues here are mostly bug related, so you'd want a bug finding group.

the real trick as a tester is sticking within the parameters of the purpose of the test. you get 'hired' to test for bugs, but you sometimes want to start doing balancing or re-designing things, and so you sometimes fail to produce what is needed and wanted.

there's also a case of incentive. i'm not sure what the activision testers get rewarded with or if they are activision employees or not. if they are, then most likely they're not quite the rabid fans that this group here in the forums are. and you also wanna know if anything is being offered as a prize for helping test. in the we/mi test, we got a free copy of the game. in the other one, we got a nice thank you. it's also perfectly ok that the incentive is simply a better game for all. perfectly fine.

so, all i'm saying here to all the volunteers is, know what is needed and wanted BEFORE you volunteer. game testing sounds like a lot of fun, but it's often quite tedious and boring testing this unit against that unit X number of times in controled tests and then moving on to the next 100 tests of the same order. or having to re-write all the stat files and balance them across the boards and then re-testing to make sure you got it right. and maybe you dont get to test a thing to do with balance or the campaign stuff just isnt needed and wanted as far as testing and all that's wanted is to duplicate certain bugs that folks have reported. it aint all it's cracked up to be sometimes, ya know?

and i'm pretty sure i've posted this before somewhere, but you just cant hardly do an open beta on balancing issues. let's see, we'll take 5000 testers and let them try to balance the game. we'll put up a forum where they can debate their findings and discuss what they want. lol. you'd better provide bunkers and big guns cause you'll see a firefight like you wouldnt believe. bloody worse than discussing politics or religion. i wish i'd saved a copy of some of those discussions we had with just our small group. a lot of them started out with, 'jane, you ignorant slut' ;)

actually, it wasnt quite that bad, but it does get magnified with the more folks you add into the mix.

i'll also add that it's a bit of a frustration when you are hired to do bug testing and what you want to really do is re-design some known features or add some more in. well, yer pissin in the wind on that one. each test really does have certain parameters that are expected to be met, so you send up your nice report about all the things you think shld be re-designed and find out you failed the test miserably cause that just aint what you was hired fer.

i know some of you out there are thinkin, 'boy, if i could just get CA's or Activision's ear by beta testing, i could re-write this game into my ideal dream'...well, that's what it is, here, a dream. it aint gonna happen. now, we did get a couple things implemented in the we/mi test and that was great, but we also found even more things out that drove us nuts and just couldnt get changed. once you start really testing from that viewpoint of a tester, and really start digging into how things work, you find some of the 'design flaws and shortcuts' that just drive you nuts and that just arent going to be changed. yuuki and tosa and doc all know what i'm talking about here, the gun models in we/mi. there was just no way to make them work right without a fairly sizable re-write of the code and that just wasnt within the scope of what was being done, so we just had to accept that do the best we could with it.

there is also another factor that comes into play in beta testing. some of you are history buffs. you've read this account of a given period of time or of this particular battle and tend to believe that that's the way things shld be, yet the game may not display it that way, so you think you're going to re-write the stats or get CA to re-write the code to fit your image of history. hehe, that, my friends, is a trap. 'History' is a perception based on available data. that means that you might have read one account but there might also be 17 other accounts you're not aware of, or, you might have read something about an occurrence in one region whereas in another region things were completely different. tosa and i used to argue about guns and gun ranges and accuracy all the time in the we/mi testing. we'd find sites to back up our particular belief and as often as not, the sites would differ. horses and horse speeds was another one. how fast can a horse ride with an armored rider. well, there's a LOT of variables in there not accounted for. what type of horse, how much armor, weather conditions, old horse, new horse, well fed, etc, ad nauseam. i finally went to a race track site and got some data on how fast a really good horse could run across a certain distance with just a jockey and saddle. we then took that and adjusted for things like armored rider, and finally, we just said, well, what plays well, though it still used some of the stat finding done earlier.

i've designed games before. and play balance can be a very tricky issue. just how much do you try and model. how much detail can you get away with and at what point does it just become pointless bit twiddling and a trial of unneeded significance and ruin, rather than enhance game play. CA has done a workable job of this. you get quite a bit of stat detail without bogging down actual game play, for the most part. and that's really all you want, workable.

i do think that fans make the toughest critics. they know the details of a game more intimately sometimes than the coders themselves. it's really not that unusual. thus, you might think fans would make the best beta testers as well. for some things i think this is true, for some others, not so true.

i also tend to believe that a lot of you out there are sort of frustrated game designers. i know i am ;) just remember, this forum IS a sort of platform for contributing to that game design. the CA guys are here and do participate and despite some old haranguing about 'nobody ever listens', we've found that to just not be true by evidence. it's also true that your wish list isnt necessarily going to happen overnight or even at all. i would point out the new article on the front page of the .org as further evidence. i was impressed by a lot of the things coming in rtw. go read it ;)

ok, i've strayed way off the topic here. i do hope there is another patch for mtw/vi. if nothing more than a PR move, i'd say it's worth it.

K.

[Gregoshi]

Kraellin - you just hit an out-of-the-park grand slam with that post. http://www.totalwar.org/forum/non-cgi/emoticons/pat.gif

[WesW]

Quote[/b] (ToranagaSama @ July 11 2003,02:04)]BTW, WesW, if you get the job, then....ahhh....what will happen to future mods of the MedMod? http://www.totalwar.org/forum/non-cgi/emoticons/shock.gif

Good Luck
Well, I don't want to turn this into a medmod thread, but I am really done with the Viking Campaign, and once I get the new concepts and units decided upon for the full version, it will be mostly administrative stuff and writing, so it won't take up a lot of time, really. I do most of my mod work when I want to unwind (which is one reason I hope to get into the industry one day...money for nothing and all that), so if the beta work gets to be a job, it's still all right.

Btw, I too want to congratulate Kraellin on his post.
For the CtP II testing, we were reporting bugs that came up, and also commenting on features that weren't working right and settings that were WAY out of balance- mostly economic stuff like trade route revenues.
In our case, we didn't know enough about what the designers were *trying* to do to worry about telling them how it *should* be done, and I don't think that any of us would have done that anyway. For me, it was a privledge to help the designers turn their vision into reality, and that is how I would approach it this time.
I have just been very, very excited about everything I have read about RTW, and it would thrill me to no end to be able to experience it early, even in a buggy state.
One thing that my mod-making has done, that perhaps someone who just plays the game wouldn't do, is that I have spent a lot of time documenting and trying to improve upon the things that most people here agree don't work well in MTW. If I were able to catch any of those things that were still present in RTW, and bring them to the designers' attention so that they could be fixed, I would consider it time very well spent.

[Odinn]

They could start by patching VI and get rid of the 56 year bug.

[+DOC+]

I was gonna write a reply to this thread, but on reading through all the posts i see Kraellin has already said everything that i would've mentioned anyway.

Thanks for saving me the time and trouble buddy http://www.totalwar.org/forum/non-cgi/emoticons/pat.gif http://www.totalwar.org/forum/non-cgi/emoticons/wink.gif

If you haven't already read his lengthy post, read it, as it will give you a great idea as to what the whole thing involved. A lot of head to wall action and at the same time rewarding and frustrating.

Regards

doc

PS oh and that Jane, she was such an ignorant slut http://www.totalwar.org/forum/non-cgi/emoticons/wink.gif

[Kraellin]

hehehe doc,

we did have some good arguements, didnt we ;) (actually, jane is what i called doc ;)

K.