CA loves it some Metacritic

This is an interesting article about how CA's game design decisions are influenced by Metacritic. My instinct on reading this is to be horrified, but at the same time I think the TW community (or at least the Org's small portion of it) is of the opinion that TWS2 was far more polished and focused than most of the previous games, at least going back to MTW. So, perhaps this is a good thing. Thoughts?

Full article:

Total War developer Creative Assembly shoots for a 90 percent Metacritic score for its games, and has a "brutal" approach to getting there, the studio tells Gamasutra in our latest feature.

Metacritic's average review scores have become increasingly important for developers in recent years, as many publishers and consumers refer to those numbers as an indicator of a game's quality, as well as the quality of a studio's output. In some cases, those scores can determine whether developers receive bonus payments from publishers.

Creative Assembly's studio director Tim Heaton says that his team goes through a rigorous process when deciding what features to include in a game. The Sega subsidiary will cut some features immediately, but it will prototype and judge the quality of others before deciding to ditch them. And while prototyping ideas and examining their process, the group will also abandon features that it feels will take too long to turn into a high-quality addition.

"Through production, actually, we do what we call 'Metacritic analysis,'" says Heaton. "We will break those features down into subsets, and we both look at it from a player's point of view, and a reviewer's point of view, and we'll weigh certain features as to how we see players and reviewers look at them, and they'll build up to a 100 percent score, and then we'll judge where we feel we are on those individual feature sets, and see the momentum on those and the velocity on those, too."

He adds, "And so if we see one flat line and it's not where we want it to be, we then will cut it. Well, we'll cut it really late in the day. I think teams are really scared about doing 90 percent of the work and then cutting it. It's kind of like, 'Well, it's nearly finished; I... I've done all the work! Please don't cut it! I'm sure I can make it better.' And we're fairly brutal on that."

Heaton says he would much rather reject a feature that the studio invested resources in, than have it left in the game and affecting its quality. "You know, every step of the way -- from the beginning to the end -- we're talking about a 90 percent Metacritic," the Creative Assembly director emphasizes. "That's our goal. That's what we tell Sega. And we communicate that through graphs, basically, of where we think we are."

He continues, "We build into that also, on that Metacritic analysis, external events. So if we think we've done a really great PR job, if there's an individual event that we've done really good, we might add, you know, a .5 percent Metacritic. If we think it's :daisy: up or somebody's not done their job right, or miscommunicated something, or whatever, we'll see that in our Metacritic analysis. And we share that with Sega on a weekly basis, so that they can figure out how we're doing, too."

Heaton admits that cutting a feature that might have been nearly completed can affect the studio's morale, but he believes the team eventually buys into the idea that these decisions are for the best: "Certainly some elements of a team -- and this always happens with every team I've ever worked with -- just go, 'Come on, just give me a game design document. Just tell me what I need to do and then I will do it to the best of my ability.' And we slowly, hopefully, educate people that's just not the best way.

"And we will enter a fog of ideas for quite a long time, and some of those things will have risks against them right up until the end, and then we might pull them. It pisses people off, absolutely, but it's for the best. But nothing makes the team prouder than delivering a 90 percent game and selling two million copies. So that's the bottom line, and people do come to understand that."

The full feature interview, in which Heaton talks more about Creative Assembly's process for building high-quality games, is live now on Gamasutra.

I'd be curious to hear the whole story (at some misty distant future date) of what happened with the Empire AI. My understanding, gleaned from snippets here and there, is that there was a crucial employee who set it up and then left unexpectedly. So the guy who knew the system and had slung the code was suddenly gone. Anyway. It's one of those HR things that I may never hear fully, but it would be nice to know what happened.

Especially given that the ding everyone (including Metacritic) gave to ETW was the AI. And given all of that talk about cutting features rather than including anything unfinished. Something went seriously wonky around the time of Empire.

I actually agree with the game company's strategy, becuase MetaCritics is probably the best available indication of game sales out there. For an unfamiliar series, I choose my games based on MetaCritics score, too.

But Metacritics score is just a pool of reviewers' 2-20 hours of exposures. So features that will increase the game's replayability after 200 hours might not be getting as much attention as the "WOW!" features.

Perhaps one day there will be website that only accepts articles from veterans who have played the game for 200 hours or more. Then the "Veteran Score" will have a strong pursuasive power for those who seriously look for a good game to enjoy for a long time.

There are also so many different types of players out there. I am personally very allergic to repetitiveness, but I know most people are almost indifferent to it. So even if we eventually find a perfect scoring system, individual experience may differ significantly.

I also like the idea of dropping features. It is good to keep things simple and fun, rather than complicated and having no point of existence. A lot of modders like to expand the game into their likings, but it is their personal preference. If I have to go through 300 different units, instead of 30, to decide which ones to use, I will probably immediately drop the game.

I suppose I assumed that the Metacritic-focused-development bit discussed in the article was implemented after NTW but before TWS2, with implications then about how TWS2 was such a significant improvement. However, that really was an unfounded assumption. If they've been using the same development strategy for games prior to TWS2, it's utility is somewhat more questionable. I suppose I'll actually have to go listen to that whole interview now...

Quote:

---

Originally Posted by TinCow

I suppose I assumed that the Metacritic-focused-development bit discussed in the article was implemented after NTW but before TWS2, with implications then about how TWS2 was such a significant improvement. However, that really was an unfounded assumption.

---

I am not sure about that. NTW was quite a polished product, certainly in comparison to ETW. The battle AI was pretty good and the campaign AI gave me a good dance. It was also rather pared down in features and scope compared to the absurdly ambitious ETW. In many ways I think it shares some of the virtues of STW2. Now, if you are talking about the Metacritic focused developmend applying to ETW and earlier, I would agree it would be more questionable.

Strangely, I find I have clocked up vastly more hours on the ambitious but buggy ETW than the more restricted but polished NTW and STW2. ETW, like RTW and MTW2, just has that amazing scale and freedom. It has also been patched to be quite servicable now.

Game design decisions driven by metacritic makes me laugh. Cause and effect are reversed.

NTW is a much more polished and presented in a tight and efficient package, that's why it's much better than ETW. It lacks the scale of it though, but S2TW had a good influence I believe and hopefully the good Metacritic scores will guide them well.

Yeah, and if they really want to take it to the next level, they can go smart mob on it and let us make all their design decisions for them. :wall:

I doubt it is just CA who aims high on Metacritic I would expect it is every big publisher / developer if that is right or wrong way to go about making games I do not know but I guess we will have to live with it or stop game reviewers giving games a random number to indicate how much they liked it.

Quote:

---

Originally Posted by TinCow

This is an interesting article about how CA's game design decisions are influenced by Metacritic. My instinct on reading this is to be horrified, but at the same time I think the TW community (or at least the Org's small portion of it) is of the opinion that TWS2 was far more polished and focused than most of the previous games, at least going back to MTW. So, perhaps this is a good thing. Thoughts?

Full article:

Total War developer Creative Assembly shoots for a 90 percent Metacritic score for its games, and has a "brutal" approach to getting there, the studio tells Gamasutra in our latest feature.

Metacritic's average review scores have become increasingly important for developers in recent years, as many publishers and consumers refer to those numbers as an indicator of a game's quality, as well as the quality of a studio's output. In some cases, those scores can determine whether developers receive bonus payments from publishers.

Creative Assembly's studio director Tim Heaton says that his team goes through a rigorous process when deciding what features to include in a game. The Sega subsidiary will cut some features immediately, but it will prototype and judge the quality of others before deciding to ditch them. And while prototyping ideas and examining their process, the group will also abandon features that it feels will take too long to turn into a high-quality addition.

"Through production, actually, we do what we call 'Metacritic analysis,'" says Heaton. "We will break those features down into subsets, and we both look at it from a player's point of view, and a reviewer's point of view, and we'll weigh certain features as to how we see players and reviewers look at them, and they'll build up to a 100 percent score, and then we'll judge where we feel we are on those individual feature sets, and see the momentum on those and the velocity on those, too."

He adds, "And so if we see one flat line and it's not where we want it to be, we then will cut it. Well, we'll cut it really late in the day. I think teams are really scared about doing 90 percent of the work and then cutting it. It's kind of like, 'Well, it's nearly finished; I... I've done all the work! Please don't cut it! I'm sure I can make it better.' And we're fairly brutal on that."

Heaton says he would much rather reject a feature that the studio invested resources in, than have it left in the game and affecting its quality. "You know, every step of the way -- from the beginning to the end -- we're talking about a 90 percent Metacritic," the Creative Assembly director emphasizes. "That's our goal. That's what we tell Sega. And we communicate that through graphs, basically, of where we think we are."

He continues, "We build into that also, on that Metacritic analysis, external events. So if we think we've done a really great PR job, if there's an individual event that we've done really good, we might add, you know, a .5 percent Metacritic. If we think it's :daisy: up or somebody's not done their job right, or miscommunicated something, or whatever, we'll see that in our Metacritic analysis. And we share that with Sega on a weekly basis, so that they can figure out how we're doing, too."

Heaton admits that cutting a feature that might have been nearly completed can affect the studio's morale, but he believes the team eventually buys into the idea that these decisions are for the best: "Certainly some elements of a team -- and this always happens with every team I've ever worked with -- just go, 'Come on, just give me a game design document. Just tell me what I need to do and then I will do it to the best of my ability.' And we slowly, hopefully, educate people that's just not the best way.

"And we will enter a fog of ideas for quite a long time, and some of those things will have risks against them right up until the end, and then we might pull them. It pisses people off, absolutely, but it's for the best. But nothing makes the team prouder than delivering a 90 percent game and selling two million copies. So that's the bottom line, and people do come to understand that."

The full feature interview, in which Heaton talks more about Creative Assembly's process for building high-quality games, is live now on Gamasutra.

---

I'm going to beat my drum about my dislike for the battles. Everything outside of them seems alright but the battles are the heart. Based on my modded M2 experience I never imagined I would choose Minecraft or the Civilization, or Tropico...other than Total War.

I don't know if they could have added more, I just know they should have got the heart of the game to 100%.

It should not come as a surprise, but the "achievements" system in Steam is not really there for the player's satisfaction. It is in reality a "yes or no" survey that players fill out when they play. For example, there is one achievement that is for simply connecting two provinces with a railroad. Globally, it's at 4.8%.

I have no idea what percentage they will look for, but if it does not meet their metric then you may not see a feature like this in future games.

I happened across that interview since it was linked at the TWcenter in an abbreviated "Eurogamer" piece.

The one thing that stood out to me was that Heaton never referenced the fan base--nor any love or pride of the TotalWar games. He only used the term "players" in a context that could have just as easily been the word "customers" As far as he was concerned, he was simply talking about a product that could have been photoshop software or even kitchen appliances. To me he came across as just a "suit" in a management position and it was all about the metrics and numbers.

From some of his comments about "morale", I also bet he is not a well loved boss, but I may be reading too much into it.

Cheers

I think you've pretty well nailed it on the head is what I think.

I don't think it's news that companies want their products to have good reviews.

No, it's not news when someone announces that companies make bad decisions.

And the lack of novelty changes nothing about the fundamental truth of the statement.

Metacritic apparently hugely effects sales of a game. I'd like a wikipedia style gouge of a games quility for many people.