Thursday, July 15, 2010

The CCP Metric System

It's clear that CCP (like many of their customers!) loves numbers, and bases a lot of their decision-making on various "metrics" of performance. You can see this in several places in the minutes; here are my favorites:

Speaking on behalf of CCP, Nathan disagreed strongly with the claim that CCP isn’t committed to excellence. He pointed out that CCP probably spends a bigger part of its income on development than most other large, established game companies. He stated that this is a clear sign of this commitment.

Memory usage on the server per user has decreased in the last 12 months.

CPU Per User on the server has remained constant for the last 6 months except for a short period in late January where it went up just a bit.

Server log lines have been decreasing significantly since February, meaning fewer errors are being generated by the server

Only 0.3–0.5% of sessions end in a dirty shutdown (client crash).

But the most interesting example of all came not in the minutes, but just yesterday, in a comment by Nathan that was quickly reposted on Scrapheap Challenge:

That's funny, considering Apocrypha, the example for quality, created the most abysmal technical debt we've had.

What Nathan is saying here is that Apocrypha, an expansion the players loved, created a lot of issues behind the scenes at CCP that required them to do a lot of code cleanup afterwards.

This statement illustrates why Nathan was so upset by the CSM's complaints about lack of excellence. He apparently defines excellence as "excellence of the code providing the EVE experience". Players, on the other hand, define excellence as "excellence of the EVE experience".

Nathan is a very smart guy, and no doubt he has very good metrics on EVE code excellence. But experience excellence metrics are inherently fuzzy and hard to quantify, so when CSM presented some to him (including the huge Assembly Hall threadnaught), his response was to dismiss them as invalid or irrelevant.

And indeed, quantifying experience excellence is difficult, not the least because excellence is in the eye of the beholder, and there are over 100,000 different pairs of eyes playing EVE. I am reminded of US Supreme Court Justice Stewart's famous observation about porn: "I can't define porn but I know it when I see it."

Nevertheless, attention to experience excellence is very important, for the simple reason that CCP is not selling software. They are not selling a game. They are selling an experience, the experience of living in New Eden. EVE is not a commodity like potato chips (or World of Warcraft, for that matter); it is an emotional, immersive experience -- in fact, it is the only emotional, immersive MMO available today. EVE's crucial innovation can be summed up by the oft-repeated comment that "EVE is not a game; it's a job." You play World of Warcraft to escape; you live in EVE. The brilliance of EVE's original design can be summed up very simply: EVE is a job so good that people are willing to pay to work there.

This is why difficult to quantify metrics like experience excellence are so vitally important, because they encapsulate the very essence of the difference between EVE and other games; it's unique selling proposition.

Alas, history is replete with examples where decision-makers have learned the hard lesson that focusing on the "easy" metrics, ignoring the broader context, and dismissing contradictory information, was the road to ruin.

The whole point of the CSM is to give CCP a broader context, an extra dimension to their metrics. So when CCP staff go into a meeting with the CSM and their reaction is "how can these people be so clueless?", they ought to consider the possibility that they might also need to buy a vowel.

Or at least an umlaut or two.


  1. Or, as I put it in the tag-line of my blog: Its not a game you play. Its a life you live.

  2. On a slightly related note one thing in the minutes really stuck out at me

    "The CSM also stressed the importance of goodwill and overall player satisfaction, which is very hard to measure in statistics until players decide to quit"

    You would think CCP would bring out some sort of feedback form very once in a while maybe a month after each expansion that could be used for statistical analysis. It might also let them see if what they are developing is what the player base wants.

    As for your post I agree completely it seems that CCP and the players have a huge difference of opinion on what constitutes technical excellence as far as lag is concerned.

  3. The idea of a survey is a good one. You should consider raising it as a proposal and see if it gets traction. However, there are obvious problems with simple polling - how can you be sure, for example, that the sample of people who actually respond are representative of the population?

    Indeed, one of the complaints about the CSM is that it is not a good representation of EVE players, because most of them don't bother to vote!

    I would also be cautious when comparing different types of excellence; EVE could be evolving to be more excellent (based on particular metrics), while still having big issues in some areas, like lag. Also note that lag is a multidimensional problem; there are different types of lag, and some are not amenable to purely technical solutions. As my old CS professor put it, "exponentials are a bitch".

    The whole point of my essay was that it matters what metrics you use, and how you weight them.

  4. I really appreciate this blog and the insights you share on it. I'm not a 'technical' person, so your posts help me understand such matters.

    Nathan's focus on a technical as opposed to an experiential definition of excellence is just such an example. In public policy, we have a similar issue between those advocating 'technocentric' or 'scientistic' approaches to making and implementing policy. The value-laden and cultural dimension of policy disputes and resolutions are peripheralized by this mindset.

    Of course it is not a matter of choosing one *over* the other, but of integrating them to a common purpose.

    In this respect, focus groups would be an indispensable follow-up to a survey. The major survey research centre's now have 'qualitative hot houses' where survey results are plumbed in-depth for their meaning.

    Indeed, in one sense the CSM is an ongoing focus group, and a very valuable one at that.

    Looking forward to more of your posts.


  5. MadOverlord,

    You hit the nail on the head!

    "He apparently defines excellence as "excellence of the code providing the EVE experience". Players, on the other hand, define excellence as "excellence of the EVE experience"."

    68 degrees sounds great until you realize that one person is using a Celsius scale and the other is using Fahrenheit.

    Nicely done. =)