The CCP Metric System

It's clear that CCP (like many of their customers!) loves numbers, and bases a lot of their decision-making on various "metrics" of performance. You can see this in several places in the minutes; here are my favorites:

Speaking on behalf of CCP, Nathan disagreed strongly with the claim that CCP isn’t committed to excellence. He pointed out that CCP probably spends a bigger part of its income on development than most other large, established game companies. He stated that this is a clear sign of this commitment.

Memory usage on the server per user has decreased in the last 12 months.

CPU Per User on the server has remained constant for the last 6 months except for a short period in late January where it went up just a bit.

Server log lines have been decreasing significantly since February, meaning fewer errors are being generated by the server

Only 0.3–0.5% of sessions end in a dirty shutdown (client crash).

But the most interesting example of all came not in the minutes, but just yesterday, in a comment by Nathan that was quickly reposted on Scrapheap Challenge:

That's funny, considering Apocrypha, the example for quality, created the most abysmal technical debt we've had.

What Nathan is saying here is that Apocrypha, an expansion the players loved, created a lot of issues behind the scenes at CCP that required them to do a lot of code cleanup afterwards.

This statement illustrates why Nathan was so upset by the CSM's complaints about lack of excellence. He apparently defines excellence as "excellence of the code providing the EVE experience". Players, on the other hand, define excellence as "excellence of the EVE experience".

Nathan is a very smart guy, and no doubt he has very good metrics on EVE code excellence. But experience excellence metrics are inherently fuzzy and hard to quantify, so when CSM presented some to him (including the huge Assembly Hall threadnaught), his response was to dismiss them as invalid or irrelevant.

And indeed, quantifying experience excellence is difficult, not the least because excellence is in the eye of the beholder, and there are over 100,000 different pairs of eyes playing EVE. I am reminded of US Supreme Court Justice Stewart's famous observation about porn: "I can't define porn but I know it when I see it."

Nevertheless, attention to experience excellence is very important, for the simple reason that CCP is not selling software. They are not selling a game. They are selling an experience, the experience of living in New Eden. EVE is not a commodity like potato chips (or World of Warcraft, for that matter); it is an emotional, immersive experience -- in fact, it is the only emotional, immersive MMO available today. EVE's crucial innovation can be summed up by the oft-repeated comment that "EVE is not a game; it's a job." You play World of Warcraft to escape; you live in EVE. The brilliance of EVE's original design can be summed up very simply: EVE is a job so good that people are willing to pay to work there.

This is why difficult to quantify metrics like experience excellence are so vitally important, because they encapsulate the very essence of the difference between EVE and other games; it's unique selling proposition.

Alas, history is replete with examples where decision-makers have learned the hard lesson that focusing on the "easy" metrics, ignoring the broader context, and dismissing contradictory information, was the road to ruin.

The whole point of the CSM is to give CCP a broader context, an extra dimension to their metrics. So when CCP staff go into a meeting with the CSM and their reaction is "how can these people be so clueless?", they ought to consider the possibility that they might also need to buy a vowel.

Or at least an umlaut or two.