The Cold Equations of Soul-Crushing Lag

Yesterday, there was a bit of a commotion in HED-GP. Thousands of pilots attempted to engage in the largest battle in EVE history, and despite the best efforts of CCP, they were devoured by the monster that is Soul-Crushing Lag.

The dreaded Lag Monster has returned to New Eden more and more often in recent months, and it will continue to do so with increased frequency in the months and years to come. This post is my attempt to explain why I believe that even heroic technical achievements like Time Dilation will only put Lag into remission; to cure it requires some fundamental changes to the core game design of EVE.

As befits players of a game that has been described as Spreadsheets in Space, let us begin with some mathematics, to explain why lag can't be conquered by technical means. For most of you, especially those who are computer programmers, this will be basic stuff but bear with me. All the numbers given below were chosen simply to illustrate the problem.

The basic problem is this: "doubling the number of people in a fight more than doubles the amount of work the server needs to do."

To give a horribly simplified example (forgive me, anyone who's taken a 200 level CS course), consider a fight with 100 people; for each person in the fight, the server needs to update their position, execute their commands, and so on. That part of the computational load is roughly linear, so adding another 100 ships would roughly double the amount of work. But then the server has to tell everyone how the universe has changed, and everyone can potentially have a different view of the universe (consider: cloaked vs. uncloaked; information about the ships you have locked; watchlists; etc). So now instead of 100 ships each getting info about 100 ships, you have 200 ships each getting info about 200 ships; that's 4 times as much information. If you have 500 ships, that's 25 times as much; 1600 ships requires 256 times as much. All that data has to be organized and sent to the clients. Things get very bad, very fast; then they get even worse, even faster.

Now in the real world of extremely clever programmers, of which CCP has more than their fair share, there are a lot of things you can do to make things (to use the technical term) "less horrible". But what you can't do is get linear scaling; doubling the number of people in a fight will always more than double the amount of work the server needs to do.

So what can be done to address this problem? Well, there are several basic approaches, and CCP can use any or all of them; they are not either-or choices:

• Use faster computers (aka "If brute force isn't working, you aren't using enough"). Unfortunately, because of design decisions made back in the early days of EVE development (decisions that I think were made for very good reasons, btw), computers are not getting faster in a way that benefits EVE as much as everyone would like. Time Dilation is actually an example of this approach -- when TiDi hits 10%, the server has 10 times as much time to process each tick of the game clock, so it's like it's running 10 times faster.
• Make the code more efficient, so that it scales better. In our simple example above, 2x the number of ships meant 4x the work; if that gets reduced to 3x the work, then handling 1600 ships is only 81 times more difficult instead of 256 times as hard, and that turns out to roughly double the number of people that the server can handle before the lag monster appears. A lot of work -- extremely difficult work -- is being done in this area, and IMHO CCP needs to put even more resources into it (and not just because of lag, either).
• Change the game so that fight sizes are naturally limited to sizes the servers can handle. By "naturally limited" I mean make changes such that effective fleet commanders will have sound tactical and strategic reasons to limit their fleet sizes and/or divide into sub-fleets with different objectives.

Let's examine these possibilities in more detail.

Time Dilation was introduced about 3 years ago, and has been in-game for just over 2 years. As explained above, it basically speeds up a server by a factor of 10. Yet in only 2 years, lag is back. Why? Well, to quote that original devblog, "Here's how I envision this working for a large engagement (say, 1600 or so)". Yesterday, there were over 3400 ships in HED-GP. Even worse, the meta has changed and the type of ships being used likely made the load even worse.

TiDi sped things up by 10x, and EVE players chewed through that in 2 years. While I expect that making the code more efficient will reap great benefits, particularly in chopping the peaks off lag spikes that occur when very "expensive" events occur (such as bridging), I don't think it's going to bring a further 10x improvement. If Team Gridlock proves me wrong, then I will be very impressed and will nominate them for the Galactic Institute's Prize for Extreme Cleverness, but even so, that's just another 2 years or so before EVE players start whining about lag again.

Four years ago, when I first ran for CSM, one of the planks of my manifesto concerned Lag. Here is what I wrote at the time:

While there are clearly many cute hacks that can (and will) reduce lag, the blunt fact of the matter is that such fixes are at best temporary fixes, because as soon as you defeat the lag-monster for N-player battles, the current design of the game encourages bringing extra people to the fight -- which means you have N+500-player battles, and lag returns. In other words, "Fleets expand to fill the lag available". As the EVE population grows, the problem will only get worse.

And it's even worse than what I said back then, because since the introduction of TiDi, the size of "large engagements" has increased much faster than the number of EVE subscribers.

Anyway, the tl/dr is that technical fixes, while wonderful and needed, just address the symptoms; they don't tackle the disease. The disease itself is simple: in EVE, like Soviet Russia, quantity has a quality all its own.

So what game design changes could be made to address this? Figuring that out is why CCP devs get paid the big bucks. But until they do, lag will never go away.