Monday, January 11, 2010

Single Server Blues

Over at Stabbed Up blog  the author, Stabs , has speculated that Eve's recent lag issues afflicting large null sec fleet fights is a symptom of a growing population base and he feels the only answer is to create more servers.
Now bugs can be fixed of course and more computing capacity can be purchased. But there's a
limit to how far and how fast Eve can grow. They already run the game on one of the most sophisticated supercomputers in the world.
So if Eve will keep growing both in player numbers and in graphical complexity then a second server becomes inevitable. And it's not like there isn't some precedent: the Chinese play Eve on a separate server and there is also a separate Test server.


After almost every major Eve expansion (about two a year) there is unintended consequences due to the code changes. Sometimes the new bugs are minor or very hard to track down like the desync issues of early last year, or major issues like the gate jumping CTD bug of late last year. I know in a perfect world that Quality Assurance teams would catch these bugs but in a code base as big and complex as Eve, every change can potentially spawn many other unintended issues, and if QA catches and helps kill 99 out of 100 of them its the last one we see and grumble about.

So bugs happen. In Eve they usually manifest as large groups of players congregate, one of the use cases that is hard to reproduce in a lab. Thus fleet fights, where a lot of people are doing lots of things, tend to be the place that players notice the bugs the most and feel the effects the worst when they are on the losing side of a black loading screen.

So is another server to reduce population density the answer to these lag issues? I don't think so.

First off, the Eve server is actually a massive cluster of servers that allow player movement between them. Every one of Eve's 7500+ solar systems is called a Node and each server can support 1 to X nodes depending on how populated that node is or is expected to be. This means that while a second Eve server cluster would reduce overall player density, it would not prevent local node density from climbing since players can freely move from node to node. A lower population density might reduce the number of occurrences of large player congregations for the purposes of fleet fights or trade, but they would still happen and would still suck.

EDIT: As several people have pointed out, I got my terminology all mixed up. See James Harrison's excellent post here to get it right. However, the concept is correct: a system cannot be broken up across nodes and thus limits the workload sharing.

Secondly, the downsides of splitting the Eve community would be severe.

- The new server would be starting from scratch with no established economy and no working supply chains to provide the equipment for players. It would be like living in a third world country compared to the rich economy of the current Eve server (Tranquility) and it would take months of skill training and player base growth before the basic supply chains would exist, and it runs the possibility of never matching the rich and deep economy of the original anyways.

- The Chinese server, Serenity, averages around 4100 players at peak times which is less than 10% of Tranquility's peak players. The systems of Serenity are barren, the economy is weak and bare bones in the trade hubs, and the combat for the most part small skirmishes compared to Tranquility's fleet fights. If the effort is made to create a second Eve cluster it runs the risk of never reaching the critical mass to foster sustained growth, especially when in direct competition with Tranquility.*

- The splintering of the player base should the second server succeed against all odds would create some of the issues many other MMOs face such as lack of easy grouping of friends across servers, less importance of in game happenings ("that fight didn't occur on my server so I don't care"), less importance of player identities, forcing corporations that provide goods or services to either expand on the second server or ignore a potential customer base.

Those are just the downsides that I thought of. I'm sure there are many more.

In the end I believe that the current fleet battle lag issues will be resolved by CCP like they have done for previous issues, and in the meantime the player base will adjust and adapt their tactics to the new reality.

Does Eve's "server" have an upper limit? Perhaps but I would rather add more systems to the current cluster in the form of more null sec space, or more resources in wormhole space and allow outposts to be built there to allow players to live there truly, before creating a copy of the current cluster.

Time will tell.

* -I don't consider the Chinese server Serenity a true second server because the rest of the world doesn't get a choice to use it and Chinese citizens don't have a choice to choose Tranquility over it (in theory). Its a unique situation imposed by Chinese government restrictions and serves best as an example of how a new Eve cluster might develop.


  1. Very interesting post!

    I was online yesterday for most of the day, and while I have no idea how things went in deep 0.0 (read 49-U6U) I was out and about flying in Heimatar, and I had never seen it this crowded. But frankly, the only thing I found that was lagging was the IGB which was just a bit slower than usual but not by much. I even pushed my luck and opened up my scanner in the middle of a mission hub; the amount of wrecks was hilarious!

    About adding systems, I also believe that it would be the way to go; maybe one region for each, hi-sec, low-sec, null-sec and W-space? But I can see where CCP is iffy on adding stuff to W-space that would allow for permanent residence; there would be a huge gold rush and systems would get depleted of their ressources in no time. It would make T3 prices come down and create an entirely new form of Sov warfare though.

    But I'm an optimist... I say we'll break 60000 before my birthday in July :)

  2. Anonymous9:14 am

    I agree with you, and I'll add a thought of my own to the argument.

    If a second server was created, would it be all new equipment in addition to the current server? I doubt it. The point of a second server is to reduce the load on the current server, so there'd be a way for players to move to the new server. But I think that CCP would also move equipment out of the current cluster to serve in the new server, so there'd be fewer resources for the current server. Or maybe they'd have a way to take an element in the cluster and move it from one server to the other dynamically as needed. Either way, the business drivers suggest you'd still have only so much computing resources per 1000 players either spread over 2 servers or just on one.

    If the current resources per 1000 players isn't enough, then more needs to be added. I just don't see how splitting the existing resources over 2 servers will fix the issue.

    And since the current lag problems came with Dominion (AFAIK), it's far more likely that there's a software problem in the new expansion, and proper fix is a software solution that problem.

  3. Totally agree with you.

    Creating a new server would be the WORST thing to do to EVE. It is the single server community that makes EVE so incredibly unique.. if they took that away... well... much of what I like about EVE would be gone.

    Your quoted post is what happens when people play WoW for too long and think the solution to every problem is to do it the way they do it.

  4. Splitting into multiple servers would be like selling EVE's soul. They would also have to delete that Butterfly effect promo video!

  5. As a minor nitpick, you've confused the terminology. The individual servers in tranquility (the separate machines) are called Nodes, each node can host multiple in-game systems (which are still called systems in this context).

    Other than that, 100% spot on.

    I have to say though, that I think you glossed over the biggest reason the addition of another server is so farcical. Adding more servers (even with split populations) doesn't do anything to reduce the scale fleet fights can reach at their peak. If 800 people get together in one system (and hence, on one node) then lag is going to be a major issue. Always. Adding more servers does nothing to change this. The fact that EVE's cluster is [b]already[/b] a collection of nodes should make this fact patently obvious. You [b]cannot[/b] slay the lag monster with additional hardware.

    As it stands, the maximum number of nodes a single system can be on is one. Which means that there is always a finite amount of computing hardware that can be dedicated to a system.

    At least, not with the way the Eve's networking is currently designed.

    CCP has mentioned in previous dev blogs that they are experimenting and thinking about how they could rewrite the network code for Eve from the ground up. If they do, they could (and want) to rewrite the code to allow multiple nodes to host a single system (one node per grid, perhaps?). Of course, such a fundamental change would represent a gigantic undertaking at this point, and would be fraught with technical peril.

    If they pulled it off though, then the rules by which CCP struggles against lag would be completely rewritten. Dynamic allocation of nodes to various systems, rather than allocation of systems to nodes, would allow for some truly terrifying fleet fights.

    Until the user base grows so big that people are bitching about how their 2000 person fight the other night was a slide show, anyway.

    Keep up the awesome blog, as usual.

  6. Three years ago 50 versus 50 was a lagfest. 2 years ago 150 versus 150 was near impossible. 1 year ago 400 versus 400 was doomed to node death. No matter how much the lag dragon is slayed, players will make a new one. :)

  7. Az has some really good points, but I don't agree on them all. Cloud computing is where the industry is going and makes sense. As the virtual population of these different games grow and our links from our home improve the load on these clusters would increase. Being able to dynamically add more processing power (CPU and memory) to a node "on the fly" would be nice, but not even close.

    Re-writing or cleaning up of the code is definitely a big undertaking. The problem is that is not only the more difficult, but can bring the biggest issues as new bugs unfortunately will be introduced.

    This leaves us with the good old "slap more hardware" approach. As pointed before, each physical box can host multiple systems. I would love to see them moving these system in conflicts to their own servers with more memory, CPU and fatter ethernet connections.

    10GE nics have been around for over a year already. Intel and others keep pushing the limit on their hw (there will be 6-core and 8-core CPUs before the end of the year) and we have even seen a single compute blade that can hold up to 384GB of RAM.

    All sw companies eventually go thru a "code cleaning" process, but in the mean time... throw hw at it! :)

  8. I don't really understand what you're saying with regard to cloud computing.

    Part of the point was, however, that we're already at the limit of what adding more hardware can do under the current network code.

    As anyone who has participated in a fleet fight even on a reinforced node (that is, in a system that has been moved to be alone on 1 node, which will be from the 50 or so shiny new ones they got half a year ago or so (I don't remember the number)) before Dominion, large fights (600+) could actually run fairly well. Laggy, but functional. After Dominion things have been going to shit, clearly there is some bug somewhere. (and hence the testing on SISI)

    But higher than that (800+), both before and after Dominion, a single cutting-edge node simply cannot handle running the system. Upgrading to shinier and newer nodes, with faster CPU's and more RAM, would marginally increase this number (900+, 950+, 1000+?) but does not come anywhere near the kind of gains that could be seen were the code magically rewritten tomorrow to allow multiple nodes per system. (I feel a jump from 600 to 900(playable) would be reasonable, if not conservative.)

    Also, I didn't mean to make rewriting the network code sound trivial. You're quite right that even after it was brought to an initial production release we'd prolly be looking at 6 months or more of terrible bugs. But such issues are constantly occuring, and I don't see it as much of a downside as compared to the current situation.

    Also, with regard to the actual hardware. IIRC, CCP doesn't use NICs to interface their blades. They're actually hooked up directly with a really bloody amazing serial cable. The kinda stuff that can drop hundreds of gigabit/s without being saturated. They had a dev blog about them a year or two ago, when they were new.

    We're at the limit where CCP faces exponential costs for ever better individual nodes, for very little return in real performance. The limit is, and will continue to be the software, and they've acknowledged this themselves. It's just not a problem that can be fixed tomorrow. *shrugs*

  9. I would have to agree, post a lot of waffle and wind on a couple blogs that linked here agreeing as well. The Single 'server' is what makes Eve the ever changing rich environ that keeps people coming back for more. great blog BTW, need to read it outside Capsuleer more often.

  10. Actually I would like to point out:

    "* -I don't consider the Chinese server Serenity a true second server because the rest of the world doesn't get a choice to use it and Chinese citizens don't have a choice to choose Tranquility over it (in theory). Its a unique situation imposed by Chinese government restrictions and serves best as an example of how a new Eve cluster might develop."

    I was in China earlier, last summer, (mainland and Hong Kong) and I must say I was playing on Tranquility.

    I had no clue how to connect to Serenity but as soon as I booted up my computer and ran EVE it connected to Tranquility, I didn't even have a choice to connect to Serenity?

    I am not sure what is suppose to happen but that's my 2 ISK.

    Still good point though, no sharding!