As games approach the age of ubiquitous connectivity, a developer at a pioneering online studio warns that live launch problems are by nature difficult to anticipate.
Blizzard Entertainment lead producer Alex Mayberry, who in 2012 was engulfed by outrage when gamers couldn't play Diablo III at launch due to the infamous Error 37, says such problems border on the inevitable.
"In spite of our very ambitious projections, even more people showed up for Diablo 3," he told CVG in an interview on Monday.
"We doubled and quadrupled our estimates on how many people were going to show up, and it turned out that the hardware was just overwhelmed. It certainly didn't feel good when that happened."
Blizzard spent weeks rebuilding its network infrastructure to ensure customers could start playing Diablo 3, but the game's reputation had already been tarnished by then.
"We doubled and quadrupled our estimates on how many people are going to show up"
But the legacy of Error 37 was that it became a foretoken of problems that lay ahead in the online age. In 2013 alone, the live launches of SimCity, GTA Online and Battlefield 4 were widely considered disasters.
"I was trying to play SimCity at the time, they definitely had my sympathy. I totally understood what they were going through," Mayberry said.
But, he added, much of the problem lies in the unpredictability of human nature, multiplied by the millions of people who are playing the game.
"It's not an 'x plus y equals' problem. It's not as easy as knowing 14 million people have bought this game and thus we need server capacity for all of them. What if all people enter the same part of the game at the exact same time? There are certain behaviours that we cannot anticipate or prepare for."
Though Mayberry did say measures can be taken to anticipate sheer volume, there's only so much a games company can prepare for.
"There are some people who are just incredible in understanding how systems work, and they suddenly figure out exploits. What happens when one person discovers the easiest way to get a sought-after item, then that goes out on the internet, and then you have millions of people doing the same thing? Well, our servers may not be ready for that trend."
Asked if games studios can never be fully prepared for live launches, Mayberry replied "that's right".
"I would love to ship a bug-free game, but if we've never encountered certain behaviour, how can we prepare for it?"
The great unknown
Ultimately, he said, venturing into uncharted territories of online systems inevitably means going in blind.
"These things happen because we're doing incredible new things with technology that never happened before. We're inventing things, inventing new technologies, we're writing code that's never been done before.
"So, every time you do that, you're just going to have issues. It may not be the code, or the equipment, or even the service IPs that you're using - you have to account for all the potential problems and try to mitigate risk."
"These things happen because we're doing incredible new things with technology that never happened before"
The lead producer's comments were made hours ahead of the launch of the Diablo 3 expansion, Reaper of Souls, of which there appears to be no major technical online issues.
Mayberry said expansions, while not straightforward either, were easier to prepare for.
"It's much easier to plan this out. Diablo 3 patch 2.01 went out recently, and it contains a lot of Reaper of Souls code and data, so the launch process is akin to flipping a switch. It's been live for several weeks, and we've been patching other stuff in there already.
"We have a team of people who will be living at the office for the next seven days, ready to respond to any situation."