Streaming a World Cup game? You can thank these engineers in Cambridge
You couldn't find a better spot for a World Cup viewing party.
In an operating center near Kendall Square, dozens of screens showed games as they unfolded from myriad perspectives. Smaller panels flicked between camera angles: bird's-eye views, sideline shots, close-ups of star players.
But nobody seemed to care so much about the action on the field. For the engineers in Akamai Technologies' Broadcast Operations Control Center, victory meant ensuring the uninterrupted flow of millions of data streams bringing World Cup action to computers, tablets, and phones around the globe.
"We're sitting at the epicenter of all that; it's really exciting and rewarding work," said Campbell Foster, a vice president at the Cambridge company. "It's fun. It's live sports."
Live sports are an increasing area of focus for Akamai, which carries data across the Internet for some of the world's largest tech and media companies, including Fox Sports, which is showing the World Cup in the United States.
Akamai recently cited large sporting events as one of the factors behind a first-quarter increase in revenue for the division that handles streaming.
And that was before the World Cup. With a few weeks to go, soccer's worldwide championship is already the biggest sporting event Akamai has ever carried, in terms of data.
During the June 26 game between Germany and Sweden, Akamai saw a peak of 15.5 terabits per second in game feeds traverse its network. That's enough data to download about 9,688 entire high-definition movies in one second, according to the company.
It's also one of the largest yet in terms of the number of people streaming videos over the network, with close to 7 million users per hour for some games.
World Cup streams, broadcast by Fox and other outlets, travel across the Internet on Akamai's content delivery network, which looks for efficient ways to route data from the cameras and microphones in Russia to users' computers and mobile devices.
The process relies on hundreds of thousands of Akamai servers around the world that relay data to fans. The sheer complexity of the network and the volume of data mean there are plenty of opportunities for problems to arise, slowing or blocking the data flow and frustrating viewers. So some clients pay Akamai extra to keep an eye on its feeds from Cambridge, spotting and trouble-shooting problems when they emerge.
As Mexico battled Sweden on Wednesday and Germany took on South Korea, Mike Shea sat in front of the control center televisions at a desk that held four computer screens.
He was also monitoring a feed of one of the games on a tablet, and talking on the phone to a team of colleagues around the world monitoring the feed. They call this process "eyes on glass."
Workers at the center can diagnose problems by looking at data on the network, examining code, and talking over issues with staff at the scene. They flag problems for Akamai engineers who try to fix them before they become widespread.
Sometimes, the broadcast center will spot a problem that has nothing to do with Akamai's system. For example, an overhead angle had gone out on one of the television screens shortly before, and Shea had flagged it; it turned out to be a problem with one of the cameras.
A few days earlier, the equipment that a broadcaster was using to feed data into the network experienced an outage. Shea had an eye out to make sure it wouldn't happen again.
One of his screens showed a stream of code on a white screen, with new entries appearing every six seconds to indicate the game feeds were getting into Akamai's system. If it stopped, that would be a problem. But the code kept running as Shea kept watch.
"If we can't ingest it, we can't deliver it," he said. Streams go dark, he said, and videos begin buffering. "That means angry people."