Massachusetts is a global leader in heavy-duty technology fields such as robotics and artificial intelligence, yet the state government continues to have a nightmarish time running a simple website to let residents easily book COVID-19 vaccine appointments.
The latest and most spectacular failure occurred Thursday morning, when the state’s main vaccine scheduling site repeatedly crashed under heavy traffic from thousands of newly eligible residents chasing the few available appointments. It resumed service later in the day.
The state issued a statement Thursday evening from PrepMod, one of its vendors for the online appointments, in which the company took responsibility for the site crash. “As public health servants and your partner, we are sorry for not meeting expectations,’' the PrepMod statement said. “We accept full responsibility for the problem today.’'
The company blamed the breakdown on “a sudden and unprecedented surge in traffic to the site. Unfortunately, the system did not scale fast enough to accommodate the increased volume.’'
Earlier in the day, computer specialists theorized several reasons for what went wrong.
One is that the state system was cobbled together too quickly to permit adequate testing of software and servers, according to Stuart Madnick, director of the cybersecurity center at the MIT Sloan School of Management.
“A new system is likely to have flaws in it,” Madnick said. “When you try to rush things out, that’s one of the likely outcomes.”
Another possible reason is that state officials simply may not have added enough computer capacity in anticipation of the sudden and large spike in traffic.
“Websites today need to be designed to work with flash crowds,” said Mani Sundaram, chief information officer at Akamai Technologies, a Cambridge company that manages worldwide delivery of Internet data. “A common reason why these sites go down is their existing infrastructure that they’re built on is not designed for these peak loads.”
Akamai, for example, offers government agencies a service to manage vaccine distribution called Vaccine Edge. And as part of its setup process, Akamai runs stress tests designed to simulate a sudden surge of traffic, whether from legitimate vaccine seekers or from bot attacks generated by hackers.
It is unclear if the Baker administration performed a similar trial, though such practices are common in website design. Thursday evening, state officials said they were “pressure testing the site to ensure the vendor’s adjustments are adequate.’'
After a series of hiccups in the morning, the state site was back up and functioning, and the state said 60,000 new appointments were logged by day’s end.
Multiple states and local governments around the country have experienced similar computer problems with their online vaccination websites, many caused by a huge volume of residents chasing a relatively small number of appointments.
The Massachusetts site was so frustrating to some Thursday morning that people were turning to an alternative homemade website developed by a software executive who has been at home on maternity leave.
Olivia Adams said she experienced a surge in traffic on her website, macovidvaccines.com, but some people were unable to use it to make appointments at major vaccination locations like Fenway Park and Gillette Stadium because of the problems with the state system.
Even so, despite the traffic, Adams said, her own website did not crash. She designed it to work off an Internet cloud-based system that has ample capacity for traffic spikes.
“Our site is doing just fine, which is great and to be expected,” Adams said. “We had, like, thousands of requests per minute.”
The Massachusetts vaccine finder site had already come under heavy fire before Thursday’s surge. Many residents complained that it was too clunky, forcing them to navigate among multiple individual providers to find open appointments, most of which required them to fill out lengthy questionnaires before learning if the location even had a slot.
Madnick, for example, experienced those frustrations first-hand before Thursday, when he and his wife labored to book their own appointments. He filled out multiple online applications at CVS, Walgreens, and other providers, only to be told time and again that no doses were available.
“Usually they say, ‘sorry,’ after you’ve typed in four or five pages,” Madnick said. “There’s so many opportunities to do things a heck of a lot better that just weren’t done.”
Multiple state and federal lawmakers from Massachusetts have been pressing Baker to adopt a centralized reservation system. West Virginia, for example, used the Massachusetts technology company Everbridge to build a website that allows residents to quickly register for an appointment and then be notified by the state of their options later.
Now, after the debacle Thursday, Baker said in an interview on GBH radio that the state is considering a system similar to West Virginia’s. But he cautioned it would be more complicated to manage in Massachusetts, which has a much larger population and many more individual vaccine providers.
In the radio interview, Baker did not explain what exactly went wrong with the state site, except to acknowledge his staff had not adequately prepared for the onslaught of traffic. Residents 65 and older or with two qualifying chronic illnesses were first able to book vaccine appointments starting at 8 a.m.
“It’s awful and it’s going to get fixed and I’m going to work very hard to make sure this doesn’t happen again,” Baker said.
Despite the deep history of technology innovation in Massachusetts, the state government itself has had a checkered history of crippling computer woes. Last spring, as the pandemic lockdown caused massive layoffs, the computers at the state’s unemployment insurance system buckled under the strain. And in 2013, the launch of the Massachusetts Health Connector website, where citizens could apply for health insurance, was plagued with technical glitches that prevented thousands of people from signing up.
One quirk developed in Rhode Island’s system this week, when residents reported the state’s website did not have a way to verify that applicants meet the criteria for a vaccine — in this case, proving they were 75 or older. Several residents under that age reported they were able to schedule an appointment.
The latest computer problems in Massachusetts will be a central part of an oversight investigation that state legislators are conducting of the administration’s various stumbles on the vaccine front.
“We need to learn more about the complexities,” said Representative Bill Driscoll, the House chairman of the Legislature’s committee on COVID-19 preparedness, which already had scheduled a hearing for Feb. 25 to address public complaints that the website was too difficult to use. “We want to get to the bottom of why Massachusetts is so seemingly underprepared.”