Metro

Search underway for root cause of commuter rail woes

A workman used hand signals for the arrival of a train at South Station Thursday.

David L Ryan/Globe Staff

A workman used hand signals for the arrival of a train at South Station Thursday.

The fix came in at 4:50 a.m. Friday.

Twenty-four hours after a computer outage paralyzed South Station and inconvenienced tens of thousands of Boston-area passengers, commuter trains were back to normal Friday morning.

Advertisement

Paul F. O’Mara, superintendent of operations for Amtrak, said the computer system controlling switches that move trains onto the tracks at South Station had failed Thursday.

The breakdown meant crews had to manipulate the switches manually, a technique that has not been the norm at busy train terminals since the 1940s, O’Mara said. He described the computer problem as unprecedented.

Get Fast Forward in your inbox:
Forget yesterday's news. Get what you need today in this early-morning email.
Thank you for signing up! Sign up for more newsletters here

But even as regular train service resumed Friday morning, Amtrak officials said they did not know why the computer system went down.

“Our personnel are still out there,” O’Mara said in an interview at South Station. “They’re still going over what the root cause of this was . . . and what we can do in the future to minimize the impact.”

David L. Ryan/globe staff

Paul O’Mara, superintendent of operations at Amtrak.

The computer system is used by rail dispatchers, who send electronic commands to the 63 switches on the tracks at South Station. The collection of switches, where trains change from track to track, is called an “interlocking” and is located just outside of the station, Amtrak said.

Advertisement

Commands for the switches are routed through “Tower 1,” a bungalow the size of an office trailer that houses thousands of wires, relays, control boards, and other electronic components, according to Amtrak.

When the system went down, dispatchers who work in a control center on the fifth floor of South Station, could no longer manipulate switches that they normally maneuver remotely using a computer screen, Amtrak officials said.

The outage also made it impossible for dispatchers to see the trains in the interlocking area where they change from track to track, O’Mara said.

Amtrak sent crews into the field to align the tracks manually using large wrenches, and extra personnel were summoned from Pennsylvania and Rhode Island to assist, O’Mara said.

To fix the problem, Amtrak said it replaced software controlling the electronic switches. The repair was made by bringing in a replacement computer system from Amtrak’s main engineering office in Philadelphia and reinstalling the software, said Mike Tolbert, a spokesman for the rail service.

Finding the solution took multiple tries Thursday and the early hours of Friday as Amtrak engineers experimented with a variety of fixes, including replacing software on computers in Tower 1, replacing electrical components, and changing power supply units, Tolbert said in an e-mail. Hundreds of components were switched out, replaced, and checked, he said.

“It wasn’t a case that we knew one particular thing broke and we know, ‘OK. We can go get a part and we can fix it, and we think it’s going to take a couple hours,’ ” O’Mara said. “It was actually just troubleshooting for most of the day [Thursday] and bringing in more resources to try to determine what the cause was.”

While engineers searched for a solution, other workers devised a temporary plan to restore train service to South Station in case the computer system could not be restored in time for the Friday morning commute, Amtrak said.

That proposal would not have eliminated train delays, but would have cut wait times to 15 to 20 minutes, as opposed to the one- to two-hour lags some passengers faced Thursday, O’Mara said.

Steve Sherman, who works at the souvenir and sports memorabilia shop Teddy Ballgame’s in the station, said it took him two hours Thursday to get to Boston from Canton.

“It was a nightmare,” he said. “What’s wrong with these rails?”

Transportation officials warned commuters Thursday evening that the problems would likely persist Friday.

But soon after the first commuter rail train bound for Boston left Forge Park/495 station in Franklin at 5:05 a.m. Friday, news surfaced that Amtrak had fixed the computer system.

“It was just a huge sigh of relief for everyone,” said Leslie Aun, a spokeswoman for Keolis Commuter Services, which runs commuter trains for the Massachusetts Bay Transportation Authority.

At 5:51 a.m., the commuter rail published a tweet reporting the problem was solved.

“I’ve never had a tweet ‘liked’ by so many people so quickly,” said Aun as she stood under the train schedule board in South Station, which was aglow with “on time” notices.

Kimberly Mello said she waited at JFK/UMass Station Thursday evening for a train to Brockton that never showed. Instead, she said she jumped on a Kingston-bound train and got off in Abington, where her daughter picked her up.

But Friday morning, Mello said she resumed her normal commute on the Middleborough-Lakeville Line. Her fellow passengers, Mello said, were happy for a hassle-free commute.

“We were clapping when they said you could go all the way to South Station,” she said.

Steve Annear of the Globe staff contributed to this report. Laura Crimaldi can be reached at laura.crimaldi@globe.com. Follow her on Twitter @lauracrimaldi.
Loading comments...
Real journalists. Real journalism. Subscribe to The Boston Globe today.