Rail outage that stranded thousands in Toronto area caused by software upgrade: CN

The massive outage on Canadian National Railway Co. lines that delayed thousands of Toronto-area commuters during the evening rush hour Tuesday can be traced to a software upgrade, the company says.

“At this time, our teams have determined that this was caused by an internal systems upgrade, which affected CN’s ability to connect to the internet,” CN spokesman Jonathan Abecassis said in an interview.

“There’s no indication of a cybersecurity incident whatsoever. At no time was the safety of the public compromised, and at no time was data impacted.”

The digital blackout “made it impossible for CN to connect to the internet,” he clarified. It also barred regional transport authority Metrolinx and its GO Transit trains from connecting to CN servers — essential to digital communications that direct locomotives where to move — Abecassis said. GO Transit trains run partly on CN-owned tracks.

Partial connectivity was restored at about 3:45 p.m., as some trains lurched back into motion hours behind schedule after the outage began at 12:30 p.m. System recovery was nearly complete by 8 p.m., and fully resolved overnight, Abecassis said.

Earlier, the computer malfunction forced GO Transit commuter trains in the Greater Toronto Area to stop at the nearest station, while riders crowded the city’s Union Station downtown during rush hour in hopes the network would resume.

The rail link connecting the downtown core to Toronto’s Pearson airport also stalled, while more than 30 Via Rail arrivals and departures at Union Station were late — 11 by more than two hours, including busy routes to or from Montreal, Ottawa and Windsor, Ont. — according to data compiled by Greg Gormick, who heads On Track Consulting.

No CN freight trains experienced major delays, while EXO commuter trains in the Montreal area and Amtrak passenger trains from the U.S. were also in operation, Abecassis said.

After CN’s computerized dispatch system went down, a handful of rail traffic controllers began issuing orders by pen and paper that were then sent out to conductors via radio to move trains to stations or other safe areas of track, where they were halted, according to two railway sources who spoke on condition of anonymity because they were not authorized to speak on the record.

EXO trains are dispatched directly by CN, which was able to revert to manual techniques to keep them moving, Abecassis said.

During the outage, CN worked with GO Transit to temporarily take over their train dispatching responsibilities, CN said. “This allowed for the partial resumption of GO and Via services” starting in late afternoon.

“We’re going to be working with Metrolinx to put in place processes to avoid this happening again,” Abecassis said.