Rogers chief executive officer Tony Staffieri today appeared alongside Ron McKenzie, the company’s newly appointed chief technology officer, before the House of Commons Standing Committee on Industry and Technology to answer questions from regulators regarding the company’s network outage that affected millions of Canadians on July 8.
Staffieri began by outlining four steps Rogers is currently undertaking to improve its network’s resiliency.
- Building an enhanced reliability plan. Rogers is working with the government and competitors on a formal agreement to ensure emergency services stay up at all times.
- Separate wireless and wireline traffic, which Staffieri said could cost around C$250 million. He further said that if the Rogers and Shaw merger deal is approved, this could be completed in half the time.
- Rogers will be investing C$10 billion over the next three years to build out its networks.
- It will also conduct more stringent testing. Additionally, it will conduct a thorough review of the incident with its partners and will share the results with competitors once it’s complete.
On Friday, July 22, Rogers sent a 39-page response to inquiries from the Canadian Radio-television and Telecommunications Commission (CRTC). The report attributed the failure to a deleted routing filter in its core networks that overwhelmed router capacity, rendering them incapable of directing traffic. The full report has been redacted for privacy and competitive reasons.
The error managed to slip through testing. At the hearing, McKenzie described the incident as unprecedented.
“We have extensive–in this particular program over two years–full rigour, full review, full analysis that was done,” said McKenzie. “This part of the implementation was a part of a seven-stage program and this was stage six; the previous five, no issue whatsoever. So this was truly an unprecedented incident.”
The root cause was in the different ways equipment from different vendors processed the update. Rogers uses hardware from a mix of vendors to build its networks, and it didn’t anticipate that the removal of a routing filter would not be handled identically on hardware across manufacturers.
“There was no belief at the time, no information at the time that there [the update deployment] was going to be any issue. And what happened was when the code change was executed–and the filter removed–the behaviour of the equipment in the way it’s designed between one vendor and a second vendor is very different,” explained McKenzie.
Not only did the outage leave millions of Canadians without connectivity, but it also impacted 911 emergency services across the country. This became the key topic of discussion at the hearing. In response, Staffieri said Rogers will have a plan prepared.
“We will have failover measures so that in the event that there is an outage and it doesn’t fall over to our alternate network, it will fall over to one of our competitor’s networks within seconds,” Staffieri said.
The untimely outage occurred while the company is trying to persuade regulators to approve its acquisition of Shaw Communications for C$26 billion. Rogers had already failed to address competition concerns in its mediation with the Canadian Competition Bureau, which is seeking to block the deal.
As a part of its resiliency plan, Rogers wants to separate its wireless and wireline traffic to confine the scale of a similar outage in the future. Staffieri said that this expensive but necessary endeavour could cost around C$250 million, but can be completed in half the time if it’s able to acquire Shaw Communications.
IT World Canada has reached out to Rogers for a detailed explanation but did not receive a reply at the time of publication.
In an earlier portion of the hearing, François-Philippe Champagne, Minister of Innovation, Science and Industry, demanded telecommunication providers arrange an agreement of mutual assistance. Additionally, he asked all telcos to establish a communications protocol with the government and customers when a failure occurs.
The outage caused a major leadership change at Rogers. Last week, Ron McKenzie replaced Jorge Fernandes as the new Rogers chief technology officer, effective immediately. Staffieri expressed his confidence in McKenzie as Rogers’ new CTO at the hearing.
Ownership isn’t the issue
Members of the committee at the hearing also raised the question of whether turning telecommunication into a utility would increase competition. Ian Scott, chairperson of the CRTC, dismissed its relevancy to the matter of understanding the network outage.
“What we’re really looking at is a technical challenge,” said Scott at the hearing. “The challenge is to anticipate these kinds of technical challenges and have a degree of cooperation.”
When asked about what penalties the CRTC has planned for Rogers, Scott said that it remains to be determined, as the investigation had just begun.
“This outage had nothing to do with the ownership, ” said telecom consultant Mark Goldberg in a statement to IT World Canada. “Telecoms are already regulated and the CRTC is examining what happened BEFORE it draws conclusions on whether changes are needed in the regulations.
That seemed to be a novel concept for many of the MPs who squandered their limited time with the witnesses, seemingly less concerned with ‘studying’ the outage (as was the committee’s mandate) in favour of building transcripts for their constituent newsletters.”
Correction: Rogers will be making the $10 billion investment over the next three years and not 10 years as initially reported. The article has been updated to reflect the change.
