The federal government may impose more conditions on telecommunications providers to improve their resiliency as the investigation into the huge July 8 Rogers outage continues, the Industry minister told a House of Commons committee this morning.
François-Philippe Champagne said on learning of the collapse of Rogers’ internet and wireless networks, he demanded the chief executive officers of Rogers, Bell, Telus, Videotron, SaskTel, Eastlink and Shaw come to a formal agreement within 60 days covering mutual emergency network assistance, cross-company 911 roaming, and better communications with the public about network outages. That had been previously announced.
But, Champagne warned, that may not be all.
“This is only a beginning. I will demand even more from them. We will investigate with the CRTC [the federal telecom regulator], there will be recommendations, and following that I will demand additional measures. But for the time being the CEOs of the various companies have agreed with me more has to be done to increase the resilience of these networks.”
“This was a Rogers failure,” he emphasized. “This was a coding error in a routine update … Once we understand better the chain of events and what failed at Rogers we will obviously make sure we take additional steps to close that gap. I have been very clear to the CEOs I will demand more of them.”
Champagne was speaking this morning to the House of Commons Industry committee looking into the unprecedented July 8 outage of Rogers internet and wireless networks, which caused millions of Canadian individuals and businesses to lose connectivity.
Related content: Public version of Rogers explanation has blanks
In a letter to the CRTC, Rogers has promised to increase resiliency in its networks and systems, which will include fully segregating the wireless and wireline core networks. It has hired an external review team to assess and provide insights into the outage. “This will involve a complete evaluation of all our processes, including the performance of network upgrades, disaster recovery procedures, and communication with the public,” Rogers said.
Champagne deflected a question from NDP MP Brian Masse on what penalties Rogers may have to pay from the incident. Masse also demanded a consumer telecom bill of rights. In response, Champagne repeated that the first thing he did on learning of the outage was to “hold them [carriers] to account” and they had to create a formal, binding agreement.
In his testimony Ron McKenzie, Rogers’ new chief technology and information officer, gave more detail on what happened, essentially saying the incident was unexpected and unpredictable.
As outlined in the Rogers CRTC explanation, he said the update to the core network was the sixth phase of a seven-phase process that had begun weeks earlier. The first five phases had proceeded without incident. However, two IP traffic distribution routers from different manufacturers reacted differently to the new code that removed a traffic filter, leaving the routers overwhelmed with traffic.
“There was no way for the engineer [responsible for the change] when it had been performed five times prior with no incident whatsoever, there was no information at the time there was going to be an issue.
“When the code change was executed and the filter removed, the behaviour of the equipment and the way it’s designed between one vendor and a second vendor is very different. That was the unknown at the time: The behaviour of one device of one manufacturer who executes a device one way, versus another manufacturer. It was that sequence that caused essentially the event to flood the core network.”
“This core design of a common IP core is the consistent architecture used by operators all around the world,” he added. “This was an unprecedented event in the way two elements of equipment interacted together that was not known.”