Singtel has refuted reports that a system upgrade carried out on its backbone network caused the massive outage suffered by its Australian subsidiary Optus.
The “routine software upgrade” on a router was carried out at Singtel Internet Exchange (STiX), which is one of Optus’ international networks that facilitates peering connectivity with global internet networks.
Also: The top cloud storage services you can buy
This upgrade was planned and prior notice was sent out to affected customers, including Optus and other telecom companies, Singtel said in a statement Thursday. During the upgrade, data traffic was directed to other points of presence on the STiX network and routed back to customers’ networks.
The software upgrade was completed within 20 minutes and customers’ routers that were connected to it, including Optus’, were up and running, Singtel said.
“We are aware Optus experienced a network outage after the upgrade, when a significant increase in addresses being propagated through their network triggered preset failsafes. However, the upgrade was not the root cause,” said the Singapore telco, of which Optus is a wholly-owned subsidiary.
The November 8 service disruption left many without access to telephone and broadband services, affecting more than 10 million Optus customers, including 400,000 businesses. The hours-long outage also brought down payment systems, grounded train services in the state of Victoria, and left emergency call services inaccessible. Services resumed progressively more than six hours after the outage occurred and were fully restored 12 hours later.
Also: What on earth happened to Cloudflare last week?
The STiX software upgrade was rolled out at 1 am local time, or 4 am in Sydney, on November 8.
In an FAQ explaining the outage, Optus said its network had “received changes to routing information from an international peering network following a software upgrade” at 4:05 am on November 8.
“These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves,” the FAQ said.
Restoring the affected systems involved significant effort from its team, some of whom had to be sent across a number of sites to physically reconnect or reboot routers, according to Optus.
The operator said it has made changes to its network to prevent the issue from recurring.
Also: 10 ways to speed up your internet connection today
Optus CEO Kelly Bayer Rosmarin is scheduled to attend a Senate inquiry on Friday to address the incident.
Australia’s minister for communications Michelle Rowland last week said the government was conducting a post-incident telecommunications review into the network outage. Industry regulator Australian Communications and Media Authority also has initiated its own assessment to determine if Optus had remained in compliance with rules mandating that emergency calls are successfully carried from mobile carriers to the “emergency call person.”
