Google Cloud outage in London: a controller fault kept the backup generators off
Google has published its incident report on the October 20 Google Cloud outage in London, and it says the backup generators never started because their controller was misconfigured. That turned a cable fault on the local power network into a disruption of more than 6 hours, and the report explains how a failure in one zone spread across the whole europe-west2 region.
What happened
At 6:53 a.m. BST on Tuesday, October 20, the utility feed powering a data center that hosts europe-west2-b, one of the zones in Google's London cloud region, went down. Google says an underground cable fault on the utility network outside the building caused it.
The site was already running on a single feed. Google says one of its two utility feeds had been switched off for repairs after problems showed up in a quarterly generator load test on October 14. That left the facility depending on the remaining feed and its backup generators.
The generators should have started within the roughly 5 minutes of runtime the site's UPS batteries provide. They didn't. According to the report, the programmable logic controller (PLC) that runs the generator plant had its voltage sensors wired upstream of the feed breakers. It still saw live voltage on the disconnected feed, concluded that a healthy utility source was available and never sent a start signal. Google calls this a "voltage-sensing design flaw and PLC misconfiguration."
The batteries ran out and servers in the zone lost power at 7:00 a.m. BST. The controller also blocked the on-site team from transferring the load remotely, so the team switched the building to a standby utility connection. Power was back in the data halls by 8:20 a.m.
Why recovery took 6 hours
Restoring power didn't bring services back right away. Because the servers had shut down without warning, Google says its engineers held back services while a large number of storage devices recovered, to avoid making things worse. Google Cloud services and APIs were fully available again at 12:11 p.m. BST, though some Compute Engine virtual machines were still down. By 1:08 p.m., 99.5 percent of Compute Engine virtual machines were healthy. Google puts the total disruption at 6 hours and 8 minutes.
The outage also spread beyond the failed zone. Google says customers in the region's other 2 zones, europe-west2-a and europe-west2-c, saw errors and delays because traffic failing over from the dead zone flooded their capacity, and because some regional services depended on components in the offline zone. Some customers' failover attempts for virtual machines also failed because of capacity limits and difficulty replicating the underlying storage.
The Register reports that the outage disrupted services at Lloyds, HSBC, Royal Mail and Revolut.
What Google is changing
Google says it reconfigured and validated the controller on October 20 and again on October 26, and it lists these follow-up steps:
- A full review of the PLC code with the facility vendor, due by November 6
- Checks of PLC configurations at other sites with a similar design and the same PLC makers, due by November 15
- A freeze on planned maintenance at similar sites, which Google says took effect October 30
- Repair of the damaged utility cable, expected by November 13, then a return to both feeds and new generator tests by November 17
- Automated alerts so customers hear about incidents that hit a whole location faster
What it means for you
If you only use apps and services built on Google Cloud, there's nothing to do. This report explains the October disruption, and the fixes are on Google's side.
If you run workloads in europe-west2, read the per-product impact section of the report. Google notes that zonal resources such as Compute Engine VMs need failover that customers set up themselves, and this incident shows that a second zone in the same region can be swamped when everyone fails over at once. If your service can't afford downtime, a backup in another region avoids depending on zones that share the same capacity.