›bobbyencoded_
/posts/newsletter/about
/posts/newsletter/about

The Decode

The week's AI & software news, decoded.

Subscribe →

// © 2026 BobbyEncoded Inc. — A Canadian corporation

← Back to Blog

The Decode #017

Why One Zone's Power Cut Reached a Whole Region: Google's Report

October 30, 2026 · 7 min read

The Decode, Google Cloud, Reliability, Distributed Systems, GitHub, Copilot, AI Agents

Last week I ended with one question: when a single zone in Google Cloud's London region lost power, why did services built to survive a zone failure show errors too? On Tuesday Google published its incident report, and it answers that question directly. Issue #016 has the background.

Why the power went out

The report walks through the chain, all times US/Pacific on Monday, October 19:

  • 22:53. An underground cable fault cut utility feed A to the europe-west2-b data centre. The site has a second feed, feed B, but it had been opened for repairs, so feed A was carrying the whole load.
  • The generators didn't start. Google attributes this to "a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position." In plain terms, the controller that decides when to start the generators didn't correctly see that the building had lost its supply.
  • 23:00. The UPS batteries, which bridge the gap until generators take over, ran out, and the data halls lost power. Monitoring alerted engineers at 23:04, and by 23:09 traffic was failing over to other zones.
  • 23:54. Utility power came back to the site, and data hall power at 00:20. Then came the slow part: bringing clusters, network and storage back in order, and letting traffic return to the zone gradually from 03:45.
  • 05:08. Most workloads had recovered. Google puts the total impact at 6 hours and 8 minutes.

Each of those safeguards is standard: two utility feeds, generators, batteries. What made this incident is that one feed was out for maintenance at the moment the other failed, and the logic meant to cover exactly that case had a gap. It's a reminder that the rare failures are usually combinations of normal states, not one exotic event.

Why the rest of the region felt it

This is the part I was waiting for. Google says the outage was physically confined to one zone, but "there were a set of unexpected failures which caused impact outside of the zonal failure domain." It names two:

  1. "Instantaneous demand surges on neighboring zones (a and c)." When zone b went dark, everything designed to fail over did, all at once, onto the two remaining zones.
  2. "Control-plane dependencies across zonal boundaries." Some of the machinery that manages services in zones a and c still depended on something in zone b.

Google says the failover traffic "caused transient latency and elevated error rates across the broader europe-west2 region," and that storage writes in the region saw a transient error rate during the initial failover. The report calls it "a degraded state" for the region rather than a regional outage.

Both causes are well known in distributed systems, and Google's engineers know them better than almost anyone. That's what I find useful about the report. Knowing a failure mode exists isn't the same as having every path covered at the moment a rare power failure lines up with a maintenance window. Google's remediation list reflects that: fix the PLC logic and check it at similar sites, freeze planned maintenance at sites with the same layout, rehearse emergency procedures, and keep improving zonal failover and regional disk recovery.

What it changes in my own planning

Last week's checklist still stands. The report adds two things I'll plan for:

  1. Expect the healthy zones to be busy. If my backend survives a zone loss by moving to the others, so does everyone else's. Capacity that's already running beats capacity I'd have to create in the middle of an incident, which echoes Google's own "may not be able to yet create new workloads" warning on the day.
  2. The app should handle "degraded", not just "down". A regional slowdown with some failing writes is harder for an app than a clean outage. Timeouts, retries with backoff, and making sure a failed write is retried safely (without creating duplicates) matter more than a pretty offline screen.

Quick hits

  • GitHub Copilot stops asking you to pick a model (Universe, Oct 28). Copilot Fusion is now generally available on every Copilot plan in VS Code, JetBrains, the Copilot app and the CLI. You choose an effort level (Standard, High or Max) and Fusion routes each task: one model directly, a cheaper model with escalation if a quality check fails, or a draft-review-revise loop between two models. You're billed for each model it uses, with a temporary 30% discount on paid plans. GitHub says Fusion Max scored "within two points of Claude Code with Opus 5.5 Max" on an Artificial Analysis coding index at about half the cost per task. That's GitHub's own claim, so read it as a vendor benchmark.
  • Agents get a supported home in GitHub Actions (Oct 28). GitHub Agentic Workflows is now GA on Free, Team and Enterprise. You write a workflow in Markdown, it compiles to Actions YAML, and it runs on Copilot or your own coding agent. The defaults are what I'd want: read-only permissions, a sandboxed container behind a firewall, and a threat-detection job that scans every proposed change before it's applied. Issue triage and CI failure analysis are the obvious first uses.
  • Apple's first touchscreen Mac (Oct 27–28). The new 14- and 16-inch MacBook Pro with a "PROLED XDR" display supports touch, on M5 Pro and M5 Max. Apple says existing apps work because macOS turns touches into clicks, and "updated system gesture APIs" let developers design for touch directly. A base 14-inch MacBook Pro with the new M6 chip arrived a day earlier; Apple claims Xcode builds up to 1.3x faster than M5. If you ship a Mac or Catalyst app, it's time to check how it feels under a finger.
  • Google Play's October policy update (Oct 29). New Local Network and Manage Contacts permission policies need Play Console declarations, there are rules for the AppFunctions API (apps must not "deceive or manipulate AI agents"), and Android 17's location button becomes mandatory on Jan 27, 2027. Most changes give at least 30 days to comply.
  • Anthropic files publicly for its IPO (Oct 26). It plans to list on Nasdaq under the ticker "ANT"; the share count and price range "have not yet been determined."

What I'm watching

November 10. .NET 11 launches at .NET Conf, and on the same day .NET 8 and .NET 9 both reach end of support. If you have services on either, the upgrade clock starts now.

Sources

Google Cloud europe-west2 incident report

  • Google Cloud Service Health, Incident Report: multiple services in europe-west2 (full report Oct 27, 2026)
  • Computer Weekly, Google Cloud power event knocks out UK banks, courts and couriers

Quick hits

  • GitHub Changelog, Route and orchestrate models with GitHub Copilot Fusion (Oct 28, 2026)
  • GitHub Changelog, GitHub Agentic Workflows is generally available (Oct 28, 2026)
  • VentureBeat, GitHub Copilot Fusion mixes models to achieve near-Opus 5.5 coding performance at half the per-task cost
  • Apple Newsroom, Apple unveils MacBook Pro with PROLED XDR display (Oct 28, 2026)
  • Apple Newsroom, Apple introduces the new 14-inch MacBook Pro with M6 (Oct 27, 2026)
  • MacRumors, Apple announces new MacBook Pro (Oct 28, 2026)
  • Play Console Help, October 2026 policy update (Oct 29, 2026)
  • Anthropic, Anthropic files registration statement (Oct 26, 2026)
  • TechCrunch, Anthropic officially files for its public debut (Oct 26, 2026)
  • Microsoft, .NET and .NET Core support policy · .NET Conf 2026
← Previous

More Code, Not More Software

Next →

Spec-Driven Development with Claude Code: One Setup for Swift, Python, .NET and Android

STAY UPDATED

Get new posts on software engineering and AI in your inbox. No spam, unsubscribe anytime.