Two Azure Outages, and Why Known Problems Still Get Through

Azure had two incidents this week. One disrupted Azure OpenAI and Foundry in Sweden Central for about six hours on Monday. The other degraded gateway services (ExpressRoute, VPN Gateway, Application Gateway, Azure Firewall and more) across many regions on Tuesday evening, UTC.

Microsoft has already published a post-incident review of the first one, and it's worth reading in full. It's candid and specific, and it's the kind of write-up most of us only ever get to see from the inside. This issue is my attempt to understand it, not to second-guess the people who ran it.

The Sweden Central chain, as Microsoft describes it

The review covers Sep 29, 10:03 to 15:58 UTC. Paraphrasing the chain it lays out:

1. A deployment cleared caches at peak. A planned deployment of an internal service started around 09:38 UTC and cleared its caches during peak traffic.
2. Two defects sent lookups past the cache. Most lookups went straight to the backend. Microsoft says backend traffic reached about 20 times its normal level by 10:00 UTC.
3. The caching layer saturated, latency climbed, and client retries added more load.
4. Connection exhaustion. Infrastructure nodes ran out of network connection capacity and couldn't open new connections to the cache or the database.
5. Health checks restarted instances. Each instance's health check depended on connecting to the database. Instances that couldn't connect were marked unhealthy and restarted, which left fewer instances to carry the load.

At 15:40 UTC Microsoft changed the health check so it no longer depended on remote connectivity. The restarts stopped and the service recovered by 15:58.

Microsoft also lists what it's changing: the health check stays independent of remote connectivity, the backend gets more throttling and rate limiting, and the caching layer gets better monitoring. The review notes that the existing metrics "did not effectively indicate the connection exhaustion issue."

If you've read about distributed systems, steps 1 to 3 will sound like what's often called a thundering herd: lots of normal requests arriving at once. That's my label; the review doesn't use the term. What strikes me is that none of the steps is exotic. Each is a reasonable behaviour on its own (a deployment, a cache, a retry, a health check), and the outage came from how they combined under peak load.

The gateway incident: too early to say

Tuesday's incident (Sep 30, 20:30 UTC to Oct 1, 02:15 UTC) has much less public detail so far. Microsoft's early explanation, as ITdaily reported on Oct 1, ties it to infrastructure maintenance: "A recent change in the regional gateway management service caused an unexpected load during the maintenance work." Microsoft rolled the change back and promised a full incident report within fourteen days.

I'm not going to guess at the cause. I'll come back to it when the report is out.

Update, after publication: Microsoft's full review of this incident (tracking ID 7Q30-010) is now on the status history page. It says the issue began when a platform-triggered gateway update that had been paused resumed across multiple regions, causing a surge in gateway restarts and configuration-retrieval requests, and that existing throttling and load protection "did not sufficiently protect critical processing." Microsoft says it's changing how paused updates resume so they follow staged rollout practices.

Knowing the problem isn't the same as planning for it

Here's what I take from this. Retries with backoff, throttling, cache warming and careful health checks are well known, and Microsoft documents all of them in its own Azure Architecture Center. Azure had protections in place, too: the reviews mention existing throttling and metrics. They still didn't catch these exact combinations in time.

That's not a knock on Microsoft. It's the honest lesson: even when you know the failure modes, it's hard to plan for every way they can line up in a real system, at peak, during a routine change. Some of them you only find in production. What separates good teams is how fast they recover and how openly they write it up, and these reviews are a good example of that.

Plan for the herd

So the herd isn't something to dismiss as a textbook problem. It's something to design for on purpose, because it shows up in exactly the moments you aren't watching for it: a deploy, a restart, the minute after an outage ends. Here's how I understand the standard defences, and none of them is specific to Azure.

1. Spread out retries. Use exponential backoff with random jitter, so a thousand clients that failed in the same second come back spread over a window instead of together. Cap the number of retries, and stop retrying entirely when a circuit breaker says the dependency is down.

2. Keep the cache warm through changes. Avoid clearing a cache at peak. Pre-warm it before switching traffic, or deploy so old entries stay valid until new ones are filled. When many requests miss the same key at once, let one of them go to the backend and have the rest wait for that answer (often called request coalescing or single-flight).

3. Resume work in stages. A paused rollout or a backlog of queued jobs is a burst waiting to happen. Release it the way you'd release a new deployment: a small batch, check the signals, then widen. That's the change Microsoft says it's making for paused gateway updates.

4. Keep health checks honest. Kubernetes and most platforms separate two questions that a single /health endpoint often blurs:

• Liveness: is this process stuck, so that restarting it would help? Check only the process itself.
• Readiness: should traffic be sent here right now? Dependency checks belong here, because failing readiness takes an instance out of rotation without restarting it.

A liveness check that depends on a remote database restarts healthy instances whenever the database struggles, which is the loop Sweden Central ended up in.

5. Protect the backend from its own clients. Throttle and rate-limit, and when load exceeds what you can serve, reject the excess quickly rather than accepting everything and timing out slowly. A fast "try again later" is cheaper than a slow failure.

6. Watch for the early signs. Alert on the things that move first: cache hit rate dropping, backend request rate jumping several times over, connection pools filling up, retry counts climbing.

7. Test it on purpose. Restart a whole test environment at once, flush its cache, or replay a burst of traffic, and see what happens. The combinations are hard to predict on paper; they're much easier to find in a rehearsal than in production.

What this looks like in my own apps

My backends are tiny next to Azure, but the same ingredients are there:

• Mobile retries. When my API blips, does every phone retry at the same moment? Backoff with jitter belongs in the app's networking layer, not just on the server.
• Push notifications. A notification sent to everyone at once brings everyone back at once. Sending in batches spreads the load.
• Reconnect after an outage. When the backend comes back, every app that was waiting tries to sync. A small random delay before syncing helps.
• My own /health endpoints. Which question is each one answering, and what does the platform do when it fails?

Quick hits

• OpenAI DevDay brings a Decisions API (Sep 29). Among the announcements (Codex in the cloud, always-on "Dots" agents, GPT-6.1 Sol), the one I noticed first: a Decisions API, in limited preview, that picks from predefined answers instead of writing text. OpenAI says it's powered by GPT-6 Luna. Two weeks after Jev launched, a big lab has answered with its own decision layer.
• Android developer verification enforcement begins (Sep 30). In Brazil, Indonesia, Singapore and Thailand, apps on certified Android devices now need to come from a verified developer, with the rest of the world following in 2027. If you sideload or distribute outside Play, check Google's guide now.
• Gemini 4 Argon goes to cyber defenders first (Sep 30). Google announced its new frontier model and gave first access to trusted defenders through its Fairwind Program. Benchmarks are Google's own for now.
• Apple's new EU App Store terms take effect (Oct 1). The per-install Core Technology Fee is replaced by a 5% Core Technology Commission, as announced in August. If you ship in the EU, your next payout statement is the first under the new terms.

What I'm watching

Whether OpenAI's Decisions API leaves limited preview quickly, and what it costs next to Jev.

Sources

The Azure incidents

• Microsoft, Azure status history with the post-incident reviews (tracking IDs 7QL5-Z50, Sep 29, and 7Q30-010, Sep 30, 2026)
• ITdaily, coverage of the multi-region Azure gateway outage (Oct 1, 2026)
• Azure Architecture Center, Health Endpoint Monitoring pattern, Retry pattern and Throttling pattern

Quick hits

• OpenAI, DevDay 2026 recap (Sep 29, 2026)
• OpenAI Developer Community, DevDay 2026 announcements and developer resources (Sep 29, 2026)
• Android Developers, Android developer verification guide
• Google, Gemini 4 Argon announcement (Sep 30, 2026)
• Apple Developer, apps in the EU support page