Google Cloud London Outage (europe-west2): What Failed

What Happened

Everything in this section comes from Google's preliminary incident report. Times are London time (BST).

The report lists 40 affected Google Cloud services, including Compute Engine, Cloud Run, Cloud SQL, Google Kubernetes Engine, Cloud Storage, BigQuery and Cloud Load Balancing. Google says a final report with preventive actions will follow.

Power cuts happen, which is why data centres have batteries and generators. Here the batteries worked, and the generators did not start. The report says both the primary and the backup controllers failed to start them.

Why the Other Two Zones Also Struggled

A cloud region is split into zones, which are separate data centre facilities. The idea is that if one zone fails, the others carry on. That mostly happened: Google says services with failover, such as Cloud SQL, AlloyDB, Memorystore for Redis and Cloud Bigtable, failed over into the zones that still had power.

But customers in europe-west2-a and europe-west2-c also had problems. Google's report gives two causes:

1. A rush of demand. Everything that failed over from the dead zone landed on the remaining capacity at once and filled it. That caused queues, slower database failovers, and intermittent HTTP 500 and 503 errors across the region.
2. Hidden links to the dead zone. Some control systems in the region still waited for answers from components in the offline zone, and timed out. Google lists BigQuery, Looker, Compute Engine, Google Kubernetes Engine, Regional Persistent Disk and Cloud SQL among the services hit this way.

The second cause is the uncomfortable one. Being "multi-zone" on paper did not fully protect those customers, because part of the system they relied on still depended on the zone that went dark.

How Google Brought It Back

The report describes the recovery in five steps, and each one is a useful pattern for a much smaller system:

1. Isolate the dead zone. Engineers routed regional traffic away from europe-west2-b, so requests stopped timing out against machines that were offline.
2. Protect what is still running. They changed scheduler allocations and capacity limits in the healthy zones to put existing customer reservations first and clear queues.
3. Restore power properly. Facility engineers restored a stable utility feed within an hour of supply coming back, then reset and tested the generator controllers.
4. Restart in stages. Core infrastructure and services were restarted step by step, not all at once.
5. Let traffic back in slowly. Only after checking storage integrity and system health did they return traffic to the zone, gradually, so job backlogs could clear.

Google also notes that storage writes in the region saw a short burst of errors during the first failover. If your app writes to storage, retries with a back-off are what carry it through a moment like that.

It Has Happened Before

Backup power and cooling are built to be redundant, and Google's own incident pages show they still fail together now and then. Three earlier European incidents, each from Google's status site:

The pattern is the same each time: a backup system failed along with the main one. So plan for losing a zone, and decide in advance what your app does when it happens.

What This Means for a Small Team

A small app does not have to survive a data centre failure, and that can be a fair trade. But choose it on purpose. These are the questions the report raises:

How to Fix It: Health Checks and Outside Monitoring

Failover only works if something can tell your app is unhealthy. Load balancers, container platforms and uptime monitors can be set to ask a health endpoint. One that always says "ok" keeps traffic flowing to a copy of your app that cannot reach its database.

// BAD: says "ok" even when the database is unreachable
app.get('/health', (req, res) => res.send('ok'));

// GOOD: answers 503 when a real dependency fails, and never hangs
async function withTimeout(promise, ms) {
  let timer;
  const timeout = new Promise((_, reject) => {
    timer = setTimeout(() => reject(new Error('timeout')), ms);
  });
  try {
    return await Promise.race([promise, timeout]);
  } finally {
    clearTimeout(timer);
  }
}

app.get('/health', async (req, res) => {
  try {
    await withTimeout(db.query('SELECT 1'), 2000);
    res.json({ status: 'ok' });
  } catch {
    res.status(503).json({ status: 'degraded' });
  }
});

The timeout matters. During this incident Google saw API timeouts across the region, and a health check that waits forever tells nobody anything. Our health endpoint guide covers liveness and readiness checks and Docker Compose health checks.

Second, watch your app from outside the cloud it runs in. Google's alerts fired 8 minutes after the zone went offline, but those were alerts to Google's engineers. Your customers only see what you tell them, and you only know what your own monitoring tells you. Our production monitoring checklist lists what to watch.

VibeDoctor's Vibe Check (vibedoctor.io) flags apps that have no health endpoint such as /health, and Vibe Monitoring checks your live URL every few minutes, recording whether it is up, how fast it answers and when it went down. Free to sign up.

Third, write down your plan before you need it: which region you would move to, where your backups are, and how you will tell users. A status page or even a pinned post is enough for a small team.

FAQ