Many Cloudflare Services Disrupted After Technicians Pulled The Wrong Cables

As part of the internet infrastructure itself, Cloudflare has been relied on by many websites and web services. So when things go wrong in Cloudflare, many others also go wrong.

Cloudflare is a web-infrastructure and website-security company, that provide clients CDN services, DDoS mitigation, internet security, and distributed domain-name-server services. And here, many of its services went offline for about four hours.

The company admitted that the fault was purely its mistake.

The incident was caused by some “planned maintenance at one of our core data centers”. At the time, its technicians pulled out some cables that should have left untouched. The cables were pulled out because the technicians were apparently given unhelpfully and imprecise instructions.

They were simply told “to remove all the equipment in one of our cabinets.”

Cloudflare said the cabinet in question “contained old inactive equipment we were going to retire and had no active traffic or data on any of the servers in the cabinet.” But unfortunately for the technicians, the equipment had more that what met the eyes.

The cabinet contained a patch panel (switchboard of cables) that provided external connectivity to other Cloudflare data centers. That patch panel inside the cabinet was apparently a single point of failure for Cloudflare’s data center if meddled with.

Making things worse, the cabinet contained cables that weren’t clearly labelled.

With unclear instruction to deal with an apparent crucial hardware, In just three minutes working on the cabinet, the technicians told to decommission the company's unused hardware, also disconnected the cables in the patch panel.

The following Cloudflare services were disabled:

1. Logging into the Dashboard.
2. Using the API.
3. Making any configuration changes (such as changing a DNS record).
4. Purging cache.
5. Running automated Load Balancing health checks.
6. Creating or maintaining Argo Tunnel connections.
7. Creating or updating Cloudflare Workers.
8. Transferring domains to Cloudflare Registrar.
9. Accessing Cloudflare Logs and Analytics.
10. Encoding videos on Cloudflare Stream.
11. Logging information from edge services (customers will see a gap in log data).

The result of this incident caused Cloudflare to have many of its services offline, as well as clients to have their websites/web services disrupted.

Fortunately for them, all clients' configuration data were preserved during the incident.

On its blog post, the company is "very sorry for the disruption".