Failed attempt to block phishing link results in massive Cloudflare outage

Cloudflare R2 is a zero egress-fee object storage similar to Amazon S3. The service offers free data retrieval, S3 compatibility, data replication, and integration with other Cloudflare solutions.

The incident occurred at the end of last week when one of the employees responded to a complaint about a phishing URL on Cloudflare R2.

“During a routine abuse remediation, action was taken on a complaint that inadvertently disabled the R2 Gateway service instead of the specific endpoint/bucket associated with the report. This was a failure of multiple system level controls (first and foremost) and operator training,” – Cloudflare.

In total, the incident lasted 59 minutes; in addition to the R2 object storage, it affected other Cloudflare services, including:

• Stream – 100% of operations (upload & streaming delivery);
• Images – 100% of operations (uploads & downloads);
• Cache Reserve – an increase in requests during the incident window as 100% of operations failed. This resulted in an increase in requests to origins to fetch assets unavailable in Cache Reserve during this period;
• Vectorize – 75% of queries to indexes failed and 100% of insert, upsert, and delete operations failed during the incident window;
• Log Delivery – latency and data losses (up to 13.6% data loss for R2 delivery jobs and up to 4.5% data loss for non-R2 delivery jobs); and 
• Key Transparency Auditor – 100% of signature publish & read operations to the KT auditor service failed during the primary incident window.

Some services were affected indirectly resulting in partial outages. For instance, Durable Objects had an increase in errors by 0.09% due to reconnections; Cache Purge, an error rate of 1.8% (HTTP 5xx) and a 10x increase in latency; while Workers & Pages, a 0.002% deployment failure rate affecting only R2-bound projects.

According to Cloudflare, the incident occurred due to both human error and insufficient validation safeguards.

The company has already taken appropriate measures. The possibility to disable systems has been removed from the interface used by abuse remediation personnel. Also, restrictions have been added to the Admin API to prevent internal accounts from disabling services.