A little bit geek, wonk, and nerd. Repeat entrepreneur, recovering lawyer and former ski instructor. CEO & co-founder of CloudFlare. [ my public key: https://keybase.io/eastdakota; my proof: https://keybase.io/eastdakota/sigs/_uDY0ZsLTEWaNu5daRtuwzZtJDJJrtGS4uXoYxwI634 ]
Submissions
Cloudflare outage on November 18, 2025 post mortem
blog.cloudflare.com
1,465 points·by eastdakota··916 comments
[untitled]
1 points·by eastdakota··0 comments
Cloudflare's 2025 Annual Founders' Letter
blog.cloudflare.com
16 points·by eastdakota··2 comments
Cloudflare beats "patent troll" tying cloud services to old router hardware
arstechnica.com
2 points·by eastdakota··0 comments
Post Mortem on Cloudflare Control Plane and Analytics Outage
blog.cloudflare.com
486 points·by eastdakota··231 comments
Cloudflare's 2023 Annual Founders' Letter
blog.cloudflare.com
3 points·by eastdakota··3 comments
BigPinapple Powers 1.1.1.1
blog.cloudflare.com
9 points·by eastdakota··1 comments
Welcome to Cloudflare’s Impact Week
blog.cloudflare.com
3 points·by eastdakota··0 comments
Edge Network and Compute Performance Comparison
blog.cloudflare.com
1 points·by eastdakota··0 comments
Leading VC to provide up to $1.25B to back startups built on Cloudflare Workers
blog.cloudflare.com
2 points·by eastdakota··0 comments
The mechanics of a sophisticated phishing scam and how we stopped it
blog.cloudflare.com
20 points·by eastdakota··1 comments
NSIT’s Post Quantum Encryption Surprise
blog.cloudflare.com
4 points·by eastdakota··0 comments
NIST’s Post Quantum Encryption Surprise
blog.cloudflare.com
5 points·by eastdakota··0 comments
Cloudflare doesn’t have to cut off copyright-infringing websites, judge rules
arstechnica.com
407 points·by eastdakota··176 comments
Edge Network Performance: Akamai, Cloudflare, CloudFront, Fastly and Google
blog.cloudflare.com
15 points·by eastdakota··11 comments
Patent Troll Needs to Learn a Lesson: Cloudflare Wants to Destroy Another Troll
techdirt.com
85 points·by eastdakota··1 comments
A quirk in the SUNBURST DGA algorithm
blog.cloudflare.com
2 points·by eastdakota··0 comments
How Initial Screenings Should Be Done
dev.to
1 points·by eastdakota··0 comments
The Edge Computing Opportunity: It’s Not What You Think
That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.
There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.
Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.
We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
Free customers are served from the nearest location where we have capacity. If we’re capacity constrained then free customers will be the first to be rerouted to another facility with capacity. That typically only happens for a very narrow window during any day. It has nothing to do with load to your particular site. It has to do with a region’s capacity and a group of customers (e.g., FREE PRO BIZ ENTERPRISE).