INTERNET FACILITATOR Cloudflare has been shown up by Google's Tavis Ormandy, who has revealed that the firm has suffered a memory leak, has a problem with user data diarrhoea, and a joke of a bug bounty program.
Ormandy finds bugs like insects that eats bugs do, and he found this issue around a week ago and took it directly to Cloudflare. He seems rather disappointed at the reaction and action from the internet giant, suggesting that it didn't even communicate well with him during the cleanup process.
But first the bug. Probably the least interesting part of this, but definitely the most confusing, is that Ormandy found that something within the Cloudflare network was crap, not his words, and was letting a load of user crap just flow out into the internet.
He said he was analysing some information from publicly available databases when something interesting happened.
"It became clear after a while we were looking at chunks of uninitialized memory interspersed with valid data. The program that this uninitialized data was coming from just happened to have the data I wanted in memory at the time," he said.
"That solved the mystery, but some of the nearby memory had strings and objects that really seemed like they could be from a reverse proxy operated by cloudflare - a major cdn service.
"A while later, we figured out how to reproduce the problem. It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like heartbleed, but cloudflare specific and worse for reasons I'll explain later).
"My working theory was that this was related to their ‘ScrapeShield' feature which parses and obfuscates html - but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers".
Ormandy did what Ormandy does and took his discovery to Cloudflare. He said that the firm reacted swiftly, but slowly, not communicating with him very well and overestimating how well its clean up operation was going.
Things started well, but a chain of updates from Ormandy show how swiftly things changed.
"I worked with cloudflare over the weekend to help clean up where I could. I've verified that the original reproduction steps I sent cloudflare no longer work. We're discussing some remaining issues and Cloudflare are still working on their investigation. From our data, we believe the issue has been present for some time, I've given cloudflare all the information we have.
"Cloudflare have assured me they will prepare a detailed postmortem for their customers once the issue is resolved. They have an excellent reputation for transparency, so that's good enough for me" he said on day one.
However, just a couple of days later, and after a couple of incidents of more leakage were uncovered by Ormandy and Google's Project Zero, the tone of the posts changed.
"We keep finding more sensitive data that we need to cleanup. I didn't realize how much of the internet was sitting behind a Cloudflare CDN until this incident. The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup," says one.
"I had a call with cloudflare, and explained that I was baffled why they were not sharing their notification with me. They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response," added another after Cloudflare kept coming up short of notification about a public notification.
Ultimately though, he had us with this: "Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt. Needless to say, this did not convey to me that they take the program seriously."
Cloudflare has done all of the work it needed to do apparently, and has issued a blog post that explains what happened and what it did. Ormandy said that it makes for good reading, but doesn't give users the full gory detail.
It's long enough, and the gang does thank Tavis, but we can see his point. µ
Firm Kracks down on WiFi flaw
Razring the stakes
The pop-up camera is being binned
But Google buries privacy weakness