The information security community is collectively and justifiedly furious over the revelations by Brian Krebs this week of an unconscionable and wholly preventable data breach by Panera Bread.

Brian based his initial reporting on evidence provided by a super-patient security researcher, Dylan Houlihan, who, emails show, informed Mike Gustavison, Panera Bread’s future former Information Security Director*, of the breach in August of 2017. Houlihan, who asked for neither reward nor public recognition for his act of public service, was blown-off by Gustavison and Panera in ways that alternated between insulting and supercilious.

Despite what Gustavison said in his August 3, 2017 email to Houlihan, having read Houlihan’s original, it sure didn’t appear to me that Houlihan was in any way shaking Panera down, or applying for a “beg-bounty.” He was instead simply informing the company that he found something — and offering to send it encrypted, lest Houlihan make a bad situation worse.

In his August 3, 2017 response email — an email that I am certain he felt made him look “tough” — Gustavison did absolutely everything wrong.

The first of a series of emails to Panera Future Former Chief Security Officer Mike Gustavison received from Dylan Houlihan in which Gustavison treats Houlihan with scorn, derision, suspicion, and disrespect (From Houlihan’s

Ultimately, Panera may have breached 37 million customer records including PII precisely in the way Houlihan warned them they were going to breach them. Panera’s soulless flacks then told us they care about our security.

Houlihan, who wrote up his observations and a timeline of his continued attempts to get Panera’s CISO to listen, then acknowledge, then actually, you know, do something about venting 37 million customer records onto the public Internet, did everything the way the Mullahs in the “coordinated disclosure cabal” say one should. The coordination of the disclosure failed because the vendor had no interest in disclosing to anyone — or even admitting to themselves — that anything was wrong.

The Panera Breach is yet another example of scurrilous, cynical, irresponsible companies blowing-off security researchers, and it is these exceptions that test the high-fallutin’ rules about “coordinated disclosure” proffered by security chinwaggers making their way round the conferencenti-circuit: the only way to make companies like Panera, that demonstrably couldn’t give a toss about customer data, is to go directly to the public —at a minimum, to the press, and at best directly to public equity markets — with the disclosure.

The information security community has gone way, way too far in abetting irresponsible companies. Our “coordinated disclosure” rules have bent backwards so far that many times, irresponsible companies face zero consequences for blowing off (as Gustavison did) a legitimate security researcher with a legitimate issue — and the only way it comes out is when legitimate security researchers are unafraid to go public with findings that a company blows off.

Coordinated disclosure as envisioned today is like a hook-and-eye lock for your front door: it keeps honest people honest — and since most people are honest, some people conflate that with “effective.” I aver disclosure rules should work to coordinate the three percent that are outside the norm; that don’t conform to some ham-brained, Socialist ideal of how corporations should behave when confronted with news they don’t want: disclosure that doesn’t punish the recalcitrant, the deliberately obtuse, isn’t just worthless. It’s harmful to our society.

That’s why I thought it was so important when MedSec and Muddy Waters held St. Jude accountable in the public equity markets for St. Jude’s abject refusal to fix problems they knew were real.

Now, let’s be clear: I’m not against coordinated disclosure if you’re dealing with a company whose people have, you know, ethics, and a soul. I’m not yelling at Panera for having a vulnerability, or even for breach — everyone has vulnerabilities, and on a long-enough timeline, everyone gets hacked. Information security is, like, very hard.

I’m yelling at them for deliberately ignoring it, shining on and fobbing off a legitimate researcher who played by what some keep quaintly referring to as “the rules” — rules that protect the evil, not the good.

Rather than prescribing what everyone should do with specifics, I like to think of this strategically — when you’re dealing with complex systems, things can go wrong, and there’s no “right” answer for everything. That’s why pilots, faced with impending disaster, follow a strategy: the priority is Aviate, Navigate, Communicate.

So, for example, when I get a note like the one Gustavison got from Houlihan (it happens sometimes), the first thing I do is presume it is well-intentioned and genuine, and quickly write back to say “Thanks, we’re looking at it.”); the second thing is to check whether the facts are correct (is the thing he says is broken, broken?); the third thing is to move to fix immediately the problem and determine whether there’s more; and the last thing is to communicate with the person who alerted us to let them know what we’ve found, and what we’d like to do next.

This all should take place within single-digit days. And here’s a good rule-of-thumb: no more than two business days should elapse between notification and response to the notifier at least letting them know you appreciate their reaching out and that you are examining the issue.

That’s not apparently what Panera and Gustavison did here. And that’s not all that rare. For every Buffer — a company that took its responsibility so seriously it was issuing blog posts every couple of hours after a small breach — there’s a Panera engaging in the age-old practice of “Deny, deny, deny.”

Note that this morning — days after the revelation that 37 million records may have been lost, and months from the notification, the Panera Bread “news” page mentions nothing.

Screen capture from 4 April 2018 of the Panera Bread Press Room

With Panera, we even can’t punch them in the ticker: the company was bought last year by a privately-held, German-owned, Luxembourg based conglomerate, JAB Holdings, which also owns Keurig Green Mountain, Peets, Einstein Bros, Noah’s, Brueggers Bagels, Caribou, Krispy Kreme, and other stuff. Mmm, just think of the homey, hand-made advertising images of Panera (clean), Keurig, and Peets — these pure images just practically conjure visions of German henchmen in their tax-haven, no?

No, but Panera needs a kick in the testicles. Or, if you will, since they’re now German, a kick in the eier. How dare Panera and Gustavison let a problem this big — millions of customer records — sit unencrypted and available to all on their website for months, in order to avoid some hard work of re-engineering some processes that themselves amounted to technical debt and leadership failure?

Being polite to people like this is not what is called for.

They all collectively need, and deserve, a punch in the throat. Heads should roll — from the CEO down. This wasn’t neglect.

Ignoring this problem was intentional.

Heads should roll.

____

*As of May 30, Gustavison is still listed as CISO of Panera on his LinkedIn Page.

Home

Coordinate This: Being Polite To Bad Companies Abets Bad Security