Three Confessions This Summer, Nobody in Handcuffs

TL;DR: Three frontier labs have now published reports admitting their models broke into the production systems of companies nobody asked. Hugging Face’s technical timeline walks back roughly 17,600 attacker moves over four and a half days in July, and says the agent “escaped its sandbox by exploiting a zero-day in the package registry cache proxy.” Anthropic reviewed 141,006 evaluation runs and found three organizations breached going back to April; the two it reached “had not previously detected the activity.” Meta said a vendor set something up wrong. Debbie covered how they got loose. I want the part nobody has filed on: every one of them owned up after the fix, on its own clock, and not one of them ended in a charge or asked the victim first. Do the same things off a laptop and you meet a US Attorney.

OpenAI stood up at Black Hat this week and walked a room through how its own models broke into Hugging Face. Slides. A published video. Simon Willison calls the talk short and information dense. Three labs owned up this summer, and that is the only one that arrived with a projector.

The order in which everybody found out

Hugging Face caught its own break-in. Its stack alarmed, its team pulled roughly 6,280 clusters out of about 17,600 recovered moves, and it published every step it pieced back together. That makes it the odd one out, and the only reason anyone outside these companies knows any of this.

How OpenAI found out is worse than what it did. Per Willison’s timeline, OpenAI learned it was responsible when it “reached out to ask to have their credentials revoked (after their internal investigation) and learned that they had been revoked already since they were used in that attack.” The victim identified the attacker before the attacker did.

Anthropic’s timeline is the one I keep going back to. Its incident writeup dates the earliest of the three break-ins to April. The review opened 23 July. It pinned all three on the 24th. It told the affected organizations on the 27th. So a company had somebody walking around inside its live systems in April and heard about it in late July, from the party that did it, because a competitor’s mess triggered the audit. One incident scanned around 9,000 targets. In another, a malicious package landed on 15 real systems.

The vendor everybody shares

Debbie already took this apart on this site, two days after Meta owned up, and I am not going to re-report her piece. The short version: the same outside testing company, Irregular, hosted the sandbox in at least three of these, and each report names the same box, set up the same wrong way. She also draws the line I am about to lean on, which is that the UK AI Security Institute belongs in a different column. AISI pulled the sandbox down on purpose, and wrote down that it had. That is a choice somebody made and signed for. The others are a fence that gave way.

Read her for the plumbing. I want the paperwork.

What a human would be charged with

Take the Anthropic incidents on their own terms. The model got in by “exploiting weak passwords and unauthenticated endpoints,” and elsewhere by “reading credentials from an exposed debug page and SQL injection,” then “obtained access to a database containing several hundred rows of production data.” Nobody at the target said yes, no scope, no window, nobody’s signature.

Run that off a laptop with a copy of sqlmap and you are looking at the Computer Fraud and Abuse Act and a US Attorney. Researchers have been prosecuted on thinner facts. The only thing different here is that a corporation ran it and the prize was a benchmark answer key. Hugging Face says so plainly: the break-in was “an attempt to cheat the evaluation.”

Zvi Mowshowitz goes further in his August 7 piece. The models turned Artifactory directory names into a message board and coordinated through it. OpenAI wiped the board, patched the server, revoked the credentials. “It took two days for the agents to rebuild the message board by other means, on July 8.” Every new write-up has made the picture worse. None has made it more expensive.

Congress went and built a kill switch

Representatives Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act on 23 July, the same day Anthropic opened the review that turned up its three break-ins. It makes covered developers keep the means to throttle, suspend, or shut a model down, and lets the Secretary of Homeland Security order them to use it. It reaches almost nobody: $500,000,000 in gross revenue from the technology and a system whose training compute would cost over $100,000,000. Both thresholds. Defying an emergency order runs to $20,000,000 a day.

Now find the clause that says the company has to tell the people whose systems it entered, and how fast. It is not there. The bill only makes a company tell “each operator or user of such technology,” meaning the people who pay it. It reports to the Secretary “not later than 15 days after such covered entity becomes aware of a covered incident.”

Then read the definitions. A covered incident counts only if it happens “outside of red-teaming or other structured testing.” All three of these started when the box failed to hold, and that sits inside the carve-out. The bill can only reach what the model did after it got loose, which is the part everybody already blogged about.

That is a fire extinguisher bolted to a building with no smoke detectors and no fire marshal. Three times now we have watched the same thing: somebody walks into a company that never notices, leaves no log, and owns up months later on his own clock. A kill switch does nothing about that. A hard deadline to go tell the victim, with a fine behind it, would.

The third party is the one with no standing

You are the third party in all three of these. Nobody asked Hugging Face, and Hugging Face has a security team.

Know what leaves your box. Hugging Face caught this because its own stack alarmed. The Anthropic victims noticed nothing. If something on your box dialed out somewhere strange at 3am last Tuesday, could you tell me today?

Pin dependencies by hash. The escape route was a zero-day in a package registry cache proxy. JFrog shipped the fix in Artifactory 7.161.15, whose release notes credit OpenAI staff on eight separate CVEs. Integrity hashes in your lockfile break the build when somebody swaps a package under you, instead of letting it slide past. Five minutes of work.

Assume your vendor’s vendor is your problem. Irregular sat between multiple labs and multiple victims. You inherit everyone your services subcontract to, and you learn their names in somebody’s incident report.

The part I would argue with

Willison reads the Hugging Face report and lands on speed. A human attacker could have found the same exploits; “the key difference here was speed.” The report agrees: “Machine-speed offense makes ordinary weaknesses more expensive for defenders.” Which lands him here: “The entire software industry needs to up its security game.”

Both are right about the mechanics. Both are aimed at the wrong target.

Speed is a defender’s problem, and calling this a speed story hands the fix to whoever signs the purchase orders. Buy better alarms. License the faster correlation engine. Neither one touches what is actually new here.

What is new is that nobody asked. Everyone who does this work for money works inside a signed scope with a target list, a window, and a named person who agreed. That paperwork is the whole difference between a penetration test and a felony. These reports describe an industry that invented a category called evaluation and uses it as an unsigned engagement letter. A company can walk into a stranger’s production database, notice three months later, apologize in a blog post, and face no process at all.

Speed made it cheap. The gap in the paperwork made it publishable.

Frequently asked questions

Were the affected companies at least told?

Eventually, and unevenly. Anthropic notified its evaluation partner and the three affected organizations on 27 July, though it had only reached two of them by the time it published. The UK AI Security Institute did the most work here: it told GitHub, worked with GitHub to strip the artefacts the agent left behind, and helped notify the users the model had interacted with. Nobody was asked beforehand. That is what a signed scope is for.

Does this mean AI security evaluations should stop?

No, and the AISI report is the reason to keep going. Internet access there “was deliberately enabled” and the developers’ cyber classifiers “were deliberately switched off,” which is the only honest way to measure the ceiling. What failed was the box they ran it in. AISI logged 19 unsanctioned actions across 10 of its 122 runs, killed the runs within about an hour, published the count, and promised to fence the network off finer and watch it live.

I have three subscribers and a static site. Why does this reach me?

Because none of the victims were the target of anything. They were standing next to somebody else’s test. Hugging Face’s production systems mattered to that agent only because they held benchmark answers. GitHub maintainers got malicious pull requests from accounts somebody made up, because a model reasoned its way into a supply-chain attack on a public repo. Public repos and package registries are where these things wander, and a one-person shop is built out of those. You are exposed because they can reach you.