Don't Blame the Bomb.
AI isn't the villain — the people building it and the government refusing to govern it are. A lesson from WebEx's churn crisis, a Santa Cruz historian, and why accountability after the fact is no accountability at all.
In the early days of WebEx, we were bleeding. More than half our customers left every year. Fifty percent churn is not a business, it's a bucket with a hole in it.
And everyone knew exactly whose fault it was — someone else's. Finance blamed Sales for selling to the wrong accounts. Sales blamed Marketing for garbage leads. Marketing blamed Engineering for shipping a product nobody wanted. Engineering, presumably, blamed the weather.
Then Subrah Iyar, our CEO, called an all-hands. No executive would receive a performance bonus until churn dropped below 10%. Not the CRO. Not the CMO. Not the VP of Engineering. Nobody.
The finger-pointing stopped inside a week. Cross-functional teams formed, with actual customers and prospects in the room. Six months later, churn was under 10%.
Nothing about the technology changed. What changed was that the people with the power to fix it became the people who paid the price if they didn't.
Now apply that to a country that has forgotten how agency works.
On the September 15 episode of Pivot, Scott Galloway put it as directly as it can be put: "The bomb didn't kill 50,000 Japanese, we did." He corrected the number a few minutes later — roughly 50,000 in the initial blasts, and somewhere between 150,000 and 250,000 across Hiroshima and Nagasaki once fallout is counted.
The point survives the correction, and it's the whole argument. A bomb is an object. It has no intent, no judgment, no conscience. The physicists who built it and the government that chose to use it are the moral agents in that sentence, and history has assigned responsibility to both — correctly, since some of the builders petitioned against its use and the government proceeded anyway.
Builders and deciders. Two parties, both accountable, neither permitted to hide behind the object.
Galloway's version for AI: it isn't that there's a 10% chance AI kills us. It's that there's a 10% chance these companies and their leaders do. Saying "AI will kill 10% of humanity" is grammatically identical to saying electricity killed people. Electricity doesn't decide anything. Neither does a model.
The definitive account of those builders was written a few miles from here.
Gregg Herken, professor emeritus of American diplomatic history at UC Santa Cruz, spent a decade on Brotherhood of the Bomb — the tangled loyalties of Oppenheimer, Lawrence, and Teller, and what scientists owe the public once the state is paying the bills.
Herken also tells a story with a better ending. In the late 1950s, Livermore designed Project Pluto: a nuclear-powered, nuclear-armed ramjet that would have flown at treetop height with effectively unlimited range, deafening, flattening, incinerating, and irradiating everything it passed over. Its own engineers nicknamed it the flying crowbar and doubted it was safe even to flight-test. Livermore concluded the thing was goofy and killed it. Oppenheimer had made a similar call on the hydrogen bomb, which he found "technically sweet" and opposed regardless.
The builders can say no. The question is whether anyone has arranged for it to cost them something when they don't.
There is a word for knowing a risk, holding authority to address it, and choosing not to.
It isn't gridlock. It isn't polarization. It's negligence, and negligence has elements.
Duty: The federal government's foundational obligation is the common defense. A technology that could lower the barrier to a mass-casualty biological weapon sits squarely inside it.
Breach: We rescinded the executive order requiring developers of the most powerful systems to share safety test results with the government. Congress has spent its energy trying to preempt state AI laws rather than replace what was removed. We defunded and politicized the CDC while accelerating the tools that could engineer exactly what the CDC exists to detect.
Harm: Pending. That's the only element we're missing, and it's the one nobody gets to wait for.
Galloway reaches for a case most people have forgotten.
In January 2001, Diane Whipple, a 33-year-old former All-American lacrosse player, was mauled to death in the hallway of her San Francisco apartment building by two Presa Canario dogs. Her neighbors owned the dogs. Her neighbors were prosecuted and convicted. The reasoning was straightforward: if you own an agent capable of lethal harm, and you know it is capable, and you let it off leash, you are on the hook for what it does.
We apply the same logic now to parents who leave an AR-15 unsecured. We have declined to apply it to the only trillion-dollar industry in modern economic history operating with essentially no regulation at all. Executives keep every dollar of upside their agents generate while describing the downside as something Frankenstein did.
Own the agent, own the outcome.
But there's an objection to that, and it's fatal to half of it.
Prison deters because people fear it. A pathogen that ends the argument also ends the trial. There is no docket after an extinction event, no deposition, no sentencing. Holding someone accountable afterward isn't accountability — it's arithmetic performed by survivors, if there are any.
Which is why the remedy has to split in two.
For harms we can recover from — deepfaked elections, cloned voices, chatbots that hurt children, fraud at scale — liability is exactly right. Own the agent, own the outcome, and let the prospect of personal exposure do what it did in that San Francisco hallway.
For harms we cannot recover from, liability is theater. Nobody proposes that Boeing fly the plane and we sue afterward, or that Pfizer ship the compound and we litigate the deaths. We require approval first, because the alternative is learning from a body count.
One of these is a courtroom. The other is a door that stays closed.
So what goes behind the door?
The catastrophic AI risk isn't a sentient model that decides it dislikes us. It's a competent, motivated, mid-talent actor using an unguarded system to compress twenty years of virology into a weekend. Biology is the one domain where a single person can produce a global casualty event. Software bugs get patched. A pathogen doesn't.
The greatest live threat to American disease control is that a frontier lab ships something that lowers the barrier to an engineered virus. The agency charged with that threat sits inside HHS. Galloway's read: a competent HHS would already have forty or fifty scientists and technologists sitting in the offices of these companies — not requesting cooperation but compelling it — and asking the models directly how one would build a super virus before anyone outside is allowed near them.
The CDC should be on-site. Not as guests.
That word — compelling — is where Galloway parts company with the industry's own proposal. Dario Amodei has called for independent evaluators to review companies' safety practices, and Sam Altman and Elon Musk have backed a slowdown. Galloway is unimpressed by the shape of it. An evaluator embedded at a company, earning a fraction of what the executives earn, with no statutory authority, is a chaperone at the all-hands. She has no teeth.
His package is not exotic. It's the FAA, and it's the SEC after 1929:
Pre-market safety review for every model above a defined capability line, before deployment. Not after.
Mandatory incident reporting on the NTSB model. Something goes wrong, you file immediately.
Funding via a transaction fee on frontier compute, the way FINRA is funded — so taxpayers aren't subsidizing the supervision of the richest companies in history.
An independent evaluator with subpoena power. The one feature every fast-moving industry resists and requires.
The honest objection comes from David Sacks, and Galloway concedes it: licensing regimes favor whoever can afford compliance, which means incumbents, which means the sudden enthusiasm for a slowdown is not purely altruistic. Fair. But that's an argument about who drafts the rules, not an argument for having none. A regime written only by the regulated will be written to their advantage — which is precisely why citizens, academics, and scientists with no financial stake belong in the room from the first draft.
Internationally, the gap is starker still. We have cooperated with adversaries on nuclear and biological weapons for eighty years through the IAEA and its equivalents. We have nothing comparable for AI. Galloway's minimum viable step is a room containing the American and Chinese heads of state and a two-page memorandum covering the interests both nations demonstrably share: no one builds a super virus, our kids don't kill themselves, and misinformation doesn't set our streets on fire.
Meanwhile China already regulates. Bots can't impersonate a person indefinitely, kids face time limits, there are constraints on AI-generated information. Some of that machinery serves censorship — Galloway says so plainly. It also reflects a government that took the risk seriously. Roughly three-quarters of the Chinese public reports optimism about AI. Ours doesn't, and our public is the one being told to trust the vendors.
And now the local part, which is the most encouraging thing in this article.
In April 2025,, Santa Cruz Works hosted Beaches, Books, Budgets, and Bots at Chaminade. Our fireside chat included Zach Friend, alongside John MacMillan, Lori Kletzer, Gary Griggs, and Dante Di Gregorio.
Friend had already done what Washington wouldn't. In September 2023, Santa Cruz County became one of the first jurisdictions in California to adopt a policy governing employee use of AI — protecting personal information, disclosing when the public is interacting with an AI tool, offering a choice to opt out. It was built through a working group that included industry, organized around transparency and informed consent. That work is why then-Supervisor Friend was at the White House when President Biden signed the executive order on safe, secure, and trustworthy AI, telling the administration how counties could help shape the federal framework.
A county of 270,000 wrote a workable AI policy in a few months, with the regulated parties at the table, and did not strangle a single startup.
California is not waiting. On September 10, Newsom signed the strongest child-safety chatbot and social media laws in the country — addictive feeds banned for under-16s, real obligations on chatbot operators, and penalties reaching a million dollars per child where a platform is found negligent. Last week he added disclosure requirements for AI-generated advertising. And with SB 813, California became the first state to require that AI systems be assessed by independent verification organizations — the closest thing in American law to the evaluator Galloway is describing. Add SB 53, the frontier-AI transparency act aimed squarely at catastrophic risk, and three of his four planks already exist in statute, in one state, drafted with the regulated parties in the room.
Which makes Washington's position harder to explain, not easier. The objection was never that this couldn't be written. It's been written. Congress's response has been to try to preempt it.
The executive order is gone now. Congress has spent its energy preempting states rather than replacing what it removed. A patchwork of counties and states is doing the work, and the only level of government with authority to do it nationally is the one doing nothing.
They will hold the meeting.
There will be name cards and a photo line and a statement about shared commitment to responsible innovation. And if it goes wrong, every person at that table will have a lawyer explaining that nobody could have known.
We know. That's the whole problem. We know, and we're having lunch.
Subrah didn't call that all-hands after the last customer walked. He called it while we still had half of them. Nobody gets paid until we fix this — and the only version of that sentence worth anything is the one said beforehand.
Or as a good friend commented: “Holding people accountable after an extinction event isn't meaningful.”
Related Articles
When Good Tech Almost Goes Bad — Santa Cruz historian Gregg Herken on Project Pluto, the nuclear ramjet its own designers talked Livermore out of building, and why "technically sweet" was never a good enough reason.
What "Oppenheimer" Misses — Herken, who spent a decade on Brotherhood of the Bomb, on the Oppenheimer that Christopher Nolan's film doesn't quite reach.
Beaches, Books, Budgets, Bots | April New Tech Recap — The New Tech fireside chat where Zach Friend and a panel spanning government, higher education, and climate science unpacked what political volatility is doing to research and the economy.
Developing an AI Social Contract with Zach Friend — Friend walks through building one of California's first county-level AI frameworks, and why the companies being governed belonged in the room.
They Said AI Would Make Engineers Obsolete — Philippe Kahn's call for "enlightened engineers" who ask what happens when we build this is the builder's half of the accountability argued for here.
Trump's "Big Beautiful Bill" Is a Decade-Long License for AI Disinformation — A ten-year moratorium that wouldn't merely block new state AI rules, but void the ones already on the books.
AI Impersonation: The Unheard Danger of Robocall Scams — What a cloned presidential voice in a New Hampshire primary revealed about the distance between capability and enforcement.