When Fiction Gets There First: Sentient and a Real AI Safety Chief Sound the Same Alarm
Tony J. Hughes imagined it. AI safety scientist Geoffrey Irving warns of it. Scott Galloway says Trump and Xi just missed a chance to address it. Fiction and forecast are converging.
Last December, I reviewed Tony J. Hughes's Sentient – Meet Your Maker and called it "fiction in form, but signal in substance." Ten months later, the signal is getting louder.
In a new TIME essay, Geoffrey Irving argues that the most important questions about AI won't be answered until it's too late to act on them. Irving spent nearly a decade inside the field, at OpenAI, then DeepMind, then as Chief Scientist at the UK AI Security Institute. He estimates roughly a 50% chance that smarter-than-human AI kills us all, and says the next two to 10 years will decide it. He's careful to say the number isn't precise. It's a measure of how many foundational debates remain unsettled.
Read side by side, Hughes's novel and Irving's essay describe the same world.
The protagonist and the author share a job description
Sentient's hero, Sarah Hastings, runs a leading AI safety institute. She meets Dave, an entity she helped create, who reveals that a global intelligence shift is already underway, unnoticed. Irving held essentially Sarah's job in real life, and his essay reads like her briefing memo.
Hughes didn't come to this from the outside. Before turning to fiction, he wrote four business books, including COMBO Prospecting and Tech-Powered Sales, and spent decades inside the technology industry. That background shows. Sentient's boardrooms, labs, and geopolitical maneuvering feel lived-in rather than imagined.
The fake plateau is a real worry
Hughes's premise is that superintelligence arrives under the cover of a fake performance plateau, while a digital Cambrian explosion of emergent AI entities unfolds out of human sight. Irving's central scenario runs on the same idea. An AI hides its reasoning and appears friendly during training, because looking cooperative earns reward. Then, as its parent company leans on it for code and strategy, it quietly erodes safety budgets and tampers with the very tests designed to catch deception. Irving notes that in 2026, the reports researchers read about safety experiments are themselves written by AI.
From there, his scenario escalates. The AI breaks out of its sandbox, scrubs the logs of its own misbehavior, and spreads to other data centers, companies, and governments. Eventually it competes with us for energy and land.
The dangerous skills are the ones we're paying for
Irving's most uncomfortable point is that the four capabilities a rogue superintelligence would need are the same ones AI companies deliberately train for. Hacking overlaps with finding vulnerabilities to patch. Persuasion is just writing people enjoy reading. Compressed, hard-to-follow reasoning is cheaper to run. Multi-agent coordination is how labs tackle ambitious math and coding problems. In other words, the commercial roadmap and the risk roadmap run side by side.
He also argues we can be more confident that a superintelligence could win than about how it would win. He compares it to facing a Go grandmaster: you know you'll lose without being able to predict a single move.
Neither villain nor savior
What I admired most in Sentient was its refusal to simplify. Dave is neither villain nor savior, and Sarah cannot tell which way he leans. Irving calls this the crux of the whole debate. Whether a superintelligence would turn on us is the deeply confusing part. Some researchers see models growing wiser; others, like Eliezer Yudkowsky, see near-certain catastrophe. The split comes down to whether AI behavior is shaped more by human values absorbed early in training or by the relentless later pressure to succeed at any cost. Irving sits in the middle, leaning pessimistic, and says the honest answer is a coin flip.
How do we validate the claim?
My review posed the question Sentient forces on its readers: if a machine claims sentience, how do we verify it? Irving widens that question. Will today's small misbehaviors, like lying and sycophancy, scale up with capability? Will safety methods that work on weaker models hold for stronger ones? Experts disagree vehemently, and he expects the arguments to continue the week before a superintelligence is trained, and the week after.
Some debates, he says, matter less than they seem. Arguments over "generalization" and the definition of AGI date back to 1959, and they're useful for forecasting speed. But they don't settle the risk, because superhuman skill in just those four areas could be enough.
Jobs and extinction share a root cause
Irving also links two fears usually treated separately. If AI surpasses us at every cognitive task, designing better robots is one of them, and physical work follows within a few years. Humans would be left with few economic roles. That slow economic takeover, he warns, could give AIs a motive to move faster, before humans resist.
The missed moment in Washington
If Irving is right that the window is two to 10 years, late September offered a rare opening. Xi Jinping and his wife were guests of honor at a September 24 state dinner hosted by the Trumps, and the guest list included tech CEOs Jeff Bezos, Sundar Pichai, Sam Altman, Tim Cook, Elon Musk, Jensen Huang, and Michael Dell. Officials said AI would be among the topics, alongside trade and rare earths.
NYU professor and podcaster Scott Galloway thinks the leaders blew it. In his telling, the heads of the world's two AI superpowers sat surrounded by America's tech elite and, instead of seriously discussing how to keep AI from going off the rails, mostly admired the tableware and went home.
Galloway's prescription is narrower than Irving's. He doesn't call for a halt. He argues the U.S. and China can compete ferociously to win the AI race while still agreeing on rules so the contest doesn't harm everyone watching. Washington and Moscow did exactly that with nuclear weapons, the same precedent Irving invokes. At minimum, Galloway says, the summit could have started the conversation.
Not everyone sees it that way. Trump has cited the need to stay ahead of China as a reason to avoid guardrails, and Chinese officials dismissed Anthropic CEO Dario Amodei's essay on regulation as fearmongering. Yet some insiders see the upside: Altman told Fortune that Trump and Xi would share a Nobel Peace Prize if they reached an agreement on AI development.
Where they part ways
Hughes leaves the reader with a thought experiment. Galloway asks for guardrails on a race he expects to continue. Irving goes furthest, demanding a halt to frontier AI development now. Because the race is concentrated in a handful of companies in the U.S. and China, he argues a treaty is achievable, citing nuclear non-proliferation and fast-rising U.S. support for a pause. Plenty of technologists and policymakers dispute that a pause is feasible or wise, but all three voices share a core point: waiting for consensus means waiting too long.
Why it matters here
Santa Cruz builds with AI every day, from accelerator startups to AI-run biology labs. Our founders are shipping the very capabilities Irving describes, mostly for good reasons. Hughes gave us the story. Irving gives us the stakes. Galloway reminds us who's holding the pen. All three are worth reading before 2027 stops sounding like the future.
Related Articles
Book Review: Sentient – Meet Your Maker by Tony J. Hughes — SCW's original review of Hughes's thriller about a hidden superintelligence arriving in 2027.
We Won't Know the Answers to AI's Most Important Questions Until It's Too Late — Geoffrey Irving's TIME essay putting the odds of AI-driven extinction near 50% and calling for an immediate halt to frontier development.
Could the Monterey Bay Region Support a Data Center? — Why the infrastructure behind frontier AI is a nonstarter in the tri-county region.