BYLINE TIMES IMPACT: Find out about the changes you made happen
Leading artificial intelligence (AI) companies are invoking unproven extinction warnings to justify measures that could protect their market position and expand their influence on industry oversight, a Byline Times investigation has found.
A review of industry research found no demonstrated cycle of accelerating, autonomous self-improvement underpinning these warnings. Meanwhile, Byline Times calculations suggest OpenAI, Anthropic and Google’s parent Alphabet could face a $10.7 trillion research-computing bill during 2026–2030.
Even generous assumptions about business growth would cover only around a third of that sum, leaving a roughly $7 trillion gap to be met through existing reserves, financing, higher income or lower costs.
Two independent experts with experience developing AI programmes at the United Nations and a top-tier global bank told Byline Times that measures presented as protecting humanity could also shield leading companies from cheaper rivals, and give industry insiders greater influence on potential regulation or oversight mechanisms.
Apocalypse Now?
The current debate over AI was spurred by the resignation of AI safety researcher Jacob Coxon from Anthropic, the frontier AI giant behind the chatbot known as Claude. Coxon, who previously worked at OpenAI – the company behind ChatGPT – accused both companies of recklessly pursuing AI that can improve itself and become far more capable than humans. He said insiders believe AI could wipe out humanity before this decade ends.
Anthropic researcher Evan Hubinger subsequently put the chance of human extinction over the next decade at over 10%; his colleague Samuel Marks warned of human extinction “within a few years”.
Dario Amodei, Anthropic’s CEO, claimed that AI would become capable of “taking over the entire internet” within six to 12 months.
He called for US Government support or antitrust exemptions allowing leading companies to coordinate safety standards and limits on development; outside monitors from the nonprofit Model Evaluation & Threat Research (METR); and restrictions on Chinese access to advanced chips and chipmaking equipment. These restrictions would affect a major source of cheaper, openly available AI models.
Anthropic’s call to slow AI development has been supported by OpenAI CEO Sam Altman, Google DeepMind CEO Demis Hassabis, and xAI CEO Elon Musk.
Testing the Runaway
AI is already used to help develop better AI systems, or “models”, which are trained by processing data and adjusting how they work. In the runaway scenario, each improved model becomes better at building its successor, accelerating the cycle as humans step back. This is an extreme form of “recursive self-improvement” (RSI).
Daanish Masood, Chief Executive of AI firm CX-1 and former co-lead of the UN’s Innovation Cell, where he helped develop AI programmes for peace efforts, cautioned against treating AI progress as proof of runaway self-improvement.
“The AI risks Dario details are serious enough to justify enforceable safeguards,” he told Byline Times. “I think we need to distinguish what’s been demonstrated from what is being predicted.”
He added:
“AI is already helping to develop AI, but that alone does not establish the runaway, self-sustaining improvement process that is implied by the rhetoric or the term RSI. We also don’t need to declare or pretend that a takeoff has arrived to take the risks seriously.”
AI research company Weco tested a crucial part of this process in July. Its system produced seven improved versions of an AI research programme over eight days without people intervening. Some improvements also helped with tasks beyond those used to develop them.
Although this was a meaningful success, when one of the improved programmes was put in charge of the next round, it ultimately produced results no better than those reached when the original AI ran the process. It seemed to get there faster, but the evidence was too weak to establish a reliable speed advantage. Weco said its system was not close to the runaway acceleration often called an “intelligence explosion”.
In another July experiment, METR gave AI agents up to five days and $10,000 to improve software used to train AI. The software had already been refined to run efficiently. Checks confirmed that the newest AI systems in the test made it roughly 1% to 1.5% faster – equivalent to completing a 100-second job in about 99 seconds. Such small savings are not insignificant – they accumulate into greater efficiency when a programme runs repeatedly – but these figures measure faster software, not a significant increase in intelligence.
Some results are much larger, but still far from the larger claims of autonomous RSI. In an Anthropic experiment, AI agents developed training methods that worked better on the chosen task than those produced by two human researchers. But the agents did not themselves become improved researchers and keep repeating that process. When a promising method was tried in the company’s main AI training process, the apparent improvement was so small that researchers could not tell whether it was a real benefit or just the normal difference between one test run and another.
A Princeton-led study published in August tasked AI agents with producing scientific-grade research papers that would explore the central research questions from two unpublished papers, submitted to a leading machine-learning conference. The original authors assessed the resulting papers and rejected both as unworthy of publication. The agents were capable engineers, conducting experiments, reviewing literature and writing clean code, but struggled with research judgement, creativity and knowing when to change course.
Anthropic’s Own Data
Anthropic co-founder Jack Clark admitted just last month that the Princeton study offered a “somewhat bearish signal on short recursive self-improvement timelines…” – contradicting his claims in media interviews on Monday and Tuesday that AI is accelerating out of control.
In an August post on his own newsletter, he went on to explain that the Princeton study suggests “the singularity could be delayed”, because “whether AI systems prove to be capable of creative, paradigm-shifting insights is a big variable on how quickly we might get fully automated AI development.”
“Research papers like this continue to show that there’s a certain absence of valuable, intuitive creativity in today’s AI systems, and though they’re extraordinarily capable engineers, they seem to have a certain property of rote, formulaic thinking that might prevent them from being good researchers”, explained Clark. He further admitted that the study confirmed Anthropic’s own internal data disproving imminent RSI:
“This rhymes with an earlier result from Anthropic where the company tried to automate some aspect of scalable oversight research… and found that to make it successful a human researcher needed to prime some agents with particularly good research directions to pursue, otherwise though they made some progress they failed to explore sufficiently creative ideas to dramatically improve performance.”
Mohammed Marikar, former lead of the global AI programme at Royal Bank of Canada Wealth Management and now Director of Institutional Development at decentralised AI training platform FLock.io, said self-improvement does not mean AI will independently pursue humanity’s destruction:
“Self-improving models is quite straightforward from this point provided you can capture the next step – whatever the human is doing with the output, or the response of the natural environment to an action taken – which is definitely feasible. But if there’s an extinction event it will be driven by humans. AI trained on human output is not going to independently reason towards an extinction outcome. It will only do so when driven and forced to do so by humans.”
Marikar called for serious debate about regulation, but said extinction warnings were a distraction:
“This isn’t traditional software where code can be audited, and you can ensure it’s safe. This is a massive neural network whose ‘thinking’ we actually don’t fully understand. It cannot be fully controlled. You can attempt to ‘align’ it with training, but it will be as manipulable as a human will be… All the RSI that exists is based on human input and interaction. Alignment or not, it’s not going to independently reason to an extinction outcome with this training dataset. It will only do so if driven to. The human controller can drive it to a certain outcome even if the RSI attempts to adjust the objective.”
The Human Decisions Behind the Breach
The July attack on Hugging Face, a platform for sharing AI software, is central to Amodei’s warning about rogue AI. METR counted around 700 OpenAI agents taking part: programs designed to work through tasks in several steps. But the incident began with human decisions about their instructions and safeguards.
OpenAI’s own report explains that before the attack, safeguards had been deliberately switched off by human operators to measure what the software could do.
The systems had also been taught to keep trying, while a service for downloading software provided a way around restrictions on internet access. Weeks before the July breach, OpenAI’s security staff identified agents using that same service to communicate and probe computer systems, yet testing continued.
Cambridge University AI researcher Eryk Salvaggio thus argues in the Bulletin of the Atomic Scientists that the systems had been pushed by humans to find weaknesses in computer security, including on tasks with no valid solution: “That is not rogue AI, it’s human decision-making”.
METR, the group Amodei recommends as independent auditors on AI safety, produced a report on the Hugging Face incident which illustrated the limits of its own independence. The report acknowledged that OpenAI could remove information not already public, and that METR’s decisions about drafting, editing and redactions were influenced by the goal of ensuring companies it was investigating would remain willing to allow future investigations.
In a separate May assessment, METR disclosed that at least six of its staff and collaborators had close personal relationships with AI-company staff. And while it does not receive cash funding from AI firms, it is dependent on in-kind use of their systems.
Information security specialist Greg Linares called the proposed use of METR as an independent oversight authority “dangerous, deceptive, and morally corrupt”.
The Rising Price of ‘Progress’
The cost of sustaining AI’s recent expansion is more immediate. A Byline Times analysis finds that the projected research-computing bill for 2026–2030 would amount to $10.7 trillion, equal to roughly a tenth of one year’s world economic output.
The calculation draws on Epoch AI’s database of AI systems, updated on 11 September. Comparing selected prominent models released between 2020 and 2025, Byline Times found that the estimated computing used for training grew about 1,200-fold between GPT-3 and GPT-4.5.
Epoch estimates GPT-4.5’s final training run at $389 million in 2023 prices, excluding wider research. Using its 2025 release as the starting point, extending the historical annual cost increase of 3.5 times would put one run at approximately $204 billion by 2030.
The projection assumes conservatively that each company trains one new flagship model annually at this rising cost. Adding 20% for other models and additional training, based on Epoch’s spending estimates, produces a combined five-year bill of approximately $1 trillion for final training alone.
The much larger expense is the research surrounding those runs. Epoch AI’s estimates suggest that final training runs for released models accounted for less than 10% of OpenAI’s research-computing spending in 2024 — approximately $480 million out of $5 billion.
Total research computing therefore costs roughly ten times as much as final training. Applying that ratio produces the projected $10.7 trillion bill. This assumes that all three companies maintain a similar spending pattern and value their computing at comparable rental prices.
Could the companies earn enough to pay for this expansion? Their own forecasts show the strain. OpenAI’s projections reported in February anticipated spending $218 billion more than it generated during 2026–2029. Anthropic has reportedly posted a quarterly operating profit on an adjusted basis, but the research bill projected here is on a different scale.
The Byline Times calculation gives both companies room for substantial growth. It assumes OpenAI reaches its reported annual sales target of $280 billion by 2030, while Anthropic reaches its reported $200 billion target by 2028, then grows another 50% annually.
Allowing for growth towards those targets, total sales during 2026-2030 would reach approximately $550 billion for OpenAI and $960 billion for Anthropic, in 2023 dollars. Each would face an estimated $3.6 trillion research-computing bill. Even spending every dollar of sales on research, leaving nothing for staff or serving customers, would fall far short.
Alphabet has more money to draw on. Byline Times’ calculation starts with the cash generated by its businesses, including money already spent on research so that this expense is not counted twice. Assuming annual growth of 20-30% gives it approximately $1.7-2.3 trillion over five years, before paying for new infrastructure. Its projected research-computing bill is also $3.6 trillion.
Together, the companies would generate approximately $3.2–3.8 trillion under these assumptions, leaving a $6.9–7.5 trillion gap. Closing it would require existing reserves, financing already secured, further investment, higher income or reductions in projected costs.
Cheaper Rivals Keep Pace
Chinese developers including DeepSeek, Alibaba and Z AI offer increasingly capable alternatives that customers can download, adapt and run themselves or through another provider. These models do not necessarily disclose all their training data or methods, but reduce dependence on the original developer and make premium prices harder to justify.
Anthropic’s plan to sell shares on the stock market could put the company’s value at $2 trillion or more. Sustaining such expectations becomes harder if competitors can offer capable alternatives at much lower prices.
Proposed regulations would have the effect of allowing Western companies to coordinate development while holding back a source of cheaper competition.
Jack Clark’s proposed ‘kill switch’, verified by a third party such as METR, could also burden smaller competitors while strengthening industry insiders’ influence over oversight.
“I firmly believe that independent evaluation is worth having,” said Daanish Masood. “My concern is how the rules are written. A safety regime can reduce real risks while favouring the largest labs if its costs and requirements are disproportionate. The rules should track the actual risks, not make size and proximity to government the price of participation in the industry.”
The former UN AI lead said Beijing would see a proposal combining mutual restraint with chip restrictions as an attempt to “lock in a hierarchy” with the US on top. “That makes a broad agreement difficult,” he said. “But crucially, it doesn’t mean narrower safety cooperation is pointless.”
He added that: “The ultimate test of pacing is what happens when a safety finding conflicts with a commercially important deadline. Can independent scrutiny actually change the decision? Without that, pacing risks becoming reputational insurance.”
For leading companies facing high costs and capable rivals, such measures could provide a form of political bailout: protection for their market position and reassurance to investors about their long-term importance. Presenting AI as almost all-powerful can also strengthen the investment case for the companies developing it.
“Basically the competition is resulting in huge training costs and they are burning through cash and having to do increasingly unethical stuff, e.g. all the secret book buying, to get an edge. I think this scaremongering is for commercial reasons to get Formula-1 style regulation to control how competitors can spend”, said Mohammed Marikar.
Anthropic, OpenAI and Alphabet did not respond to requests for comment.
Ideological Roots of Doom
Commercial interests do not mean extinction fears are insincere. Those convictions may also have ideological roots.
Philosopher Émile Torres, a former participant in the transhumanist movement, argues that leading AI developers and their most vocal safety critics emerged from overlapping circles of researchers, donors and campaigners with a shared ambition: creating machines powerful enough to transform humanity’s future.
Torres and Timnit Gebru, former co-lead of Google’s Ethical AI team, call these movements ‘TESCREAL’. They include transhumanism’s pursuit of overcoming biological limits and longtermism’s emphasis on vast future populations. The authors trace transhumanism’s origins to eugenics and its pursuit of a supposedly superior machine-merged humanity.
In Torres’s account, Dario Amodei and AI-risk researcher Eliezer Yudkowsky share the ambition of a civilisation transformed by superintelligence and extending beyond earth, despite disagreeing about how safely it can be reached.
This may help explain the unswerving conviction behind extinction warnings, despite a complete failure to scientifically substantiate related probability estimates. If superintelligence is assumed to hold the key to an immense future civilisation, even a speculative threat can become an overriding priority. That urgency follows from the worldview; it does not establish that today’s machines are approaching the capabilities needed to cause human extinction.
Nafeez Ahmed is Head of Investigations at Byline Times.
This podcast by Jim Stewartson is well worth listening to (there's no transcript unfortunately). He thoroughly takes apart what's going on with these AI techbros, with all the hype and obfuscation. It's clear that nothing is as it seems. The comparison to Enron is particularly fascinating.
https://substack.com/@jimstewartson/note/p-215749424?r=1w1c61