BYLINE TIMES IMPACT: Find out about the changes you made happen
The Warning
There is something deeply strange about the way we talk about artificial intelligence.
We are increasingly being warned that the technology we are rushing to develop could pose an existential threat to humanity. Not by science-fiction, but by the people who actually built these systems.
Engineers and researchers from companies like OpenAI and Anthropic have warned that increasingly capable AI could cause catastrophic harm including environmental catastrophe, biological warfare and active war dangers, as well as the possibility of human extinction by 2030. These aren’t predictions we can dismiss as the ravings of technophobes; they reflect a critical debate within the industry about what happens when system capabilities outstrip our ability to contain them.
These systems are no longer merely sophisticated chatbots. AI agents built by humans are now acting autonomously: navigating computers, writing code, accessing sensitive data, and pursuing goals with minimal human supervision. Researchers have already demonstrated systems exploiting software vulnerabilities and behaving in ways their creators did not anticipate. Anthropic has even faced scrutiny over its refusal to submit a recent model for evaluation by the UK’s AI Security Institute. They aren’t required to, so why would they?
None of this proves AI will wipe us out. But if there is even a meaningful possibility of catastrophic harm, the rational response is to assess that risk before accelerating further. Instead, we are building systems faster, granting them more autonomy, and competing to make them more capable all while the people building them admit we may not understand where this ends.
Which leads to an urgent and quite sophisticated question: what the f**k are we doing?
The Race
If we genuinely believe AI could become an existential threat, why are we treating its development as a race?
The world’s most powerful companies are competing to build increasingly capable systems, while governments scramble to ensure their domestic champions don’t fall behind.
The logic is depressingly familiar: if OpenAI slows down, Anthropic moves ahead; if the US hesitates, China takes the lead. Every individual decision is justified in isolation as necessary for competitiveness or responsible development.
And collectively, those decisions produce pure acceleration.
Nobody wants to apply the brakes first. We are repeatedly told AI will transform medicine, boost productivity, and solve intractable problems. Those possibilities might be real, but potential benefits do not make catastrophic risks vanish.
Our current model lets profit-driven companies set their own safety limits. That isn’t a safety system—it’s a commercial free-for-all.
We’ve Seen This Before
There is a reason this all feels vaguely familiar. We have been here before. The closest parallel may be the climate crisis – not because AI and climate change are remotely the same problem (though the former contributes to the latter), but because the political failure is remarkably similar.
For decades, we knew that burning fossil fuels at an enormous scale was changing the climate. Scientists warned governments. Governments commissioned reports. Researchers refined the models. The evidence accumulated. And still, we kept going.
The problem was not simply that politicians didn’t understand the science. It was that acting on the science required confronting some very powerful interests and accepting some very unpopular short-term costs.
Fossil-fuel companies had enormous financial incentives to keep extracting and selling fossil fuels. Industries depended on cheap energy. Politicians worried about jobs, prices and economic growth. Governments competed with one another. Lobbyists worked to weaken regulation. And every individual decision to approve another project, delay another target or water down another policy could be presented as perfectly reasonable.
We could deal with the problem later. Later, unfortunately, is now. We are living with record temperatures, increasingly severe weather, pressure on food and water systems and ecosystems under extraordinary strain. None of this was unforeseeable. The tragedy of climate change is precisely that so much of it was foreseen.
And this is where we should be extremely wary of repeating the same mistake with AI. We are already hearing the warnings. We are already being told that increasingly capable systems could create risks that are difficult to control. And we already have an industry worth hundreds of billions of dollars whose commercial success depends on developing those systems as quickly as possible. Why would we expect the political incentives to be any different?
We shouldn’t.
This isn’t about assuming Sam Altman, Dario Amodei, or Elon Musk are inherently evil (though they do have a Bond villain vibe). The problem is structural. Companies are built to compete, grow, and create shareholder value – not to make decisions for humanity as a whole. Even well-intentioned executives – if they exist – face pressure from investors, competitors, and governments.
If restraint looks like unilateral disarmament, dangerous systems become normalised because everyone has a plausible excuse to keep going. We saw this dynamic play out with tobacco, financial markets, opioids, and fossil fuels: private profits are concentrated, while eventual costs are distributed across society. We are remarkably bad at regulating powerful industries before their risks become impossible to ignore. AI gives us a rare luxury: a warning before the consequences arrive.
Getting Serious
Taking the risk seriously doesn’t mean banning AI or halting progress every time a risk is named. The principle is far simpler: if we don’t know whether something is safe, we shouldn’t make it exponentially more powerful just because a competitor might get there first.
Taking the risk seriously means putting genuinely independent safety assessment ahead of commercial competition. The most powerful models must be tested before deployment, dangerous capabilities disclosed, and systems scrutinised by external researchers without relying on corporate goodwill.
Governments must establish binding, universal limits on AI development and they must do it now.
We cannot build public safety around the assumption that private companies will voluntarily sacrifice profits in the name of humanity. We don’t do this with nuclear power, pharmaceuticals, or aviation. We create rules, regulators, and independent oversight – and crucially, we reserve the right to say no.
Inevitability in tech development is not a law of nature; it is a political choice and a marketing frame to force acquiescence. We can decide that certain risks are unacceptable and that the pace of development must be set by what we can safely govern, not what market competition dictates.
The Race To Where, Exactly?
The most uncomfortable truth is that we don’t know where this race ends or why we are running it. AI may well help treat diseases, accelerate discovery, and improve lives. It may also lead to species extinction. Those outcomes are not mutually exclusive but the former don’t really matter if we’re all dead, which is why blindly accelerating toward maximum short-term profit is reckless.
If the goal of technology is human flourishing, then flourishing must be what we optimise for. And if a technology genuinely threatens human survival, then surely survival must be our non-negotiable constraint.
With AI, we have been given an extraordinary chance to intervene before the crisis hits.
We don’t know if the worst-case scenarios will materialise, but if there is even a meaningful chance they will, we must stop asking how fast we can build the future and start asking what kind of future we are actually building.
Watch Kyle Taylor in conversation with Adrian Goldberg on the Byline Times podcast here.
The parallels are there with nuclear - energy and weapons. The potential for good and the potential for utter disaster, literally destroying the planet.
A key difference is that AI is in the hands of a very few people, who have been driven primarily by profit, egged on by greedy finance. Both groups using their wealth to grab political power and block the regulation that is really needed.
Nuclear has by and large stayed in the hands of state actors. Whilst they are by no means all well intentioned, there is not that same lust for personal reward and power. Where nuclear power is concerned, the idea that there might be a 10% chance of disaster would be seen as utterly unacceptable
When you look into what the companies were actually doing when they allowed their products to "go rogue", it was so unutterably stupid, you wonder why anyone believes anything they say, or leaves them in charge of the tea fund, never mind a whole massive company.
According to a very well-informed podcast (https://podcasts.apple.com/us/podcast/no-ai-is-not-autonomously-hacking-with-cal-newport/id1730587238?i=1000785935670) that I found because Cory Doctorow linked to it, what they actually did was used an LLM to generate ideas of how to hack a file, and kept an "agent" following the LLM's instructions to the letter for days, no matter how deeply idiotic they were.
We all know that LLMs "hallucinate" (ie make things up) all the time, and anyone with any sense examines their proposed course of action carefully before following it. LLMs don't understand what they're proposing, they just provide examples of what the data they were fed might suggest might work - and there's no reason to think that what went into those datasets actually did work, or that the circumstances were in any real way similar.
The claims of "autonomous hacking" are just a sales pitch, to try to keep the investments rolling in. Because the moment enough people see what's really going on, the bubble will burst.
Don't get me wrong, specialised AI that doesn't just depend on LLMs is powerful & could be dangerous, although it's where the real value is, too. But that's not what these idiots were using, and it's not where the big investments are being made.