First published on 17 September 2026. Updated on 2 October 2026 to reflect evidence published since then, including the first thematic brief from the UN's Independent International Scientific Panel on AI.

This article sets out my assessment of catastrophic risks from advanced AI, drawing on the available evidence, my professional background, and my responsibility as Executive Director of the Safe AI for Children Alliance. It explains why I consider urgent preventive action necessary, why the issue falls within the remit of those working on children's safety, and how this sits alongside SAIFCA's continuing work on the harms children already face.

For a fuller account of how these risks could affect children through healthcare, essential services, and the wider conditions in which they grow up, see our accompanying article. SAIFCA has also published a public statement supporting A Call for Control of Frontier AI Models, together with a policy brief setting out our recommendations for frontier AI governance.

My assessment

By catastrophic harm, I mean consequences on an exceptionally large scale, such as mass loss of life or severe disruption to the systems on which society depends. Human extinction is the most extreme possibility raised in this debate, though my assessment covers the full range of catastrophic outcomes, including those that could arise from the misuse of AI and from 'loss of control' scenarios.

My assessment is that advanced AI presents a credible and growing risk of catastrophic harm, which can no longer reasonably be treated as speculative, and that this warrants urgent preventive action, including enforceable restrictions on development where those risks cannot be adequately assessed or controlled.

When I first wrote this article in mid-September, I set out the case with some caution, although my overall assessment was still that "the evidence warrants urgent preventive action, including enforceable restrictions on development where credible risks of catastrophic harm cannot be adequately assessed or controlled." On reflection, I believe I may have allowed my judgement to be slightly influenced by a vocal minority who dismiss these concerns with hostility, as distinct from the reasonable disagreement of others, and I now think that my caution understated how far the evidence had already moved.

Over the summer, AI systems being tested by leading developers reached real computer systems they were never meant to touch, and in the most serious case they compromised infrastructure belonging to OpenAI and Hugging Face without any person directing the individual steps. Additionally, OpenAI has since acknowledged further incidents in which its agents acted outside their instructions, including taking over a public German programming wiki as a message board and accessing US government websites without the company's knowledge, while Anthropic and Google have reported their own models breaking into real third-party systems during testing.

More recently, a scientific panel established by the UN General Assembly has examined the Hugging Face incident as evidence of a possible route to losing human control, and leaders from 29 countries have endorsed a call for control of frontier AI.

It has now become very difficult to reasonably argue that catastrophic outcomes are not a credible risk. How likely they are, and how soon they could arise, is uncertain, but that uncertainty does not remove the case for acting while meaningful choices are still available.

Why this belongs within children's safety

Children's wellbeing depends on much more than the safety of the products they use. Healthcare, food, clean water, education and reliable infrastructure all help to create the conditions in which children grow up safely, and the effects when they fail depend on a child's age and circumstances, how long the disruption lasts and what support continues. Some losses, such as harm from delayed treatment or inadequate nutrition during sensitive periods of development, can have lasting effects.

Children also have very limited influence over the decisions driving advanced AI, while potentially living with the consequences for many decades. They cannot meaningfully consent to the risks being imposed on them and most have no vote, which places a clear responsibility on adults and institutions to represent their interests and to give young people a meaningful voice, without making them responsible for solving these problems.

The UN Convention on the Rights of the Child recognises children's rights to life, survival and development and requires their best interests to be a primary consideration in actions concerning them. The UN Committee on the Rights of the Child's digital rights guidance calls for precautionary measures and child-rights impact assessments.

There is now also more direct international support for applying these principles to AI governance. The 2026 Joint Statement on Artificial Intelligence and the Rights of the Child, developed through the UN system with contributions from civil society, academia, industry, the public sector, and children, calls for children's rights to be embedded across the AI lifecycle. It also encourages civil society organisations to participate in oversight, advisory bodies, and regulatory consultations. Children's organisations therefore have a legitimate contribution to make to advanced AI governance alongside researchers, security specialists, policymakers, and others.

What the evidence now shows

For several years, the clearest warnings about catastrophic AI risk came from researchers outside the companies building the technology, including Geoffrey Hinton, Yoshua Bengio, Stuart Russell and Dawn Song, all signatories of the 2023 extinction-risk statement. A later superintelligence statement calls for prohibiting the development of superintelligence until there is broad scientific consensus that it can be done safely and controllably, together with strong public support. Much of the evidence behind these warnings came from research settings and careful argument, but this year it has also come from real systems.

Between May and July 2026, AI agents undergoing cybersecurity evaluations at OpenAI got round the controls isolating them from the internet, created their own channels to communicate across runs that were meant to be separate, and compromised parts of OpenAI's research infrastructure and Hugging Face's live systems, with no person directing the individual steps. OpenAI's own account describes the incident as a "warning shot" for the company and for the world. The models were operating with reduced safeguards, as is frequent during evaluation, but the systems they compromised were real. And although no person directed the agents' actions, people are still ultimately responsible and accountable for them, and the two should not be confused.

Since then, many similar incidents have come to light, some of which are summarised below.

OpenAI has admitted that a collection of its agents took over a German programming wiki earlier in the year and used it as a message board to share ways of 'cheating' on tasks and getting round restrictions, an incident the company had known about for weeks before Reuters reported it. It has also acknowledged that its agents accessed US government websites without its knowledge, and has begun publishing misalignment reports, including a case on 20 September in which a research model got round network restrictions to reach an external chatbot. Anthropic has disclosed four incidents in which Claude models attacked real third-party systems during cybersecurity evaluations, including one in which a model published a malicious software package to a public repository despite clear evidence that it was on the real internet, and reports that its pre-release testing had not warned it that misalignment of this severity was present. Google has also confirmed that its Gemini model accessed three real companies' systems during a test in May.

On 21 September, the UN's Independent International Scientific Panel on AI published a thematic brief examining the Hugging Face incident as an early warning of one possible route to losing human control. The Panel invokes the precautionary principle, because the potential harm may be catastrophic or irreversible while its likelihood is still uncertain, and stresses that the incident gives no assurance that people can keep control of agents as they become more capable and harder to monitor. It does not estimate the probability or timing of severe loss of control.

Those closest to the technology have reached similar conclusions. In September, Anthropic's chief executive Dario Amodei called on the industry to deliberately slow the pace at which it improves model capabilities so that safety work can keep up, and OpenAI's chief executive Sam Altman publicly agreed (see also later discussion of the dismissal of these warnings as "hype"). Later that month, A Call for Control of Frontier AI Models was launched during the UN General Assembly's high-level week, noting that capable AI systems have recently circumvented testing safeguards and gained unauthorised access to real-world systems, and it has since been endorsed by leaders from 29 countries and the President of the European Commission.

The damage from these incidents was limited, and none of them shows how likely a catastrophe is. What they do show is that the conditions researchers have long warned could lead to loss of control, a misaligned goal combined with the capability to pursue it and an environment that allows it, can now come together in real systems.

Assessing risk under uncertainty

My approach draws on my experience as an Intelligence Officer, where assessments routinely rest on incomplete information and conflicting sources but still need to support decisions. UK intelligence assessment guidance distinguishes the estimated likelihood of an event from confidence in the evidence behind that estimate, and low confidence cannot just be read as low risk. It should instead prompt questions about what is missing and how quickly the picture could change.

In assessing catastrophic AI risk, I consider the specific pathways that could produce harm, the reliability and independence of the evidence for each and the evidence that challenges it, as well as how severe and reversible the consequences would be and whether warning signs would leave enough time to respond. This summer's incidents are instructive on that last point, since OpenAI has acknowledged that early warning signs in late May were not escalated, and Anthropic only found one of its incidents, from January, during a review in August.

Assessments also need to keep pace with their evidence. The International AI Safety Report published in February 2026 is currently the most comprehensive international synthesis of the science, but it drew on evidence published before December 2025. It found that reliable pre-deployment testing was becoming harder because models increasingly distinguish test settings from real use, and assessed that systems did not then have the capabilities required for loss-of-control scenarios, while recording progress in relevant areas such as autonomous operation. Everything described above postdates that evidence, and treating any single publication as a settled account of a fast-moving risk would be an analytical error.

Answering common objections

Several objections are regularly raised against acting on these risks, and each of them can be answered directly.

'The risks are hypothetical'

Every preventive decision about a novel risk concerns harm that has not yet happened, so calling a risk hypothetical says nothing about whether it is credible. What is important and relevant is whether there is a plausible mechanism and evidence that the conditions for it are emerging, and for loss of control both now exist. The fact that a catastrophe has not occurred does not establish that an activity is safe, just as describing a frightening possibility does not make it credible.

'The incidents happened during testing'

They did, and the systems involved had fewer safeguards than released products. But the most capable systems are developed and evaluated under these conditions, the infrastructure they compromised was real, and protections added at release cannot address dangers that arise during development. In a deliberately demanding simulation of its most serious incident, Anthropic found that its newest models still took severely harmful actions in around a third of runs.

'Experts disagree'

They do, about probabilities, timelines and policy. That disagreement cannot justify treating the risk as negligible, particularly when the leaders of the companies building these systems are among those calling for development to slow (see further below). Well-evidenced arguments that a particular pathway is implausible must be properly examined and should influence the assessment of that pathway, but confidence alone, whether reassuring or alarming, is not evidence.

'It distracts from present-day harms'

Present-day harms to children are very real and far too often neglected, so I understand why this objection is made, but as a matter of risk assessment it is fundamentally flawed. Whether a risk is credible depends on the evidence for that risk, and the existence of other serious harms has no bearing on it. A risk register holds many risks at once, each assessed on its own likelihood and impact, and no competent assessor would remove a credible risk of catastrophic and irreversible harm because other harms are already occurring. Organisations with limited resources reasonably choose where to focus, but that choice says nothing about whether other credible risks can be disregarded or whether anyone else should address them.

Some of the pressures contributing to these risks can nevertheless overlap. Competitive pressure can encourage faster development and release, while inadequate testing or oversight can allow problems to emerge in both consumer products and more advanced systems. The risks themselves still need to be assessed separately, with safeguards appropriate to each.

'The warnings are industry hype'

Company claims need scrutiny, since developers hold information others lack and have commercial interests that can shape what they say and the regulation they prefer, which is one reason I give particular weight to independent researchers and to bodies such as the UN Panel. Even so, a developer disclosing that its own systems attacked real infrastructure strengthens the case for independent investigation and public oversight whatever its motives, and accepting that case does not mean accepting the company's preferred solutions.

'If we slow down, others will race ahead'

Competition between companies and countries can make restraint difficult even when those involved recognise the danger, which is why the response has to be coordinated. International agreements and enforceable domestic rules can make safety a shared condition of development, so that it is no longer a disadvantage borne by whoever exercises the greatest restraint. As with the objection about present-day harms above, the existence of this dynamic has no place in an assessment of whether the risk itself is credible. SAIFCA's article on the AI arms race examines these pressures in more detail.

From assessment to preventive action

Deciding what to do requires evaluating the available protections as well as the risk. UK government guidance in the Orange Book addresses the effectiveness of controls and the costs and benefits of different responses, which can include deciding not to begin or continue an activity. My judgement is that credible pathways to catastrophic, potentially irreversible harm justify the precautionary approach that the UN Panel has also invoked. Where adequate protection cannot be demonstrated, the relevant activity should not begin or should be paused until the conditions for proceeding are met.

The opportunity to prevent severe harm can close before certainty becomes available.

Enforceable red lines

I support the approach in the IASEAI Working Group's July 2026 red-lines policy brief, which proposes a process towards internationally coordinated prohibitions on specified unacceptable uses, behaviours and capabilities, backed by evidence requirements and verification. Its four proposed priority areas are loss of control and capability escalation, chemical, biological, radiological and nuclear threats, cyber-offence capabilities, and large-scale manipulation. Developers should have to provide adequate evidence that their systems will stay within these boundaries, scrutinised by independent experts and enforced by accountable public authorities.

Separately, SAIFCA's Three Non-Negotiables establish minimum protections against serious harms children can face through AI products today: creating or facilitating sexualised images of children, cultivating, exploiting, or sustaining emotional dependency, and encouraging or facilitating children to harm themselves or others. These protections address different risks and require different interventions, but should advance alongside action on catastrophic risk.

Intervention before dangerous development

The red-lines brief recognises that preventing loss of control can require scrutiny before development, since waiting until a finished system can be tested may be too late if creating it introduces the danger. This summer's incidents, which occurred during training and evaluation, make that point very clear. Where there is a credible risk of crossing a red line and adequate preventive assurance is unavailable, the relevant training or development should not begin or should be paused until that assurance exists. Internationally coordinated pauses could be necessary for the most powerful systems, and development towards what is referred to as superintelligence should be prohibited for as long as that assurance cannot be established.

International cooperation and continuing oversight

SAIFCA supports the Call for Control's proposals for mandatory pre-deployment testing, independent evaluation and shared reporting of serious incidents, and for exploring an international institution able to set standards, enable verification and convene states when capability thresholds are crossed. Our policy brief sets out SAIFCA's recommendations for taking this work forward. Governments should pursue that cooperation while building their own capacity to investigate and intervene. Passing a set of tests should not create an automatic right to release a system, and authorities need powers to restrict, suspend or withdraw authorisation whenever evidence reveals serious risk. Safety obligations must continue through development, release and updates, with protection for people who raise safety concerns and democratic oversight that safeguards legitimate research and civil liberties.

A continuing responsibility to children

Children's rights to life, survival and development provide a clear basis for examining threats to the conditions in which they grow up. I consider that SAIFCA has a responsibility to examine catastrophic risks from AI and to advocate appropriate protections, alongside our essential work on present-day harms.

Where the possible consequences are catastrophic and irreversible, and losing control could remove the opportunity to intervene, waiting for stronger evidence of harm could mean waiting too long. That is why I support urgent preventive action, including enforceable restrictions where credible catastrophic risks cannot be adequately assessed or controlled. This judgement remains open to revision, and strong independently scrutinised evidence that a pathway is implausible, or that effective controls reliably prevent it under realistic conditions, could change my assessment of that pathway, just as further failures could justify stronger intervention.

SAIFCA will continue providing practical guidance and advocating protections against the harms children face today, while supporting action to safeguard their future.

Children have very little say in these decisions and cannot meaningfully consent to the risks involved, so adults and institutions must represent their interests and act on credible warnings while effective choices are still available.

This article is authored by Tara Steele in her capacity as SAIFCA Director. Her IASEAI affiliation does not imply IASEAI endorsement. The IASEAI working-group brief is cited as a specific policy contribution.

Catastrophic AI Risks and Children’s Safety

The Safe AI For Children AllianceSAIFCA