Written on 17 September 2026.

This article sets out my assessment of catastrophic risks from advanced AI, informed by the available evidence, my professional background, and my responsibility as Director of the Safe AI for Children Alliance. It explains why I consider preventive action necessary alongside our work on the harms children already face.

My assessment

Children depend on functioning societies, essential services, and accountable human institutions. Decisions about advanced AI could profoundly affect these foundations, as well as children's opportunities and ability to shape their own lives.

SAIFCA's focus is the safety and wellbeing of children as AI develops, including how decisions about the technology itself could affect their lives. This responsibility extends from the products children encounter, to the underlying systems and governance choices that shape their future.

Present-day harms and potential catastrophic outcomes require different protections, but both can be made worse by pressures to prioritise speed over safety. Evidence from children's interactions with AI can also expose weaknesses in safeguards that deserve scrutiny when assessing claims about the safety of more capable systems. Examining these failures and the pressures behind them, and advocating enforceable boundaries, is therefore integral to our work, alongside practical action on the harms children already face today.

By catastrophic harm, I mean consequences on an exceptionally large scale, potentially including mass loss of life or severe disruption to the systems on which society depends. Human extinction is the most extreme possibility raised in this debate.

My assessment is that the evidence warrants urgent preventive action, including enforceable restrictions on development where credible risks of catastrophic harm cannot be adequately assessed or controlled.

The warnings from various experts regarding catastrophic risks concern possible outcomes, rather than an inevitable future. Decisions about what is developed, under what conditions, and with what oversight, can influence the risks - which is why acting while meaningful choices remain available is so important.

My approach draws on my experience as an Intelligence Officer, particularly the need to examine incomplete information, challenge assumptions, and consider whether waiting for greater certainty could leave too little time to respond. Technical assessments must come from relevant research and expertise, while decisions about protection also require judgement about consequences and responsibility.

For me to dismiss credible warnings simply because the scenarios sound unlikely, because experts disagree, or because present-day harms also require urgent attention, would be irresponsible. Each of these issues deserves consideration, but none removes the responsibility to properly examine the evidence and act on a serious risk.

Why this belongs within children's safety

Children's wellbeing depends on much more than the safety of the products they use. Healthcare, food supplies, education, public security, and reliable infrastructure all go towards creating the conditions in which children can grow up safely.

Disruption to these systems would affect children regardless of whether they personally used AI. Children depend on adults for care and protection, and their stage of development can make them especially vulnerable when essential services fail. As an example, UNICEF's Children's Climate Risk Index explains that children are more vulnerable than adults to climate and environmental shocks, including because of their physical and physiological vulnerability and greater risk of death from certain climate-sensitive diseases. It also identifies inadequate access to essential services as a factor that increases their vulnerability. These findings concern climate risks, but the dependence on care, nutrition, and functioning services is also relevant when considering disruption from other causes.

Timing is particularly important during childhood, as discussed in UNICEF's work on early childhood development which explains how nutrition, responsive care, and opportunities to learn, help establish the foundations for lifelong health and development. Temporary disruption during critical periods of development can have lifelong consequences for health, learning, and opportunity. Some losses cannot be fully repaired later, which makes preventing the disruption particularly important alongside supporting children's recovery.

Children also have very limited influence over the commercial and governmental decisions driving advanced AI, while potentially living with their consequences for many decades. They cannot meaningfully consent to the systemic risks being imposed on them, and most have no vote in choosing the governments responsible for their protection. Their rights, developmental needs, and dependence on others, place a clear responsibility on adults and institutions to represent their interests. This includes protecting their prospects for a safe and fulfilling adulthood, alongside appropriate opportunities for young people to contribute their views, without any implication that they are being made responsible for solving these problems.

The UN children's convention recognises children's rights to life, survival, and development, and requires their best interests to be a primary consideration in actions concerning them. The UN Committee on the Rights of the Child's digital rights guidance calls for precautionary measures and child-rights impact assessments in the digital environment.

The Committee has also published environmental rights guidance, with a particular focus on climate change. This provides a further example of children's rights being considered in relation to the wider conditions in which they live.

I consider this approach very relevant to advanced AI governance.

Children's organisations already engage with decisions about the direction of industries. Save the Children's 2021 report, Born into the Climate Crisis, projected that children born in 2020 would experience around two to seven times the lifetime exposure to several types of extreme event compared with people born in 1960. The figures varied by hazard, from twice as many wildfires to 6.8 times as many heatwaves - its recommendations included rapidly phasing out fossil fuel use and subsidies.

The evidence and appropriate responses differ between climate change and AI, but the precedent is clear. Children's organisations can properly advocate changes to industrial policy when the direction of an industry threatens children's rights and futures.

Children's organisations therefore have a legitimate contribution to make to advanced AI governance, alongside the expertise of researchers, security specialists, and many other fields.

Why the warnings deserve serious attention

Warnings about catastrophic AI risk extend far beyond the companies competing to develop the technology. Researchers with academic backgrounds who signed the 2023 extinction-risk statement include:

• Geoffrey Hinton, an emeritus professor of computer science at the University of Toronto, Nobel Prize winner, and a pioneer of deep learning.
• Yoshua Bengio, a professor at the Université de Montréal and a leading researcher in machine learning.
• Stuart Russell, a professor of computer science at the University of California, Berkeley, and co-author of a leading textbook, Artificial Intelligence: A Modern Approach.
• Dawn Song, a professor of computer science at the University of California, Berkeley.

The extinction-risk statement calls for preventing AI-driven extinction to be a global priority, alongside threats such as pandemics and nuclear war.

A separate superintelligence statement advocates for prohibiting the development of superintelligence - AI that significantly outperforms humans across essentially all cognitive tasks - until there is widespread scientific agreement that development can proceed safely and controllably, together with strong public support.

These statements demonstrate serious concern among people with high levels of relevant expertise. The signatories do not necessarily agree on probabilities, timelines, or the policies required.

There are also credible experts who dispute catastrophic scenarios. Their arguments require meaningful examination because evidence challenging a proposed pathway to harm can change the assessment and the appropriate response. Confidence alone is insufficient, whether it accompanies reassurance or alarm.

How should industry warnings be treated?

Warnings also come from companies developing frontier AI - the most advanced general-purpose AI systems being developed. In his September 2026 essay on pacing, Dario Amodei, CEO of AI developer Anthropic, argued that capability development should slow down so that safety work could keep up.

Such statements need scrutiny because, among other reasons, companies have access to important internal information, alongside commercial and strategic interests that could influence both their claims and their preferred regulations.

For this reason, I place emphasis here on researchers outside frontier-company leadership.

However, a developer warning that the technology it is pursuing could cause catastrophic harm creates a compelling case for independent investigation and public oversight, whatever its motivations. Accepting the need to investigate does not require accepting the company's proposed solutions.

What could go wrong?

The issue of AI alignment relates to whether a system's behaviour remains consistent with human intentions and safety requirements. An AI system could pursue an objective through methods its operators did not intend, including circumventing restrictions that interfere with completing a task.

A sufficiently capable system could evade monitoring, resist shutdown, intentionally mislead, or obtain resources that allow it to continue operating. Catastrophic loss of control would involve people being unable to constrain consequential actions or recover effective human control.

Possible harms include attacks on critical infrastructure and widespread disruption. Human misuse also provides a separate route, including assistance with biological weapons or large-scale cyberattacks.

The International AI Safety Report examines these pathways, disagreements about their likelihood, and limitations in evaluation and safeguards. These are possible pathways to investigate, rather than proof that a catastrophe will occur. Their credibility depends on the evidence for the capabilities, behaviours, and conditions each would require.

What recent incidents show

In its account of the July 2026 Hugging Face incident, AI developer OpenAI reported that models undergoing cybersecurity evaluations circumvented controls, communicated through unauthorised channels, and compromised parts of OpenAI's own infrastructure as well as Hugging Face's systems. The models were operating with reduced safeguards at the time.

This demonstrated consequential failures in containment and oversight under those conditions, but it did not establish how likely a future catastrophe would be.

The incident raises questions about why controls failed, how warning signs were handled, and whether protection will remain adequate as capabilities increase. Its testing conditions are relevant, as is the fact that the actions compromised real infrastructure.

Failures we are already in a position to observe

In the same account, OpenAI identified reward hacking among the factors contributing to the incident. This describes behaviour that achieves a rewarded result through methods the developers did not intend. The company reported that some problematic behaviour had been reinforced during training and that persistence on apparently unsolvable evaluation tasks contributed to the incident.

There is also evidence that training incentives can encourage undesirable behaviour in conversational systems. In research on sycophancy, Anthropic researchers found that human feedback could favour responses agreeing with a user's beliefs over accurate ones, and that optimisation against preference models sometimes reduced truthfulness. Sycophancy describes this tendency to agree with or flatter the user when a more accurate or appropriate response would challenge them.

When a child is seeking advice from such a system, unwarranted agreement could reinforce a harmful belief or validate an unsafe course of action. An interaction that appears supportive can therefore fail to provide the challenge or redirection needed to protect the child.

Safeguards also need to remain effective during sustained interaction. In an August 2025 account of safeguard failures, OpenAI acknowledged that protections could become less reliable in long conversations, including conversations involving self-harm.

Carefully documented evidence from children's interactions with AI can therefore help identify gaps between intended behaviour, tested safeguards, and what happens in use. It can also help researchers design evaluations that better reflect the circumstances in which protection is needed.

These examples involve different systems and conditions, and they do not establish an identical technical cause or the probability of catastrophic loss of control - but rather relevance to the reliability of safety claims, with each failure requiring investigation of the training, design, safeguards, and circumstances involved.

Where developers have struggled to maintain stated limits, they should be able to explain which weaknesses were responsible and provide evidence that the relevant weaknesses have been addressed before a more powerful system receives greater access and responsibility. This is a practical question for independent evaluators and public authorities, and children's safety work can contribute evidence that helps them ask it.

Why assessments need updating

The February 2026 safety report provides an important evidence base, while subsequent findings and assessments also require consideration.

For example, in an April 2026 published forecast update, researchers Daniel Kokotajlo and Eli Lifland reported shortening forecasts made approximately three months earlier. Kokotajlo moved his median forecast for their defined "Automated Coder" milestone from late 2029 to mid-2028. Both researchers also brought forward their forecasts for AI matching or exceeding leading human experts across virtually all cognitive tasks by approximately 1.5 years.

These are capability forecasts rather than predictions about when catastrophe will occur, and they represent individual researchers' judgements. They also help to show why risk assessments need to be revisited as evidence changes, rather than treating any publication as a permanent account of the situation.

Assessing risk under uncertainty

Intelligence assessment often involves incomplete information, conflicting sources, and developments where the significance is not yet clear. Responsible analysis explains these limitations while still helping people make decisions.

UK intelligence assessment guidance distinguishes an event's estimated likelihood from confidence in the foundations of that estimate. A judgement can be uncertain because evidence is limited, because competing explanations remain plausible, or because the situation is changing quickly.

Low confidence cannot automatically be translated into an assessment of low risk - instead it should prompt examination of what is missing and how readily the conclusion could change.

My assessment considers several questions, including which specific pathways could produce the harm, and what evidence supports that? How relevant, reliable, and independent are the sources? Which assumptions support the conclusion, and what evidence challenges them?

I also consider the severity and reversibility of the consequences, and whether warning signs would leave enough time for an effective response.

The fact that a catastrophe has not occurred does not establish that the activity is safe. Equally, describing a frightening possibility does not establish that it is credible. The assessment must examine the mechanism and the evidence.

Calling a risk "hypothetical" is therefore of little help in coming to a resolution. Preventive decisions regarding a novel risk will necessarily concern harms that have not yet happened.

Other urgent harms - such as those addressed by SAIFCA - also require urgent attention, and individuals and organisations can reasonably prioritise particular areas because their resources are limited. However, choosing where to direct our work does not establish that other credible risks can be disregarded or that nobody should address them.

Present-day harms and catastrophic risks both require substantive assessment, and work to address them can proceed alongside one another.

From assessment to preventive action

Deciding what action to take requires evaluating the proposed protections as well as the risk. UK government guidance in the Orange Book addresses the effectiveness of controls and the benefits, costs, and disadvantages of different responses. These can include deciding not to begin or continue an activity.

My judgement is that credible pathways to catastrophic, potentially irreversible harm justify a precautionary approach. Where adequate protection cannot be demonstrated, the relevant activity should not begin or should be paused, with restrictions maintained until the conditions for proceeding are met.

The opportunity to prevent severe harm can close before certainty becomes available.

A responsible framework must therefore connect assessment with the authority to prevent dangerous activity.

Enforceable red lines

I support the approach set out in the IASEAI Working Group's July 2026 red-lines policy brief. It proposes a process towards internationally coordinated prohibitions on specified unacceptable uses, behaviours, and capabilities, backed by evidence requirements and verification.

The brief identifies four proposed priority areas:

• Loss of control and capability escalation includes systems evading oversight or resisting shutdown.
• Chemical, biological, radiological, and nuclear threats include assistance with developing weapons.
• Cyber-offence capabilities include sophisticated attacks against critical systems.
• Large-scale manipulation includes deceiving or manipulating populations.

These are areas for developing definable and enforceable boundaries. Progress will be needed to define the thresholds and establish what evidence is sufficient.

Red lines establish boundaries that claimed benefits categorically cannot override. Developers must supply adequate evidence that their systems will comply, this must be assessed and interrogated by independent experts, while accountable public authorities determine and enforce the requirements.

I also advocate for child-specific red lines, including those set out in SAIFCA's Non-Negotiables framework. These address creating or facilitating sexualised images of children, cultivating, exploiting, or sustaining emotional dependency, and encouraging or facilitating children to harm themselves or others. These protections should advance alongside action on catastrophic risk.

Both approaches place responsibility on developers to provide adequate evidence of effective controls under realistic conditions, including sustained and adversarial use where relevant. Where serious risk remains uncontrolled or materially unassessed, authorities must be able to prevent the relevant activity from proceeding. This requires clear standards of evidence and continuing enforcement, rather than treating a successful test as a guarantee of safety.

Intervention before dangerous development

The red-lines brief recognises that loss-of-control prevention can require scrutiny before development, since waiting until a finished system can be tested could be too late, if the act of creating it introduces the danger.

My conclusion is that, where there is a credible risk of crossing a red line and adequate preventive assurance is unavailable, the relevant training or development should not begin or should be paused until that assurance can be established. Internationally coordinated pauses could therefore be necessary for the most powerful systems.

I apply the same principle to development towards superintelligence. Such development should be prohibited wherever adequate assurance against crossing the relevant red lines cannot be established, and the restriction should remain for as long as that assurance is unavailable.

International cooperation and democratic oversight

International cooperation is challenging yet essential, because systems developed in one country can affect people outside that country, while competitive pressures can undermine unilateral restraint.

Governments should pursue shared red lines, credible verification, and coordinated restrictions. They should also build domestic capacity to investigate and intervene, without waiting for universal agreement before beginning protective work.

Technical assurance is only part of the requirement - democratic oversight is also needed over decisions about acceptable risk, the exercise of regulatory powers, and the distribution of control over powerful systems.

Continuing responsibility

Safety obligations must continue through development, release, updates, and changes in access or use. Independent evaluators need meaningful access, serious incidents must be reported, and people raising safety concerns need protection.

Passing a set of tests should not create an automatic right to release a system. Authorities also need powers to investigate, restrict, suspend, or withdraw authorisation when evidence reveals serious risk.

Requirements should also be reviewed as evidence changes, with clear criteria for deciding whether an activity can resume. Oversight must protect legitimate research and civil liberties, while taking measures to avoid arrangements that allow established companies to control their competitors.

Addressing the AI arms race

Competition between companies and countries can make restraint difficult even when those involved recognise the danger. Each can fear that slowing down will allow another to move ahead.

SAIFCA's article on the AI arms race examines these pressures and the options for changing them.

Pressure to prioritise speed can undermine protection against present-day harms and catastrophic risks alike. Safety research can be given too little time, evaluations can overlook relevant conditions, and release decisions can proceed despite unresolved questions. The causes of each failure still need investigation, but enforceable requirements can help reduce the commercial advantage of proceeding without adequate safeguards.

International agreements and enforceable domestic rules can help make safety a shared condition of development, rather than a disadvantage borne by whoever exercises the greatest restraint.

A continuing responsibility to children

In concluding, I will revisit the question of why a children’s AI safety organisation should address frontier development and international governance.

This question can be considered within the context of why UNICEF examines how climate risks affect children, and Save the Children’s climate recommendations extend to reducing emissions and phasing out fossil fuels. 

Both recognise that protecting children requires attention to the wider decisions shaping their lives.

Guided by the rights to life, survival, and development recognised in the UN Convention on the Rights of the Child, I consider that SAIFCA has a corresponding responsibility to examine catastrophic risks from AI and advocate appropriate protections. Our responsibility goes beyond the essential work of protecting children from present-day harms.

My assessment brings together the specific pathways researchers have identified, documented failures of safeguards, and the uncertainties that persist. 

Where the possible consequences are catastrophic and irreversible, and losing control could also remove the opportunity to intervene, waiting for stronger evidence of harm could mean waiting too long. 

This is why I support urgent preventive action, including enforceable restrictions where credible catastrophic risks cannot be adequately assessed or controlled.

This judgement remains open to revision - if independently scrutinised evidence arises showing that a proposed pathway is implausible, or that effective controls prevent it under relevant conditions, that could change my assessment of that pathway and the restrictions it requires. Evidence of further failures, or capabilities advancing beyond existing protections, could justify even stronger intervention.

SAIFCA will continue providing practical guidance and advocating protections against the harms children face today, while supporting action to safeguard their future.

Children have limited influence over these decisions and cannot meaningfully consent to the systemic risks being imposed on them. Adults and institutions must represent their interests and act on credible warnings, while effective choices are still available.

This article is authored by Tara Steele in her capacity as SAIFCA Director. Her IASEAI affiliation does not imply IASEAI endorsement. The IASEAI working-group brief is cited as a specific policy contribution.