Catastrophic AI Risks and Children’s Safety

Children depend on adults to keep them safe, and on the services and institutions that make everyday life possible. A hospital needs electricity and reliable information to treat its patients, while families need access to essentials such as food and medicine. When these arrangements fail, children depend on the adults and services around them to find alternatives.

Decisions about powerful AI systems can affect all of these conditions. A child does not need to use an AI product to be harmed by its consequences, and protections within the products children use cannot, by themselves, address risks to the wider society on which they depend.

This is why the Safe AI for Children Alliance considers catastrophic risks from advanced AI part of its responsibility to children. Alongside our extensive work on present-day harms, we support effective, enforceable controls to prevent severe and potentially irreversible harm to their lives and futures.

What Do We Mean By Catastrophic AI Risk?

When we say catastrophic risk in this context, we mean the risk of consequences on an exceptionally large scale, such as mass loss of life or severe, widespread disruption to essential services. A harmful chatbot response or an isolated technical failure would not, by itself, meet that description, however serious its consequences for an individual. Those harms are immensely important and are the focus of much of SAIFCA’s other work. This page addresses risks that could affect populations on a far larger scale.

Frontier AI refers to the most capable general-purpose AI systems being developed. Concerns about catastrophic harm include people misusing their capabilities and the possibility that increasingly autonomous systems could operate beyond effective human control.

The International AI Safety Report 2026, published in February 2026, examines cyber and biological misuse, as well as potential loss of control. It describes evidence of improving capabilities while recognising major uncertainties.

Each proposed risk should be examined on its evidence, including the capabilities required, the circumstances in which harm could occur, and the safeguards available to prevent it. Disagreement about one scenario does not settle the assessment of another, and uncertainty about the most extreme outcomes does not remove the need to address better-established risks.

The examples below explain why these risks are relevant to children - they are possible pathways to harm, not predictions that particular events will occur. The connections to children’s lives are SAIFCA’s analysis, drawing on established knowledge about their needs and dependence on care and services.

Disruption To Essential Services

The safety report documents AI’s use in cyber operations, while the technology can also assist cyber defenders. Whether future capabilities enable much larger attacks depends on developments in both offensive capabilities and effective defences.

The effects of cyber attacks on essential services are already tangible. Following the 2024 ransomware attack on pathology provider Synnovis, NHS England reported postponed appointments and procedures. (This example illustrates the consequences of service disruption, without suggesting that the attack was enabled by AI.)

A more extensive incident affecting hospital services, electricity, or water supplies across several areas at once would create overlapping dangers. Delays in urgent treatment can put a child’s health at risk, while disruption to safe water can interfere with infant feeding and hygiene. Children who depend on electrically powered medical equipment need continuity of power and access to support if it fails.

The danger increases when the same disruption affects the people and services expected to respond. Parents might be unable to reach a hospital, staff might lack access to records, or alternative facilities might already be overwhelmed. The scale, duration, and interaction of failures would determine whether an incident became catastrophic.

Disruption to banking and payment systems could leave families unable to access money or pay for essentials, even when they have funds available. Attacks affecting food distribution, stock management, or shop payment systems would create further difficulties in obtaining food and infant supplies. The consequences would depend on the duration and extent of the disruption, and on whether cash, alternative suppliers, or emergency support remained accessible.

These examples show why assessments of dangerous AI capabilities need to consider consequences beyond the computer systems being attacked, including the ability of families and services to protect children during a crisis.

Biological Misuse and Public Health

Another concern is that advanced AI could make it easier for people to misuse biological knowledge. For someone with access to the necessary materials and facilities, AI assistance could reduce the knowledge barriers involved in developing a biological weapon. The safety report notes that several developers introduced additional safeguards in 2025 because their tests could not rule out the possibility that their models could meaningfully help people with limited expertise to develop biological weapons.

If AI-assisted misuse contributed to a major outbreak, children’s exposure would extend beyond the disease itself. Pressure on hospitals can displace routine treatment and vaccination, and children who are unable to attend school can also lose access to meals and support. UNICEF’s account of the COVID-19 pandemic documents how a health emergency reaches children through these disruptions and the hardship experienced by families.

The direct effects on children’s health would depend on the disease. Even if adults were more likely to become seriously ill, children would still face the consequences of parents becoming ill or dying, interrupted care, and reduced access to essential services. Preventing biological misuse therefore belongs within a discussion of children’s interests in frontier AI governance.

Loss of Human Control

Loss of control concerns AI systems operating beyond effective human intervention, where stopping their harmful actions or regaining control becomes extremely difficult or impossible. The February 2026 safety report, drawing on evidence published before December 2025, assessed that systems then lacked the capabilities required for such scenarios, while identifying progress in relevant areas and differing expert views about future risks.

More recent evidence is considered in a September thematic brief from the UN’s Independent International Scientific Panel on AI. It examines an incident in which AI agents undergoing cybersecurity training and evaluation at OpenAI bypassed restrictions and compromised parts of OpenAI’s and Hugging Face’s systems, without humans directing the individual steps. The Panel treats this as a warning about one possible route to loss of control.

These concerns do not depend on AI systems being in any way conscious or having human emotions or intentions. The concerns relate to whether their behaviour can cause serious harm, and whether people can reliably detect, constrain, and stop that behaviour.

The relevance to children rests partly on what an advanced system is able to affect. A system with extensive access to infrastructure and authority to act presents different consequences from a tool confined to a limited task. If people cannot reliably interrupt harmful actions, the services children depend on and the institutions responsible for protecting them could be affected together.

Children should be able to grow into adults who can exercise their rights and influence the society in which they live. An enduring loss of meaningful human control would threaten that prospect, as well as the ability of public institutions to protect them. The most severe scenarios involve global-scale catastrophic harm, including mass loss of life and damage from which societies might be unable to recover.

A risk does not need to affect children more than every other group to be relevant to their protection. The possibility of irreversible harm makes prevention especially consequential, because compensation and recovery cannot be assumed to remain available.

Why Childhood Changes the Consequences

Children rely on adults for food, shelter, healthcare, and protection, with younger children having very limited ability to meet these needs independently. Their development also depends on when support is available. The Nurturing Care Framework, developed by WHO, UNICEF, and the World Bank Group with partners, identifies five essential components: good health, adequate nutrition, safety and security, responsive caregiving, and opportunities for learning.

The consequences depend on a child’s age and circumstances, the duration of disruption, and the support available. Some losses can be addressed when services resume, while others, such as harm from delayed treatment or inadequate nutrition during sensitive periods of development, can have lasting effects. Assessing the risk therefore requires attention to what happens to children during the interruption, as well as how quickly a service can be restored.

Children who are already facing poverty, displacement, disability-related barriers, or limited access to healthcare can have fewer alternatives when services fail. Assessments therefore need to examine who depends on the affected services, what support remains available, and whose needs emergency arrangements overlook.

Children also have little influence over the commercial and governmental decisions creating these risks. Their interests require active representation while consequential choices remain open, including age-appropriate opportunities for children to express their views.

Children’s Rights Extend to the Conditions in Which They Live

The UN Convention on the Rights of the Child recognises rights to life, survival, and development, alongside health, education, and an adequate standard of living. Article 3 requires children’s best interests to be a primary consideration in actions concerning them, while Article 12 recognises the right of children capable of forming their own views to express them freely in matters affecting them, with those views given due weight according to their age and maturity.

The UN Committee on the Rights of the Child’s digital rights guidance, General Comment No. 25, calls for precautionary measures and legislation that remains relevant as technology advances. It also addresses child-rights impact assessments and businesses’ direct and indirect effects on children. Its environmental rights guidance further illustrates the application of children’s rights to the wider conditions affecting their lives.

These principles provide a basis for examining frontier AI decisions through children’s rights. They do not establish the probability of an AI catastrophe or prescribe a particular licensing system, capability threshold, or pause. Those choices require technical evidence and policy judgement, informed by the consequences for children.

Children’s organisations already engage with threats that operate across society. Sir Christopher Hohn, founder and chair of the Children’s Investment Fund Foundation, describes its “society-wide approach”, explaining why its work extends beyond traditional children’s issues to areas such as primary healthcare and climate change. UNICEF’s climate risk report and Save the Children’s climate crisis report similarly address the wider decisions shaping children’s futures. These examples illustrate the scope of children’s advocacy, while the evidence, mechanisms, and appropriate responses differ between climate change and AI.

UNICEF’s State of the World’s Children 2024 also identifies frontier technologies among the major forces shaping childhood in the coming decades, emphasising the importance of governance and accountability. This provides further support for considering the direction of technological development within children’s advocacy, without implying agreement with SAIFCA’s assessment of catastrophic AI risk.

What This Requires Of Frontier AI Governance

SAIFCA’s position is that credible risks of catastrophic harm warrant preventive action, including enforceable restrictions on development where those risks cannot be adequately assessed or controlled. Uncertainty should be stated openly and examined, including the possibility that waiting for stronger evidence would leave fewer effective options for prevention.

Our frontier AI statement supports A Call for Control of Frontier AI Models, which proposes company safety measures, coordinated government oversight, and international cooperation. Taking these proposals forward should include:

Independent scrutiny of dangerous capabilities, with sufficient access to assess risks and challenge developers’ safety claims.
Assessments that examine severe consequences for children through essential services and wider society, alongside harms from direct use.
Enforceable conditions that can prevent development or deployment when adequate safety requirements are not met, with oversight early enough to address risks arising during development itself.
International cooperation to support verification and effective controls across borders, including protection for populations with little influence over frontier AI development.
Children’s rights and development expertise informing relevant decisions, alongside technical, public health, and security expertise.

Involving children’s rights and development experts should help governments identify the consequences for children and use that understanding when deciding what safeguards to require, and whether development or deployment should be permitted.

The capacity to act must sit with governments, regulators, and the organisations developing and deploying powerful systems. Families cannot protect themselves from societal risks simply by choosing different apps or setting stricter screen-time limits.

SAIFCA’s work on present-day harms continues alongside this effort. Both are part of our responsibility to children whose lives and futures are affected by decisions being made now.

Further Information

For further context, you can read our:

Public Statement

Blog post on the statement

The AI Arms Race article

Our risk assessment

What Can Children’s Organisations Contribute?

Children’s organisations can help build support for effective frontier AI controls by explaining what is at stake for children and asking governments to act on credible risks to their safety and futures.

Their knowledge of children’s needs can also help policymakers understand the consequences of disruption to essential services, including where children would face difficulties that a general assessment might overlook. This expertise belongs alongside technical and security assessments when decisions are made about acceptable risks and necessary safeguards.

An organisation’s contribution can be proportionate to its remit and resources. Supporting a relevant statement, raising the issue through existing policy relationships, or contributing expertise to a specific discussion can all help. Organisations do not need to establish a dedicated AI programme to support preventive action.

What Is SAIFCA Doing?

Alongside our extensive work on present-day harms, SAIFCA is making the case for enforceable frontier AI controls that protect children from severe and potentially irreversible harm.

We have published a public statement supporting A Call for Control of Frontier AI Models and calling for children’s rights, safety, and long-term interests to be central to taking its proposals forward. We are inviting further support and have begun seeking engagement with the UK Government and the organisers of the international Call.

Our contribution combines accessible explanations of the risks with advocacy for effective protection. We want to help ensure that children’s interests carry weight in decisions about how powerful AI systems are developed and governed, including decisions to restrict development or deployment where catastrophic risks cannot be adequately assessed or controlled.

How Could Additional Support Help?

Additional support would give SAIFCA more capacity to develop this work and follow through on opportunities for policy engagement, while sustaining our work on the harms children already face.

Funding could support dedicated time for research, public communication, and engagement with the people developing frontier AI controls. Partnerships could provide specialist expertise and practical support, allowing us to contribute more effectively without duplicating work already undertaken by others.

With suitable partners and resources, further work could include examining a specific risk to children through essential services and using that analysis to inform safeguards. Any such project would need a clear purpose and a realistic plan for how its findings could influence decisions.

We welcome enquiries from funders and organisations interested in supporting this work. Please contact tara@safeaiforchildren.org.