12 Oct 2026 8 min read

Who Decides How Much AI Risk Our Children Should Face?

Families can’t opt out of potentially catastrophic AI risks. Who decides what risks children face, and how should governments protect them as AI becomes more powerful?

When a child needs medicine, a parent might ask a doctor or pharmacist about side effects, read the leaflet, or want reassurance that the treatment is suitable for their child. We understand that medicines can carry risks, even when they offer real benefits, and we expect those risks to have been carefully examined before a treatment reaches our family.

We don’t rely solely on the company making the medicine to decide whether it is safe enough to use. Public regulators, with legal powers, examine the evidence and decide whether a new medicine can be approved and for which uses. They continue to monitor its safety and can restrict its use or take it off the market if the evidence warrants it. These principles underpin medicines regulation in places including the European Union and the United States.

Similar expectations apply in banking and nuclear power, where public authorities oversee risks that can extend far beyond a company’s customers. Parents don’t personally determine those safety requirements, but there are institutions with legal responsibilities to protect children and the public on their behalf.

As companies develop increasingly powerful AI, families should be able to expect effective protection here too. Yet too much discretion over decisions that could affect children’s lives and futures is being left with the companies creating the risks.

Risks families can't opt out of

Children are already exposed to serious AI-related harms, from unsafe chatbot responses to tools that others can use to turn their photographs into sexualised images. Parents can take some precautions, but protecting children largely depends on companies making their systems safe and governments enforcing safeguards.

Families have even fewer ways to protect themselves from the wider risks of increasingly powerful AI, such as serious disruption to healthcare, electricity, or water supplies.

“Frontier AI” refers to the most capable general-purpose systems being developed. Alongside their potential benefits, risks include their ability to assist serious cyber attacks or biological misuse, and the possibility that increasingly autonomous systems could take harmful actions that people cannot reliably oversee or stop.

The consequences could reach children through the services and care they depend on. Prolonged disruption to hospitals could delay urgent treatment. Failures affecting electricity, water, and communications together could leave families struggling to meet basic needs, with children who depend on powered medical equipment facing additional danger. A major biological incident could cause widespread illness and overwhelm health services.

These pathways involve different evidence and uncertainties, examined in SAIFCA’s catastrophic-risk guide. The possible consequences include large-scale loss of life and lasting harm to the conditions in which children grow up.

A parent cannot meaningfully accept or refuse this exposure. Their family’s safety depends on decisions being made elsewhere, often by organisations whose products they have never chosen to use.

What the people building AI have acknowledged

Warnings about potentially catastrophic harm have come from researchers studying AI safety and capabilities, and from leaders of the companies developing these systems.

In 2015, before becoming chief executive of OpenAI (the company behind ChatGPT), Sam Altman suggested that AI would probably lead to the end of the world, while also creating successful companies along the way. In 2023, as CEO of OpenAI, he described the bad outcome from AI as “lights out” for all of us.

In 2023, Dario Amodei, chief executive of Anthropic, the company behind Claude, put his estimate of something going catastrophically wrong on the scale of human civilisation at between 10% and 25%. In 2024 Elon Musk discussed a 10% or 20% chance of an outcome ending humanity.

These figures were personal estimates, not scientifically established probabilities of catastrophe, but their statements nevertheless show that the prospect of extreme harm has been recognised by people leading the industry, while their development towards more powerful systems has continued.

More recently, Sam Altman argued that society should accept some harmful outcomes in exchange for AI’s benefits. This leaves families dependent on decisions about which risks are acceptable and how serious they could become - those judgements should not be left to developers alone.

For a parent hearing these statements, there is a reasonable question about who has the authority to make such trade-offs. Recognising a danger, or saying that it is unacceptable, doesn’t establish that it has been adequately controlled. Families need protection that can be independently assessed and enforced.

Who is making the decisions now?

AI companies operate within laws that differ between countries, and there are some government evaluations, regulatory requirements, and company safety frameworks already in place. However, these arrangements do not consistently give public authorities independent access and enforceable control over some of the most consequential development and release decisions.

Where binding requirements are missing, developers have considerable discretion over how they assess risk, what safeguards they consider sufficient, and whether to proceed.

Families are consequently exposed to risks they have neither agreed to nor been given a practical way to refuse, as companies pursue ever more powerful technology.

The benefits promised by developers don’t remove governments’ responsibility to protect those who could bear the consequences.

Voluntary commitments cannot provide the same assurance as enforceable obligations. In September, leading AI companies signed a voluntary US pact, with President Trump praising industry self-policing. Such arrangements leave an urgent question about who can compel action when a company’s precautions are inadequate.

Competition makes public oversight more necessary

Companies compete for investment, customers, and technical advantage. Governments want economic growth and fear dependence on technology controlled elsewhere. These pressures can discourage companies and governments from slowing development, even when serious concerns have been recognised.

A company might argue that a less cautious competitor will proceed if it does not. A government might fear losing investment or strategic advantage by introducing stronger requirements. If each points to the others as a reason to keep going, families are left exposed to the collective result.

Children’s safety cannot depend on every participant voluntarily resisting those pressures. Shared, enforceable requirements can help prevent companies gaining an advantage through weaker safeguards, while international cooperation is necessary because both development and its consequences cross borders.

What governments need to do

On 12 October 2026, the UK government announced measures to strengthen cyber defences and prepare for serious incidents involving autonomous AI. AI minister Kanishka Narayan told developers, “if you cannot adequately control the risks, do not release the system.” The published announcement does not, however, specify a new enforceable mechanism to ensure companies follow this instruction.

It also describes work to explore stronger legal protections and reporting of incidents and near misses. These are welcome steps, but families need to know what requirements will follow, when they will take effect, and who will enforce them.

The minister’s statement raises the practical questions central to this article. Who determines whether risks are adequately controlled, and who can stop a company proceeding if they are not?

SAIFCA’s position is that governments should establish enforceable limits on risks to the public, informed by independent technical evidence and the interests of those who could be affected. Responsible authorities need the expertise, access, and legal powers to apply those limits, while companies should have to demonstrate that they meet them.

This means scrutiny early enough to address dangers during development as well as before release. Where credible serious risks cannot be adequately assessed or controlled, authorities should be able to require changes, restrict access, or prevent the relevant development or deployment. Testing must have consequences for what companies are permitted to do.

Preparing essential services for disruption is also necessary, but families should not have to rely on emergency arrangements as their principal protection from avoidable risks.

Children’s safety is a public responsibility

Governments should not be able to delegate their responsibility to protect children to the companies developing AI.

In medicine and other regulated sectors, public authorities are expected to examine evidence, impose conditions, and act when safety requirements are not met. Families should be able to expect effective public protection as AI capabilities advance, including when commercial and geopolitical pressures favour moving faster.

Through its governance recommendations and public statement, SAIFCA explains how these risks could affect children and builds support for enforceable protections.

Parents should not have to accept preventable harm to their child, or potentially catastrophic risks to children collectively, as the default consequence of decisions made by AI developers. They should be able to rely on governments to set and enforce boundaries, with children’s safety influencing what is permitted before harm occurs.