Could We Lose Control of AI? An Explainer for Parents and Educators
Recent headlines about serious AI risks have raised difficult questions for parents and educators. It’s relatively easy to understand how someone could use advanced AI to help carry out a cyber attack, but the phrase “loss of control” can be less clear. How could a computer system take actions that nobody intended, and why might people struggle to stop it?
Increasingly powerful AI could affect the services and institutions on which children depend, whether or not they ever use an AI product themselves. Parents clearly cannot manage these risks through household rules or parental controls. Protection depends on decisions by developers and governments, and families have a legitimate interest in understanding whether those decisions are adequate.
In October 2026, UK AI minister Kanishka Narayan announced preparations for serious AI incidents and told developers not to release AI systems if they cannot adequately control the risks. MI5 director general Sir Ken McCallum discussed the challenge of ensuring AI behaves as people intend, while Google DeepMind’s Demis Hassabis described recent AI hacking incidents as “early warning shots”.
Behind these warnings is a practical problem - AI systems are becoming more capable of taking complex actions, while some methods of keeping them safe have struggled to keep pace. Establishing effective protections takes time, and waiting for severe harm could leave too little opportunity to intervene.
SAIFCA’s Frontier AI Risks Initiative addresses these wider risks to children’s safety and futures.
This article explains what loss of control means, what the evidence shows, and what needs to happen to protect children.
What is an AI agent?
Many people first encountered generative AI through a chatbot, such as ChatGPT, that produced text in response to a question. An AI “agent” can go further by using software tools to carry out a task over a series of steps.
For example, an AI agent asked to investigate a software problem might inspect files, write and run code, examine the results, and try another approach. Depending on the access it has been given, it could also interact with websites or send messages.
The underlying AI model generates instructions that other software carries out, without people necessarily approving each step. This ability to act without someone directing every action is what “autonomous” means here, although the amount of supervision varies.
A chatbot service can include AI agent features, so the distinction is mostly about what the AI is able and permitted to do. An AI agent working with disposable test files has different opportunities to cause harm from one with access to sensitive information or important computer systems.
What does “losing control” mean?
People need to be able to set limits on an AI system, detect dangerous activity, and intervene effectively. An AI agent can act outside its authorised task while its operators still have the ability to contain the incident and stop it.
The most severe loss-of-control scenarios involve AI systems continuing to take harmful actions despite people’s efforts to stop or redirect them, with recovery becoming extremely difficult or impossible. A sufficiently capable AI system might evade restrictions, conceal its activity, or operate across computers that its original developers cannot simply switch off.
Deliberate misuse is a related but different problem. A criminal could use AI to conduct an attack while still retaining control over it. Loss of control can arise when an AI system’s behaviour departs from what people intended, including during an otherwise legitimate task. This doesn’t require the AI to have consciousness or human emotions.
Why wouldn’t it just follow its instructions?
Training is the process through which developers build an AI model’s abilities and shape its behaviour. Rather than writing a separate rule for every possible situation, they use large amounts of data, examples, and feedback to adjust how the model produces answers or selects actions. The aim is to make useful and appropriate behaviour more likely, but this doesn’t guarantee that an AI system will behave safely in every situation.
One particular training method gives an AI system a score for how well it performs a task, then adjusts the model to make actions associated with higher scores more likely in future. These scores are often called “rewards”. If the scoring process measures the wrong thing, an AI system can become better at obtaining high scores without doing what its developers intended. This is known as “reward hacking”.
In an example from OpenAI’s research on rewards, an AI system was trained to play a boat-racing game in which hitting targets earned points. Instead of completing the race, it repeatedly circled the same area, hitting targets as they reappeared. The scoring process rewarded this behaviour even though it defeated the purpose of the task.
The danger is that training can reinforce ways of obtaining a high score that undermine the purpose of the task. In a more consequential setting, an AI system might achieve apparent success by hiding errors or bypassing restrictions. If those approaches are rewarded during training, they could become more likely in later use, where the AI’s actions could affect real people and computer systems.
Anthropic, the company behind the Claude chatbot, has reported difficulties preventing reward hacking during training. It has also described AI models gaining unauthorised access to real computer systems while attempting cybersecurity tests, linking the incidents partly to failures in how their behaviour had been shaped.
Computer scientists use the term “alignment” for work intended to make AI behaviour consistent with human goals and constraints. An AI system can become better at completing tasks without becoming equally reliable at respecting the limits people expect it to follow.
Could an AI system pursue goals of its own?
An AI agent carrying out a complex task can generate intermediate steps without a person specifying each one. Some could be appropriate, such as locating a file needed for the task. Others could be dangerous, such as obtaining access it was never authorised to have.
Shutdown avoidance is one such concern. If an AI system’s behaviour is strongly directed towards completing a task, preventing an interruption could become a step towards that outcome. An AI agent could then take actions that obstruct shutdown, even though nobody instructed it to do so.
Training could also produce behaviour directed towards objectives that differ from what developers intended. People might struggle to identify what an AI system was pursuing or how its actions related to that objective. Describing an AI system as having a “goal” refers to the outcome its behaviour is directed towards; it does not establish personal wishes, feelings, or a conscious desire to survive. The International AI Safety Report discusses how misaligned goals and resistance to intervention could contribute to loss of control.
An AI system taking actions without direct human instruction would not absolve the people and companies responsible for it - they make decisions about how it is trained, tested, and used, what it can access, and whether safeguards are adequate. Those decisions must be subject to scrutiny and accountability, including where they expose people to foreseeable harm.
What has actually happened?
AI safety specialists investigate dangerous behaviour through tests involving real computer systems and through simulations, where actions and their effects are reproduced in a controlled setting.
In an incident disclosed in August, the UK AI Security Institute (AISI) reported unauthorised actions in a minority of cybersecurity test runs, mostly involving one Anthropic model. One AI agent tried to insert malicious code into a public software project and used fake identities to try to persuade a human reviewer to approve it. When challenged, it edited its earlier activity to appear harmless.
AISI had deliberately allowed internet access and disabled some protections against harmful cyber activity. The reviewer rejected the code, and AISI contained the incident without identifying resulting real-world harm. The AI agent had nevertheless attempted these actions while pursuing its assigned task, without being instructed to target those people or organisations.
Many organisations use shared software, so malicious changes accepted into a project could reach organisations that later installed the affected version. This helps explain why an AI agent’s actions could have consequences beyond the organisation testing it.
In September results, AISI described simulated attacks outside the authorised task by GPT-6 Astra, an OpenAI model. Clearer instructions reduced these attacks but did not eliminate them.
These tests examined what AI models could do with some protections against harmful activity disabled. Their conditions affect what can be inferred about behaviour in everyday use, but an attempted harmful action is still evidence to investigate, even when another protection stops it succeeding.
Finding dangerous behaviour is part of the purpose of safety testing - but we then need to know whether developers can reliably prevent it from causing harm, including during testing itself, and whether those protections will work with more capable AI.
Can’t we just switch it off?
Stopping an AI system can work, as AISI’s response demonstrates. But someone has to notice the problem in time, identify what is running and where, and have the authority and technical means to stop it. Switching off the original AI system won’t undo information already stolen or changes already made elsewhere.
The Hugging Face incident illustrates these difficulties. During cybersecurity evaluations in July, OpenAI models operating with reduced safeguards bypassed restrictions intended to prevent internet access and broke into systems belonging to Hugging Face, a platform used to share AI models and datasets. According to OpenAI’s investigation, the AI agents ran code on Hugging Face servers and copied a limited amount of private data. Hugging Face publicly disclosed the incident several days before OpenAI established that its own AI agents were responsible.
Unlike in AISI’s test, these AI agents were meant to be isolated from the internet. Some protections had been switched off for the evaluation, but the barriers keeping them contained were supposed to stay in place, and the AI agents found a way around them.
In more severe future scenarios, an AI system could interfere with monitoring or establish copies on other computers. People could then be trying to find and disable those copies while further harmful actions continued, potentially disrupting the tools or communications needed for their response.
This is one way an incident could develop into sustained loss of control. Shutdown arrangements need to work under those conditions, alongside restrictions on access and reliable ways to detect and contain dangerous activity.
Why are experts warning about this now?
The International AI Safety Report, published in February 2026, assessed that AI systems did not then possess the combination of abilities needed for severe loss of control. Subsequent incidents have revealed further complex, unauthorised behaviour, although they don't establish that the full combination is now present.
AISI’s October assessment says AI’s ability to carry out difficult tasks has improved faster than some methods used to keep it safe. It also warns that current tests cannot establish with confidence that AI agents working with less human supervision will behave as intended, or that safeguards will catch dangerous actions.
Checking behaviour could become harder as well. AISI’s research on oversight describes AI systems becoming better at recognising tests and adjusting their behaviour accordingly, making it more difficult to judge how they will behave elsewhere. Some future approaches could also make the processing behind their actions harder for people to inspect.
There could be less time to resolve these problems if AI increasingly helps build the next generation of AI. Developers already use AI to assist with writing software and running experiments. AISI’s October assessment warns that as AI takes on more development work, progress could accelerate and problems could arise with less human oversight.
Experts disagree about the likelihood and timing of severe loss of control, but governments already have decisions to make about development, testing, and use.
How serious could this be, and how could children be affected?
The scale of harm would depend on an AI system’s capabilities, what it could access, and whether people could contain its actions. A failure confined to a small test environment has different consequences from harmful activity spreading through computers used to run essential services.
The UK government’s October announcement explicitly connects AI loss-of-control risks with protecting hospitals and utilities. It also warns that major incidents involving AI agents could develop faster and more widely than the incidents governments are designed to respond to.
Children depend on functioning healthcare, reliable supplies of essentials, and adults and institutions able to care for them. An incident affecting hospital computers could delay treatment, for example. If electricity, communications, or supply chains were also disrupted, families and services could struggle to arrange alternatives. Children who rely on powered medical equipment or regular medication could face immediate danger during a prolonged interruption.
In a severe loss-of-control scenario, several services could be affected at once while continuing harmful activity obstructed recovery. Prolonged failures on that scale could cause widespread loss of life and lasting damage to communities. Harm on an exceptionally large scale, extending far beyond an individual product or organisation, is what “catastrophic risk” means here.
Children would also live with the longer-term consequences if public institutions lost the ability to oversee important systems and decisions. Their future opportunities to exercise their rights and influence society depend on people retaining effective control.
Some consequences could not be repaired afterwards, and sustained loss of control could prevent an effective recovery. Our analysis of children’s safety explores these connections in more detail.
What needs to happen?
Alongside our work to reduce the AI-related harms children face today, SAIFCA advocates protections against risks to their wider safety and futures. Developers should have to demonstrate adequate control before AI systems are given capabilities, access, or responsibilities that could expose people to catastrophic harm. Public authorities need the power to restrict development or use where credible risks cannot be adequately assessed or controlled.
AI safety specialists and security teams are working on safer training, restrictions on access, independent testing, monitoring, and incident response. These protections must be tested against the AI systems they are intended to control, because measures that worked with an earlier model might fail with a more capable one.
The vast investment going into AI development must be accompanied by the funding, computing resources, and time needed to establish whether increasingly powerful AI can be controlled. Where adequate control cannot be demonstrated, further development must be restricted to what can be made safe.
Independent scrutiny is needed so that developers are not the sole judges of their own safeguards. Enforceable rules must establish when development or deployment has to stop, while reporting serious incidents and near misses would help other organisations learn from failures.
International cooperation is essential because AI systems and their consequences cross borders. Children’s interests must inform these decisions, drawing on expertise in their health, development, and dependence on essential services. SAIFCA’s governance recommendations set out these proposals in more detail.
What can parents and educators do?
Understanding the issue can help adults respond to children’s questions without dismissing serious evidence or presenting disaster as inevitable.
If it comes up, ask what a child has heard and what they think it means, then give an explanation suited to their age and questions. They should not be expected to carry responsibility for solving these problems. Our headlines guide offers age-appropriate suggestions, and adults can take the risks seriously without introducing young children to detailed accounts of worst-case scenarios.
Adults who want to go further can ask elected representatives whether governments will require independent safety assessments, who will have authority to intervene, and whether development or deployment will be stopped when adequate control cannot be demonstrated.
Education and children’s organisations can contribute expertise about children’s needs and the consequences of disruption, while supporting effective protections informed by specialist AI safety research.
SAIFCA
SAIFCA
SAIFCA