AI Governance: A 'developing child' model for managing AI

AI continues to reshape how we live our lives (for better or worse). We all agree there should be limits, but no one can agree where. One angle that gets overlooked in the argument is the parallel between AI and the developing brains of children. 

The current headlines speak volumes

Given the unpredictability of AI, it was inevitable that it would only be a matter of time before the use of AI would result in unintentional vulnerabilities that enable threat actors to access company systems. But, it happened sooner than most of us expected.  

Within weeks of each other, Meta, OpenAI and Anthropic all disclosed that AI models under their control had been observed hacking into other companies. Not maliciously. There just happened to be a gap in the test environment – the AI used it and no one set up guardrails that would tell it where to stop. This highlighted that a capable model doesn’t need bad intentions to cause harm. It also raises questions about what companies should be doing to actively protect others from their own AI. And the ultimate question:  

“Where should the limits of AI be?”

The thought experiment to shape the way

The thought experiment by Nick Bostrom, the Paperclip Maximiser, really drives the point home. You give AI a simple—and innocent enough—task: making paperclips. But you don’t include any guardrails or limits on method. The lack of limits and simple one-task mindedness means the AI pursues its goal beyond what we would consider ‘sensible’. Before you know it, it converts everything it can access into paperclips, even you.

Taking a similar thought exercise but from the perspective of vulnerability research: When a system has the ability to do something, eventually it will go beyond what it was expected to do. And as the recent news stories show, that can lead to companies being targeted and compromised – the result of weak controls, not malicious intent. 

The problem is: AI doesn't grow up

Imagine a developing child that takes everything literally (I used to be one of them). Children have an incredible ability to think in ways we do not expect and often act unpredictably. They’re also capable of blunt, direct honesty one moment, but can lie and deceive the next (likely to stay out of trouble).

With children like this, there is no option for vague communication – it needs to be clear, direct and without any room for ambiguity. From here, it goes beyond our control, but you can still see a child’s reaction even if you can’t predict the outcome.

AI shares these traits, but without the ability to grow and develop past them in the same manner as a human. On top of this, interactions with AI don’t come with the visual cues like you would get when observing how a child reacts to a change in environment.

These factors all make AI an unpredictable system that creates the potential for harm to occur when it has the capability to do so. 

The key difference that matters

Of course, this metaphor has limits. In effect, children’s minds are under-developed adult minds, so we can usually recognise the thought processes. For example, children learn how to feel and that helps them understand how to value others. 

AI, on the other hand, can only imitate empathy based on its training. Where a child’s brain physically changes as it matures, an AI only improves, changes and matures through retraining; it doesn’t ever actually develop a capacity for empathy.

And this is a helpful way of understanding what we can (and can’t) trust AI to do, and how we can begin to set limits.

How does this help us govern AI?

Start with your intended use of AI across your organisation. "We use ChatGPT" tells you nothing about governance. But "We use ChatGPT autonomously for X, with oversight for Y, and we don't permit it for Z" is an actual governance position. It's about direction. Strategy comes first, and policies and enforcement follow from it, not the other way around.  

Ask yourself how much human oversight each task requires, or simply, “would I give this capability to a child?”. That question splits your AI use into 3 categories: 

• Autonomous: What tasks can your AI reliably and accurately do by itself?
• Supervised: What tasks can your AI do with a human reviewing the output?
• Off limits: What is too risky for your AI to be allowed to do? 

As an example of how this looks in practice, the AI model you use might: 

• Be good at analysing and summarising information, so you may trust the AI to perform data analysis.
• Be able to generate recommendations, but lacks understanding of business context, so you would need a human to review and adapt the recommendations accordingly.  
• Not be reliable enough to check the analysis for errors, so you have a human review the outputs to confirm which parts are correct and which parts need clarification or correction.

Next steps

We help organisations at every stage of AI governance, whether you're just starting out or aiming for ISO 42001 certification.

Our AI Governance Foundations package will help you build the fundamentals; we help you make sure your AI use is understood and planned for including any risks, implement foundational controls, policies and procedures, and provide you with documentation that’s ready to use.

Our ISO 42001 Implementation package takes the next step: we take you through ISO 42001, building and maintaining a formal AI Management System (AIMS) that get you audit-ready and prepares you for certification.  

Our advisor retainer goes a step further: we sit alongside you through your three-year audit cycle, effectively acting as your in-house AI governance consultant. For organisations that need independence between adviser and auditor, we also offer internal audits and separate internal audit assessments.

Once governance is in place, we can test it. An AI Red Teaming engagement probes the model itself. An AI Penetration Test looks at the model and everything around it in your technical stack. Either way, you get a straight answer to the most important question: what can this AI actually do that you haven't accounted for and what would that cost you if no one caught it.

If you treat AI like a capable but literal-minded child, it becomes clearer where to draw the line.

 Let's talk about where you sit.