Who Should Define AI Safety? Why Industry Self-Regulation Falls Short-and the Case for Independent Oversight in the UK

The companies building the most powerful AI are writing the safety rules. Should the UK accept that?

A popular Reddit post argues that we are repeating an old mistake: letting the makers of a risky technology also decide what “safe” means. In AI today, the same firms that push the frontier are often setting the safety standards, shaping government advice, and grading their own work.

“The people grading the exam are the people who wrote the answers.”

This isn’t about villainising researchers at OpenAI, Google DeepMind or Anthropic. It’s about the structure. In any sector with serious public risk, self-regulation alone tends to miss problems that independent oversight would catch. The question for the UK is not whether industry expertise matters (it does), but who has the mandate and means to verify what industry claims.

Why industry self-regulation raises red flags

The Reddit post’s core claim is a conflict-of-interest problem. When the same organisation designs, deploys, and declares its systems safe, three familiar issues show up:

• Information asymmetry – builders know far more about model behaviour and failure modes than anyone else, which lets them set the narrative.
• Incentive misalignment – timelines and commercial goals can clash with rigorous testing and delayed deployment.
• Systemic risk – coordination failures mean a single firm’s blind spot can propagate across products and markets.

We don’t let pharmaceutical companies approve their own drugs. Aviation doesn’t fly new aircraft on the basis of a manufacturer’s press release. AI systems won’t map perfectly to these regimes, but the principle travels: independent testing reduces foreseeable harm.

“Trust us, we’re the experts” was never the point. The point is who checks the experts.

What independent AI oversight could look like in the UK

Independence doesn’t mean antagonistic or anti-innovation. It means verifiable, arm’s-length evaluation with powers to investigate and, where necessary, to say “not yet”. In the UK, several bodies already occupy parts of this space:

AI Safety Institute and technical testing

The government has created an AI Safety Institute to research and test frontier models. Its role, if properly funded and empowered, could include pre-deployment evaluations, red-teaming, and post-release monitoring. Red-teaming means structured stress testing by independent experts to find failure modes before the public does.

Further reading: UK government announcement on the AI Safety Institute (gov.uk): press release.

ICO for data protection and model training

The Information Commissioner’s Office (ICO) enforces UK GDPR. That includes lawfulness of training data, data subject rights, and automated decision-making rules. If a model ingests personal data without a lawful basis, that’s not a “safety” quibble – it’s a compliance problem.

Guidance: ICO: AI and data protection.

CMA for competition and consumer protection

The Competition and Markets Authority (CMA) is already analysing foundation models and the markets around them. Concentration of compute, data, and distribution can shape who sets “safety” standards and how open evaluation actually is.

Overview: CMA initial review of foundation models.

Ofcom and NCSC for downstream harms

Regulators like Ofcom (online safety) and the National Cyber Security Centre (NCSC) have stakes in misuse risks – from automated disinformation to code generation that lowers the bar for cyberattacks. Their risk appetite should influence release norms for high-capability systems.

Concrete policy options for UK-independent oversight

• Mandatory independent model evaluations for high-risk systems – require pre-deployment testing by the AI Safety Institute or accredited third parties. Publish non-sensitive summary results.
• Incident and capability reporting – standardised disclosures when models exhibit hazardous behaviour (e.g., jailbreaks, bio or cyber capabilities), plus periodic capability benchmarks. Think aviation-style incident reporting, adapted to AI.
• Compute and training transparency – reporting thresholds for large-scale training runs, with basic facts on data sources, safety mitigations, and evaluation coverage. No trade secrets required, just the who/what/when at a high level.
• Assured access for auditors – secure interfaces that let approved evaluators probe models and logs without giving away weights or IP. Watermarked evaluation builds could help.
• Post-release monitoring and kill-switch plans – clear responsibility for shutting down or rolling back models if real-world harms exceed thresholds.
• Liability clarity – align product liability and professional standards so responsibility doesn’t evaporate in a vendor maze. Buyers should know who pays if systems go wrong.
• Public-interest research funding – stable funding for universities and civil society to develop tests, benchmarks, and open tools for evaluating safety claims.

These are compatible with a light-touch, agile approach – but they add teeth where it counts: independent verification and transparency.

Reference framework: NIST AI Risk Management Framework (risk identification, measurement, and governance). And for international context, see the Bletchley Declaration on frontier AI safety cooperation.

What this means for UK developers and businesses today

You don’t need to wait for new law to improve governance. If you’re integrating GPT-style models into products or workflows, consider:

• Documented evaluations – run task-specific tests for accuracy, bias, and misuse. Keep a model card: a short document describing intended use, limitations, and known failure modes.
• DPIAs before deployment – complete a Data Protection Impact Assessment when processing personal data. Capture prompts, outputs, retention, and vendor subprocessors.
• Human-in-the-loop for high-stakes tasks – require approval for anything that can change money, health, or legal status.
• Prompt and output logging – for auditability, incident investigation, and continuous improvement. Secure those logs.
• Vendor due diligence – ask for independent evaluation evidence, red-team reports, and security attestations. “Just trust us” isn’t due diligence.

If you’re experimenting with automation, here’s a practical starter: how to connect ChatGPT and Google Sheets. Treat even small automations as production systems: access controls, audit trails, and clear rollback plans.

A balanced take: keep industry expertise, add accountability

Frontier labs have valuable context and talent. They should help define tests, publish research, and open up safe evaluation pathways. But independence matters because incentives matter. Safety claims should be checkable by people who don’t ship the product or price the IPO.

Get this balance right and the UK can accelerate useful AI while reducing systemic risk. Get it wrong and we’ll repeat a pattern we already know from other industries: private assurance, public surprise.

Original Reddit discussion

Read and join the conversation: The companies building the most powerful AI in history are also the ones deciding what counts as ‘safe.’