The Risks of AI Facial Recognition: Misidentification, Bias, and What the UK Should Do Next

Police used AI facial recognition to arrest a Tennessee woman – why this matters for the UK

A Reddit post claims US police used AI facial recognition to arrest a Tennessee woman for crimes allegedly committed in a state she says she has never visited. Details beyond the headline are not disclosed. You can read the discussion here: the Reddit thread.

Police used AI facial recognition to arrest a Tennessee woman for crimes committed in a state she says she’s never visited.

Regardless of the case specifics, the scenario is plausible and highlights the real risks of misidentification and bias when facial recognition meets policing. The UK is already deploying live facial recognition (LFR) at scale in some forces, so it’s worth unpacking what can go wrong, what the law expects, and what good governance should look like.

What police facial recognition is and how it’s used

Two common modes: 1:1 matching and 1:N search

• 1:1 “verification” – checks if a face matches a claimed identity (e.g. passport e-gates).
• 1:N “identification” – searches a probe image against a large database or watchlist to find the most likely person.

Live facial recognition (LFR) streams faces from cameras and compares them to a predefined watchlist in near real time. Retrospective facial recognition runs a similar search on still images from CCTV after the fact. In both cases, systems produce similarity scores; thresholds determine when to flag a “match”.

Why misidentification and bias happen in AI facial recognition

Technical and operational pitfalls

• Image quality and context – poor lighting, odd angles, masks, or compression artefacts reduce accuracy.
• Thresholds and base rates – set a threshold too low and you get false positives; too high and you miss genuine matches. When the true target is rare, even small error rates can generate many false alerts.
• Watchlist composition – broad or outdated watchlists increase the chance of false hits on innocent people.
• Human factors – “automation bias” means operators can over-trust a system’s suggestion, especially under pressure.

Demographic differentials

Independent evaluations have shown that some face recognition algorithms perform differently across demographics. The US National Institute of Standards and Technology (NIST) found varying false match rates by age, sex and race across many algorithms in its FRVT demographic effects report. While top-tier models have improved, the risk remains system- and deployment-dependent and must be tested locally.

The UK picture: deployments, law, and oversight

Active use by UK police

Several UK forces, including the Metropolitan Police and South Wales Police, have trialled and deployed LFR in public spaces for targeted operations. See their public pages for policy summaries and deployment notices (Met Police LFR). Scope, frequency, and independent audits vary.

Legal framework at a glance

• Data Protection Act 2018 (Part 3) governs law enforcement processing, including biometrics. Biometric data can be highly sensitive and requires strict necessity, proportionality, and safeguards.
• UK GDPR (for non-law-enforcement contexts) also treats biometric data as special category, requiring a clear lawful basis and additional conditions.
• Protection of Freedoms Act 2012 and the Surveillance Camera Code of Practice set standards for surveillance use in public places.
• In 2020, the Court of Appeal held that South Wales Police’s LFR deployment was unlawful due to insufficient policies, impact assessment, and safeguards (Bridges v South Wales Police).
• The Information Commissioner’s Office (ICO) has said LFR in public places will rarely be justified without a high bar of necessity, proportionality, and accountability (ICO Opinion, 2022).

Risks to UK individuals and communities

• Misidentification and wrongful stops or arrests – uncommon but high impact. The harm concentrates on those already over-policed if demographic differentials aren’t addressed.
• Chilling effects – visible LFR can deter lawful protest or community life.
• Function creep – databases and watchlists expanding to lower-level offences without fresh justification.
• Opaque vendor ecosystems – procurement black boxes and NDAs can block independent scrutiny.

Potential benefits if tightly controlled

• Targeted searches for serious offenders where there is credible intelligence.
• Faster identification in time-critical incidents (e.g. missing persons), subject to strict necessity and safeguards.
• Retrospective analysis that reduces manual triage time when evidence quality is high.

These benefits only hold if the deployment meets the legal tests and earns public trust through transparency and measurable performance.

What the UK should do next to reduce AI facial recognition harms

Policy and governance recommendations

1. Independent accuracy audits – require forces to publish vendor- and deployment-specific accuracy, false match rates, and demographic performance from accredited tests.
2. Minimum performance thresholds – set statutory floors for accuracy and ceilings for false matches before any operational use.
3. Strict scope limitation – reserve LFR for serious crime and high-harm threats; prohibit use for low-level offences or generalised “fishing” expeditions.
4. Watchlist discipline – narrow, current, and justified watchlists with clear inclusion criteria and expiry; publish statistics after each deployment.
5. Human-in-the-loop with accountability – no automated decisions. Require written rationale for every intervention triggered by LFR, with supervisor review.
6. Pre-deployment approvals – mandate Data Protection Impact Assessments (DPIAs) and, for LFR in public spaces, external scrutiny or judicial/commissioner sign-off.
7. Clear redress routes – make it easy to contest a match, obtain disclosure, and seek remedies where harm occurs.
8. Procurement transparency – standard contract clauses for audit rights, data minimisation, security, and bias testing; publish vendor names and model versions.
9. Data retention limits – delete non-matches immediately; strictly time-bound retention for matches with documented necessity.
10. Public notice and engagement – visible signage, post-event reporting, and community consultation in affected areas.

Practical advice for developers and data teams working with vision AI

• Measure and report – track precision, recall, and false positives at operational thresholds; validate on local, demographically representative data.
• Stress-test context – evaluate performance across lighting, angles, occlusions, and device types to expose failure modes.
• Guard against automation bias – design operator UIs that show uncertainty and require explicit justification before action.
• Log for audit – keep immutable logs of inputs, model versions, thresholds, and human decisions. If you need a simple starting point for structured audit trails, here’s a guide to connecting ChatGPT to Google Sheets to capture metadata.
• Privacy by design – collect the minimum necessary data, encrypt at rest and in transit, and set clear deletion policies.

Bottom line

The Reddit headline is a cautionary tale, even without the full facts. Facial recognition can help in tightly defined scenarios, but without rigorous oversight it risks wrongful interventions and erodes trust. The UK has a legal framework that demands necessity, proportionality, and accountability; now it needs consistent, transparent practice to match.

Further reading and sources

• NIST FRVT demographic effects (NISTIR 8280): report
• Bridges v South Wales Police (2020, EWCA Civ 1058): judgment
• ICO Opinion: The use of live facial recognition technology in public places (2022): pdf
• Metropolitan Police – Live facial recognition: policy page
• Surveillance Camera Code of Practice (Home Office): guidance