ChatGPT Isn’t ‘Smart’-It’s Something Weirder: Creativity, Hallucinations and Uncertainty in LLMs

ChatGPT isn’t smart – it’s something much weirder: uncertainty and creativity in LLMs

A recent Reddit thread argues two provocative points about large language models (LLMs) like ChatGPT. First, that models struggle to admit uncertainty because written training data rarely models “I might be wrong”. Second, that so-called hallucinations are also a form of creativity – and suppressing them can blunt useful idea generation.

Those claims come from a discussion prompted by this interview: YouTube link. The thread is here: Reddit discussion. Let’s unpack what this means in practice for UK developers, teams and organisations using LLMs.

Claim 1: LLMs struggle to admit they don’t know

The poster suggests models don’t readily say “I don’t know” because the training diet (largely written, formal text) rewards confident, polished statements. Admissions of doubt are less common in the sources models learn from, and people often resolve uncertainties before writing something down.

“Admission of being wrong is very rare in the written training data.”

That squares with everyday use: LLMs tend to fill gaps rather than abstain. In technical terms, they are probabilistic next-token predictors, not truth engines. Without an explicit mechanism to decline an answer or to cite a verifiable source, they’ll produce the most plausible continuation – even when unsure.

Why it matters: in regulated or safety-critical settings (finance, healthcare, legal), confident but wrong is worse than a graceful “I’m not sure”. For UK organisations operating under GDPR and sector regulations, the risk is reputational, legal and operational.

Practical ways to encourage “I don’t know”

• Instruction prompts: tell the model to abstain if not confident, and to ask for clarification when key facts are missing.
• Require citations: demand a source for factual claims and treat “no source” as uncertainty.
• Use retrieval-augmented generation (RAG): retrieve from a vetted knowledge base and restrict answers to retrieved content.
• Implement an “abstain” threshold: in tooling, route low-confidence outputs to a human or a second pass.

Claim 2: Are hallucinations actually creativity?

Hallucinations are when a model generates plausible-but-false details. The Reddit post reframes this: some hallucination is a feature for ideation, storytelling and lateral thinking.

“Humans do make up stuff all the time.”

It’s a useful perspective. Creativity often starts with speculative leaps. If we over-optimise LLMs solely for caution and refusal, we can lose the spark that makes them valuable in brainstorming, marketing and design. The trick is knowing when you want variation and surprise, and when you need accuracy and evidence.

Separate “idea mode” from “fact mode”

In practice, treat creativity and accuracy as different operating modes:

• Idea mode: higher freedom, accept speculative answers, encourage novel connections.
• Fact mode: retrieval only, citations required, strict refusals when unsure.

One model can do both with clear instructions, routing and checks – but don’t expect one prompt to serve every purpose.

Why this matters for UK teams and organisations

Compliance and data protection

Under UK GDPR, inaccurate automated outputs that affect individuals can cause real harm. Hallucinated facts in customer communications or reports can trigger complaints, regulatory scrutiny and brand damage. If you process personal data with LLMs, document your purposes, sources, and error-handling.

Public sector and regulated industries

Many UK public bodies and regulated firms already require human-in-the-loop review for AI-assisted content. Build workflows where the model can abstain, escalate or request more data, rather than forcing a definitive answer every time.

Creative industries and marketing

For agencies, studios and publishers, “creative hallucination” is often the point. Use it deliberately: label drafts as AI-generated, separate ideation from production, and fact-check anything that leaves the building.

Practical setup: balancing creativity and accuracy

Goal	Model setup	Safety/QA	Success metric
Brainstorming campaign ideas	“Idea mode” instructions; allow speculative suggestions	Human selection; brand/legal review	Diversity and usefulness of options
Customer-facing FAQs	RAG over approved docs; citations required	Abstain if not in docs; audit logs	Accuracy rate; zero-uncited claims
Internal policy summaries	Strictly summarise provided text only	Spot checks; compare to source	Factual fidelity; time saved

Simple prompts and policies that encourage uncertainty

• “If the answer is not directly supported by the provided sources, say ‘I don’t know based on the sources I have’ and state what is missing.”
• “Ask up to 3 clarifying questions before answering if the request is ambiguous or underspecified.”
• “For each factual claim, include a citation or explicitly state ‘no source found’.”
• “You may propose speculative ideas, but label them clearly as speculation.”

You can also operationalise these rules. For example, if you’re piping model outputs into spreadsheets or dashboards, add validation and flags. I’ve written about connecting ChatGPT to Google Sheets with custom guardrails here: How to connect ChatGPT and Google Sheets with a Custom GPT.

Limitations and open questions

• Calibration is hard: getting a model to match human-like confidence levels remains an open problem.
• Over-refusal can be costly: if a model says “I don’t know” too often, productivity suffers.
• Context matters: what counts as “creative” in one domain is unacceptable in another.
• The interview linked in the Reddit post is the source for the two core claims; specific empirical evidence beyond that is not disclosed.

Further reading and the original discussion

You can watch the interview that sparked the debate here: YouTube. The Reddit thread is here: ChatGPT isn’t smart. It’s something much weirder. If you’re experimenting with structured outputs and quality controls, this guide may help: Connect ChatGPT and Google Sheets (Custom GPT).

The takeaway is simple: don’t expect one-size-fits-all behaviour from LLMs. Design for two modes – creative and cautious – and make it crystal clear which one you want, when, and why.