Are LLMs Hitting Diminishing Returns for Software Engineering? What Still Works and What Doesn’t

Are language models hitting diminishing returns for software engineering?

A recent Reddit post from a self-described distinguished engineer at a hyperscaler argues that, for day-to-day software engineering, the newest large language models (LLMs) are no longer delivering big step-changes. The claim: beyond a certain point, iterative, test-driven workflows don’t benefit much from marginal gains in general reasoning, especially when you already work in small, verifiable chunks.

“If you did a blind test… I wouldn’t be able to tell which model I was using.”

It’s a view that will resonate with many engineers who’ve already integrated LLMs into their coding routines. Below, I unpack what the post says, where I agree (and don’t), and what it means for UK teams choosing between cloud and local models.

Why experienced engineers see a plateau

Chunked workflows limit the upside of “clever” models

Most professional developers don’t “one-shot” complex features. We decompose work into small abstractions, write tests, and iterate. In that mode, a model that’s fast, consistent and grounded in your codebase can be more valuable than one that occasionally dazzles on synthetic puzzles. Marginal improvements to chain-of-thought or long-context reasoning don’t always translate into shipping code faster.

Knowledge saturation: the internet already taught models the basics

For many common tasks, models have had access to public documentation and best practices for a while. That plateau in “what the model has seen” can make successive releases feel samey on routine engineering chores, even if benchmark scores nudge up.

Confident errors still happen in systems work

The Reddit author cites an example where the latest model confidently misdescribed AWS Application Load Balancer (ALB) and ECS draining behaviour. This is a classic “hallucination” – an LLM producing plausible but incorrect statements. It’s especially risky in systems architecture, where subtle behaviours matter. If you want the canonical source on this topic, see AWS’s documentation on target deregistration and connection draining.

“Fable got the AWS ALB/ECS draining behaviour completely wrong, and confidently so.”

What still works well for software engineering

Iterative prompting with tests and constraints

Short, scoped prompts anchored by tests, type hints and linters remain highly effective. Treat the model like a pair programmer: suggest, run, verify, refine. This plays to LLM strengths (pattern completion) while constraining risk.

RAG over your own code and docs

Retrieval-augmented generation (RAG) injects relevant, indexed snippets from your codebase or internal docs into the prompt. By grounding the model in the exact repository or service contract, you cut hallucinations and get answers tailored to your stack. RAG also helps smaller, faster models perform “above their weight”.

Tool use and orchestration beat raw model IQ

Function calling, structured outputs (JSON), and agent-like loops that run commands, read files, and re-plan are often bigger wins than switching frontier models. The right scaffolding – execution sandboxes, policy checks, and test harnesses – usually matters more than a new release’s extra IQ point.

Local models are approaching “good enough” for many tasks

The author expects that within a year, a local model on a 128 GB MacBook Pro will deliver ~90% of the value of a top-tier cloud model for their workflow. That’s consistent with what many teams report: strong open models (e.g., Google’s Gemma family) are now capable on refactors, docstrings, unit tests, small features and code reviews, especially when paired with RAG and tools. See Google’s Gemma model page for an overview of the ecosystem.

What doesn’t move the needle (much)

One-shot mega-prompts and gigantic context windows

Long context windows are helpful for reading large files or summarising PRs, but they’re not a silver bullet for building correct systems. Most engineering wins still come from decomposition and tests, not from dumping an entire repo into a single prompt.

Chasing small benchmark deltas

Improvements on reasoning benchmarks don’t always show up in real repos. If you can’t tell models apart in a blind test against your own tasks, the practical ROI of upgrading might be modest.

Implications for UK teams: cost, privacy, and sustainability

Cost control and procurement

If each new cloud model adds little incremental value for your workflow, you can pause the upgrade treadmill. Instead, invest in better scaffolding: evals on your codebases, caching, RAG, and developer enablement. For some workloads, local inference can reduce per-token costs and egress.

Data protection and residency

Local or private-hosted models can reduce exposure of source code and production data, helping with GDPR, client NDAs, and data residency policies. The UK’s ICO has practical guidance on AI and data protection – see the AI auditing framework.

Latency and developer experience

Fast, predictable responses matter more than absolute accuracy when you’re iterating in tight loops. Smaller local models with RAG can often feel snappier than a slower frontier model over the wire.

Environmental footprint

Inference costs aren’t just financial. Datacentre energy and water use are under growing scrutiny in the UK. If you’re weighing cloud versus local or edge inference, consider the sustainability angle – I’ve written about the realities of AI water use and cooling.

A practical roadmap for engineering leaders

Run blind tests on your actual tasks. Evaluate models with your repos, policies and latency budgets – not generic benchmarks.
Adopt RAG early. Index your codebase, APIs and internal runbooks. Ground answers; log citations; prefer structured outputs.
Build a safety harness. Enforce tests, type checks, linters and policy gates in the loop. Auto-retry with smaller diffs on failure.
Start a local-track pilot. Trial a well-supported open model with tool use. Measure cost, latency, and accuracy versus your current cloud model.
Instrument everything. Keep per-task success rates, iteration counts, and regression alerts. Promote models on evidence, not hype cycles.
Document the “unknowns”. For critical infra topics (e.g., load balancer draining), link to vendor docs and require human review.

Are we at an asymptote or just a pause?

The post makes a fair case: for hands-on software engineering, we may be in a phase of diminishing returns from pure model upgrades. The real gains are coming from grounding (RAG), tools, and robust workflows. That said, there’s room for genuine breakthroughs – especially in verified reasoning, better retrieval, long-horizon planning with tests, and tighter IDE/runtime integration.

If you’re in the UK balancing budgets, compliance and delivery timelines, the pragmatic path is clear: choose “good enough” models, double down on engineering discipline, and measure everything. When a model truly shifts the curve on your own evals, you’ll see it – and you won’t need marketing copy to tell you.

Read the original discussion: Models Are Hitting Diminishing Returns Within Software Engineering