The Trust Ceiling: The invisible limit on what AI is allowed to do
Trillions of dollars in value - and any real prospect of AGI - sit behind a line the industry can't scale its way past. The first of two essays describing humanity's path to trustworthy AI.
Over the past decade, artificial intelligence has advanced into territory long assumed to be permanently human. Machines learned to speak, to hear, to see, to answer our questions; then, most strikingly, they mastered language itself. Systems like ChatGPT, Claude and Gemini now converse fluently, draw on more of humanity’s written knowledge than any person could absorb in a thousand lifetimes, and increasingly reason, using language as a substrate for thought.
For the first time we can plausibly imagine the arrival of what some call Artificial General Intelligence (AGI), defined in OpenAI’s charter as “highly autonomous systems that outperform humans at most economically valuable work”. In practice: systems that automate large parts of every business, and can act as ever-present assistants in our daily lives for almost everything we want to do.
But despite this extraordinary progress, something fundamental is broken.
The systems we are building today are powerful, but they are, this essay argues, intrinsically untrustworthy.
Hallucinations, lack of accuracy, inconsistency and the inability of such systems to reliably explain their results are holding back AI adoption behind a ‘trust ceiling’ where trillions of dollars of value are stuck. The AGI prize might be imaginable but who is going to allow AI to fully automate their business or to take control of their personal life when at every step the AI system might do huge damage to their brand or reputation, cost them money or tell them something that is outrageously untrue? What regulator is going to allow AI systems to become widespread in an industry when every decision they take is a black box and subject to unpredictable error? The “...most economically valuable work” from the AGI definition will remain in human hands until this issue is addressed.
If we can solve this trust problem while preserving and extending the enormous capabilities we have developed, humanity has a path to reaping these extraordinary benefits.
This essay explains the technology issues behind this challenge and its companion essay, published alongside it (and linked at the end), describes the path that will get us there. It is not the path the industry is on - and without change, the industry’s path will fail.
Two Paradigms of Software
To understand why, it helps to step back and look at computer software itself. Broadly speaking, there are two fundamental paradigms.
The first is what we might call symbolic software. This is software from the traditional world of programming: algorithms, data structures, and databases. This is the majority of software even in use today. In this world, fully correct is the standard. There is a definition of what the software should do, and the software either meets it or it doesn’t. If a spreadsheet adds two cells together we expect it to produce the right answer every single time. If it does not, we call it a ‘bug’: the publisher finds it, fixes it, and knows it will stay fixed, because the same inputs always produce the same behaviour. When a software system that helps with the company payroll retrieves data from a database we expect it to never get a payment wrong. These systems are typically deterministic, consistent, explainable, and fully controllable, and when they fail, the failure is an exception to be eliminated rather than a permanent property of the system.
The second paradigm is statistical machine learning, and in particular the one machine learning method that has come to dominate: large neural networks (often referred to as ‘deep learning’). These systems are not programmed in the traditional sense. Instead, they learn from data. They are large mathematical formulae and begin with randomly initialised parameters and the parameters are then improved systematically through a training process, exposing them to data, until the formulae stop improving and they generate useful results.
The results however are never guaranteed. These formulae are approximations with performance always below 100%. Best guesses. However, they do very important things that symbolic software cannot.
What makes modern AI so powerful is that these deep learning systems improve as we scale them - a trend that has survived every prediction of its end. More data for training, more compute, larger models - and performance continues to increase.
When neural networks first saw commercial use in the late 1980s, they typically had tens of thousands of parameters and trained on megabytes of data. Today’s frontier models have hundreds of billions to trillions of parameters and train on a meaningful fraction of everything humanity has ever written, on hardware performing computations at rates that were science fiction then.
These forces of astronomical increase in data and compute, plus innovation, have driven a similarly astronomical expansion of AI capability.
This is the engine behind the current wave of progress.
The Trust Problem in Neural Systems
But with this power comes a fundamental problem: neural systems are intrinsically untrustworthy.
Untrustworthy here means something specific. It is not the mere willingness of people to rely on a system: people can come to rely on systems they should not, out of convenience or lack of alternatives, a pattern the research literature on trust in AI warns about directly (Everett et al., Nature Reviews Psychology, 2026). Trustworthy in this essay means the properties that make reliance warranted, and for serious work those properties go beyond just being right: knowing where an answer came from, getting the same answer to the same question every time, being able to reconstruct and contest a decision, and knowing which rules it was made under. Measured against that standard, neural systems fail in four significant ways.
The first issue is accuracy. In AI research, progress is measured in percentage scores on established benchmarks. A better model improves a score, say, from 87.3% accuracy to 89.2%, and this is celebrated as a breakthrough. But (glass half empty), this also means the system is wrong more than one time in ten. The field’s own habits concede this point: benchmarks are declared ‘saturated’ when scores reach the mid-90s, and the community moves on to a harder one. Saturation never means 100%. A residual error rate is not a temporary embarrassment on the way to perfection; it is where every benchmark ends. In some applications of AI a meaningful error rate is acceptable; in many, it is not. When errors affect one’s brand or reputation, financial decisions, legal outcomes, or safety-critical systems, even very small error rates can prevent a system being used. However, all neural systems have an error rate and at scale even a very small error rate equates to a torrent of bad results. When neural systems are wrong, there is no ‘bug’ to fix: error is a natural consequence of what they are.
Secondly, neural systems can generate answers that are not just wrong, but the worst kind of wrong: convincing and wrong. Large language models (“LLMs”) - the form of neural networks that unlocked language and debuted with ChatGPT - frequently produce answers that are fluent, detailed, completely believable but outrageously false. This phenomenon, usually called hallucination, creates a uniquely dangerous failure mode. A system that sounds authoritative but fabricates facts is dramatically worse than one that simply fails. This failure mode cannot be engineered away because it follows from how the machine works. An LLM is built to produce the most plausible continuation of a text, and when it does not know the answer, it produces the most plausible continuation anyway: fluent, confident, and fabricated. Hallucination is not the system going wrong; it is the system doing exactly what it was built to do, at a moment when that is exactly the wrong thing to do.
Research continues to reduce how often this happens but the leading labs acknowledge the cause lies in how these systems fundamentally work. OpenAI’s own researchers have shown that some level of factual error is a statistical inevitability of next-word prediction, and that hallucination persists because training and evaluation reward a confident guess over an admission of uncertainty. Their proposed remedy is telling: train models to say “I don’t know” more often. Calibrated abstention is real progress, but notice what it is - a statistical guess about whether the system is guessing. It reduces the bluffing; it makes nothing checkable. The rate falls; the kind of machine it is does not change. This is borne out in practice: several years of recognition of hallucination as a major industry problem and billions spent on R&D have reduced it, not removed it. State-of-the-art models from every provider continue to hallucinate, and trust does not average out: so long as any answer might plausibly be fabricated, every answer must be checked. For high-stakes work, trust is governed by tail risk, not by average benchmark performance.
Third, neural systems are not explainable in any meaningful sense. Modern neural networks consist of billions or even trillions of parameters, initialised at random and then shaped by training on vast datasets with astronomical amounts of compute deriving their final values. We can observe their behaviour, but we cannot explain why they arrive at a particular answer. A formula with a hundred billion weights derived this way is arguably one of the least explainable human-produced things we can imagine: the truest ‘black box’ it is possible to be. Even the researchers who have dedicated their careers to these systems do not fully understand their internal workings. An entire scientific field, mechanistic interpretability, now works to open the box, and its findings are genuinely illuminating. But a science of the black box is not the same as an explanation you can rely on: partial accounts assembled from outside, after the fact, are a different thing from a full, faithful and human-understandable record of the reasoning behind every answer. The most this field can promise is a better understanding of the box. Trust requires that there be no box.
Additionally, the reasoning traces produced by the latest LLMs are fundamentally different from what a good explanation should be: an argument we made recently in a peer-reviewed paper presented at ICML’s 2026 Trust Workshop. A real explanation points to the facts that made the difference to the outcome, but neural networks do not represent data as anything like independent facts, so there is nothing to point to. The paper’s conclusion is one this essay reaches by another route: where complete explainability is essential, some part of the system must be something other than a neural network.
Finally, neural systems are not even reliably consistent. The same system can produce different, contradictory answers to the same question. This cannot be dismissed as a settings problem: even with randomness switched off (’temperature zero’), outputs shift with trivial rephrasings and reorderings of the input that change nothing about the meaning. In isolation, this may seem like a minor issue. In practice, it erodes trust rapidly. A system that cannot make the same decision twice under identical conditions cannot be relied upon.
Taken together, these issues amount to a single, central problem.
Modern AI systems cannot be trusted.
Aren’t humans untrustworthy too?
It may be tempting to reply that humans have an error rate too, and that we handle this with checks and balances: review, audit, accountability, escalation. Surely the same machinery can wrap an imperfect AI? But look at what that machinery runs on. When a trusted employee is wrong, we assume the error was honest: they were doing their best, but inventing an answer would not be a mistake but misconduct, with consequences to match. Their errors cluster where the work is hard, so we know where to check; humans can explain their reasoning and be questioned on it; they often know when they are out of their depth; and having erred once, they learn. Neural systems invert every one of these properties. Their most characteristic failure is fluent and confident invention, the very behaviour that in a human would end a career. Their errors follow a map no human can read, landing on tasks that look trivial while hard ones sail through, so oversight cannot be targeted where a person would think to look. Their explanations are stories told after the fact; and the same error will recur indefinitely, because there is no one inside the black box to learn from it, or to hold responsible. So the difficulty goes deeper than a non-zero error rate: the errors have the wrong character for any of our existing instruments of trust to grip.
The misfire is measurable, and researchers have watched trust fail in both directions for decades. Shown a single visible error, people abandon an algorithm (algorithm aversion); with familiarity they drift the opposite way, relying on output because they are used to it, not because it has become checkable (automation complacency). What never emerges is calibrated trust - trust that tracks actual reliability - because calibration needs error to be legible: predictable enough that we know where to look, and visible enough that it is caught rather than believed. Neural error is neither.
Why Neural Systems Dominate AI
Yet, despite these limitations, neural systems dominate the field - and for good reason.
Purely symbolic systems, hand-built by humans, have largely run their course in terms of what human programming alone can reach, and left behind extraordinarily valuable problems that are unsolved, problems that neural systems are solving.
Much of the world’s data is complex and unstructured: images, video, speech, and natural language. Such complex real-world data is largely immune to human programming and non-neural methods to process it have fallen short. However, deep learning, powered by the previously mentioned tailwinds of data and compute is mastering these tasks.
By being able to process natural language they are also unlocking access to the vast body of human knowledge stored in unstructured form. Books, articles, web pages, conversations - most of what we know is written in language, not stored in neatly structured databases. Neural systems can be trained on this and start to make use of this information.
Neural systems can also now reason in natural language: human languages such as English evolved to express all human thought and mastery of English by AI now means they can represent current thinking, intermediate thinking steps and even plans for solving a problem. So called ‘reasoning models’ use language as a substrate for thought and are now solving ever harder problems, though every step of that thought remains a statistical act, inheriting all the problems described above.
And finally, language allows for natural interaction with their primary users: people. It started more than a decade ago with voice assistants like Alexa and Siri but with the advent of the LLM, reality has now fully caught up with the world we have seen forever in science fiction: a world where we can simply speak to computers to get what we want.
AI’s central tension
We are left with a stark contrast.
Symbolic software is trustworthy but limited. Neural software has valuable capabilities but is untrustworthy.
This tension sits at the centre of modern AI.
If we continue down the current path, we will build systems that are increasingly capable, but never fully dependable. If we retreat to purely symbolic approaches, we regain trust but lose the ability to fully operate in the real world and benefit from the amazing new capabilities that machines can now do.
Neither path leads to robust AGI.
The Third Way
There is, however, a third path.
What if, through innovation, we could get the best of both worlds: technology that combines the capability of neural systems AND the trust and reliability of the symbolic world?
The goal is not compromise, but synthesis. Systems that understand the complexity of the real world and communicate fluently with humans, and whose reasoning and decisions are exact, consistent and explainable - by construction. None of this means abandoning neural models: it means no longer relying on them solely to do the parts of the job that require guarantees.
This is, I will argue, the only viable path to building AI systems that are both powerful enough to matter and reliable enough to trust.
This broad idea is not new: the term is sometimes called neurosymbolic, a blend of both neuro capability and symbolic trustworthiness, and researchers have advocated hybrid approaches for decades, often against the prevailing wind. But advocacy is not delivery, and delivery is hard. The industry itself is edging this way, bolting tools, code and databases onto its models wherever an answer must be exactly right. The second essay in this series examines why this falls short, what true synthesis requires, how it can be achieved - and why ever-better models, on their own, do not dissolve the problem. Advancing that synthesis is the central mission of my company, UnlikelyAI: a technology revolution that takes trust head on; a world where AI is fully trusted.
Continue to the second essay: Breaking Through AI’s Trust Ceiling: how trustworthy AI can actually be delivered.
Humma agrees, WTP!