By Jim Germer, CPA.
Artificial intelligence is no longer experimental. It is embedded in how organizations search, advertise, operate, educate, and deliver professional services. Entire business models, control environments, and governance structures now assume AI systems will function continuously and at scale.
For years, incorrect output from these systems was described as “hallucination.” The term framed failure as anomalous and temporary, with the implicit assumption that scale and iteration would eventually eliminate the problem. That framing is now receding.
In formal disclosures and internal risk reviews, companies increasingly describe incorrect or unreliable output as the result of built-in limitations, structural constraints, or known system characteristics. This language does not promise resolution. It acknowledges persistence.
In risk and audit contexts, that distinction matters. Calling a limitation inherent places it within the category of risks that must be anticipated, governed, and priced—not merely corrected. For auditors, boards, and regulators, this marks a durable reclassification of exposure. Disclosure language changes before liability does.
I. Two Pathways, One Technology
AI can sound confident and smooth, but still give wrong answers. This creates two different types of risks.
The Hallucination Pathway
Under the hallucination framing:
- Incorrect output is treated as an exception.
- Errors are considered unpredictable edge cases.
- Improvement is expected to materially reduce frequency.
- Human review is assumed to remain effective.
This framing aligns with traditional software risk. Controls focus on detection and remediation. From an audit perspective, reliance may be justified if error rates are low and trending down.
The Inherent-Limitation Pathway
Under the inherent-limitation framing:
- Incorrect output occurs under normal operation.
- The behavior arises from probabilistic architecture, not defective code.
- Certain classes of error cannot be eliminated, only managed.
- Human reliance becomes a central risk variable.
In professional usage, “inherent” doesn’t mean frequent. It means structural. An inherent limitation is always present, even when everything is working as intended. For this reason, it must be disclosed, managed, and its costs addressed transparently. Once an organization adopts this framing, the risk profile changes permanently.
II. The Documentary Record (Hybrid, Forensic-Safe)
This shift is visible in public filings. Recent Form 10-K disclosures acknowledge that AI and automated systems may be subject to inherent flaws, biases, or incorrect output under ordinary conditions, including disclosures by Walmart Inc., Alphabet Inc., and Microsoft Corp.
In several cases, the language has evolved from describing AI failures as errors to describing them as operational risks that may persist even under normal conditions. The disclosures increasingly emphasize reliability limitations, bias, and incorrect output as characteristics requiring governance—not as issues expected to disappear through iteration.
These statements appear in Item 1A risk-factor sections. The language does not suggest elimination. It identifies persistence. For auditors and boards, this signals an internal shift away from optimism about correction and instead focusing on how to manage the system’s limits.
III. A Composite Case
Consider a major tech company whose AI systems are crucial for things like search, advertising, and running core business operations. At first, when AI made mistakes, people thought they were just isolated incidents. Errors were logged, fixes were applied, and communications emphasized improvement. But as these systems grew, internal evaluation revealed a pattern:
- Errors clustered in synthesis, summarization, and advisory contexts.
- Outputs were fluent and authoritative.
- As surface quality improved, independent verification declined.
Engineering teams recognized the behavior as architectural. Legal and risk teams reached a harder conclusion: the limitation could be mitigated, but not eliminated. At that point, the system was no longer treated as intermittently flawed. It was treated as structurally constrained. The technology did not change. The risk classification did. That shift is decisive for going-concern analysis.
IV. Metabolic Atrophy as Foreseeability Evidence
Metabolic atrophy enters this discussion not as a theory of cognition, but as foreseeability evidence. When AI systems reliably remove cognitive load, human review capacity degrades in predictable ways. Over time, people’s ability to double-check or think for themselves gets weaker. This occurs under ordinary use, not misuse. Controls that assume human vigilance fail when vigilance erodes.
Major enterprises now acknowledge in safety reviews and governance discussions that sustained reliance on AI can weaken human capacity for independent validation. Once that awareness exists, it becomes part of the documentary record. If incorrect output is known to occur and reduced human oversight is foreseeable, downstream harm is no longer accidental. Metabolic atrophy does not create liability by itself. It strengthens causation and predictability once harm occurs. Once a limitation is documented as inherent, reliance becomes a governance decision, not a technical one.
V. Converging Risk
The current instability does not arise from any single failure mode. It arises from convergence. AI-driven companies now face a full range of independently actionable risk categories on the same balance sheets—all at once: metabolic atrophy; youth suicide and mental-health injury; defamation; intellectual-property contamination; duty-of-care expansion; insurance withdrawal and exclusion risk; and governance failure.
None is novel alone. It’s the overlap of these risks that creates instability. A risk that cannot be priced cannot be ignored.
VI. Litigation and Discovery
In litigation, the distinction between hallucination and inherent limitation becomes dispositive. When failures are framed as hallucinations, defenses emphasize unpredictability. When framed as inherent, discovery focuses on:
- Foreknowledge of structural constraints.
- Consistency between investor disclosures and user-facing representations.
- Adequacy of controls in reliance-heavy deployments.
These are standard inquiries once foreseeability is established. At global scale, even low-frequency adverse outcomes can produce material exposure.
VII. Auditors and Audit Firms
Auditors are not responsible for validating AI output. They are responsible for evaluating risk disclosure, control design, and reliance assumptions.
When limitations are inherent, controls mitigate exposure but do not eliminate it. Professional judgment becomes harder to evidence. Reliance itself becomes audit-relevant. This applies not only to audit clients, but to audit firms as AI tools become embedded in audit workflows. The risk is structural, not directional.
VIII. Insurance Signals
Insurance markets respond early to persistent risk. When risk is episodic, it can be priced. When it is inherent and difficult to bound, coverage narrows. Broader AI exclusions, higher deductibles tied to automation, and increased self-insurance requirements are early indicators. Insurance markets respond to persistence, not promises.
When insurers hesitate to price risks that companies themselves describe as structural, going-concern considerations become increasingly salient.
IX. Going Concern and Structural Adaptation
Going concern does not imply collapse. Essential systems rarely disappear. Historically, when utilities generate systemic, unpriceable risk, the response has been structural adaptation: segmentation of high-liability functions, increased friction in sensitive use cases, or separation of business units.
A risk that cannot be priced cannot be ignored. When risk overwhelms existing governance structures, structure changes.
Conclusion
The retreat from “hallucination” language reflects recognition that certain AI behaviors are not anomalies, but conditions. For auditors, boards, and regulators, this places AI risk within the category of foreseeable, structural exposure—the type of risk that affects insurance availability, litigation posture, and long-term viability.
Metabolic atrophy, youth mental-health injury, defamation, IP contamination, duty-of-care expansion, insurance withdrawal, and governance failure are converging realities attaching to AI systems at global scale. Early recognition allows for deliberate governance. Delayed recognition invites compelled adjustment.
Auditors have seen this pattern before. The question is whether we recognize it early enough this time.
Sign in to get access to this free resource, and all of our whitepapers and reports.
Download this content today!
Register Now Already registered? Click here to Log In
