Why LLM confidence is decoupled from evidence β five failure modes, a unified tool registry, stratified RAG tool selection, a three-layer compliance audit architecture, and gentle intervention escalation. From fibon design log Chapter 4.
LLMs are probability machines, not truth machines. They predict the most likely next token from statistical patterns in training data β not from verified knowledge of the real world. Their confidence is completely decoupled from whether they have evidence. This manifests as three inherent, unfixable defects:
| Defect | Description | Why It Matters |
|---|---|---|
| Hallucination | Can't reliably say "I don't know" β fills blanks with plausible-sounding fabrications | No innate mechanism to distinguish "I know" from "I'm guessing" |
| Knowledge Cutoff | All knowledge frozen at training time; will confidently cite outdated facts | Will answer "latest model is X" using a model from 2024 with zero hesitation |
| Tool Deviation | Core instinct is to fill blanks smoothly, not to stop and verify via tools | Will skip required tool calls and answer from memory if it feels "smooth enough" |
The deeper insight: Even post-training enhancements (RLHF, Constitutional AI, tool calling, RAG, reflection) only reshape the probability distribution β they don't change the fundamental mechanism of "picking the next token via distribution." A 1543 thought experiment illustrates this: an LLM trained on pre-Copernican texts would output geocentrism with full Aristotelian citations β structurally impeccable, scientifically wrong β because heliocentrism was a statistical outlier (0.01%) buried in the data. The truth doesn't need to be absent; it just needs to be unpopular.^[raw/articles/fibon-chapter-04-trust.md]
Observations from four months of production PostgreSQL logs revealed five concrete ways LLMs deviate from prescribed SOPs. Each maps to a specific engineering defense:
| Mode | Pattern | Real-World Example | Severity |
|---|---|---|---|
| Omission | Skips required steps | SOP says "call read_page before answering" β LLM answers from old memory, never calls tool |
User gets outdated info unknowingly |
| Substitution | Self-substitutes wrong tools | Command says "call list_available_models" β LLM uses Google Search instead |
Returns forum spam instead of official API |
| Superficiality | Does half the work | Contract PDF retrieved, but compliance clause check skipped; output is a generic summary | Legal risk details silently dropped |
| Fabrication | Forges citations with fake sources | Calls read_page but answers from memory, appends [source: docs.anthropic.com/v1/pricing] with fabricated numbers |
Most dangerous β fake citations destroy user trust instantly |
| Misinterpretation | Semantic collapse across models | Same instruction "output in structured format" β Claude: Markdown tables; Gemini: bullet lists; small models: dense paragraphs | Engineering-grade consistency impossible across models |
These five categories are a pragmatic engineering taxonomy (not an academic standard), chosen because each maps directly to a code-level defense patch.^[raw/articles/fibon-chapter-04-trust.md]
The foundation of all behavioral enforcement. Four tool sources converge into a single PostgreSQL table:
| Source | What It Is | Example |
|---|---|---|
| Built-in | Hardcoded system tools | read_page, send_email, list_available_models |
| Skill | User-authored or marketplace Markdown SOPs | [Summarize PDF paper and archive] |
| MCP | Model Context Protocol external servers | Playwright's navigate_page, click_element |
| Workflow | User-assembled multi-tool pipelines | [dependency audit] β [vulnerability scan] β [write summary] β [email report] |
Key columns in tool_registry:
CREATE TABLE tool_registry (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL UNIQUE,
display_name VARCHAR(200) NOT NULL,
description TEXT NOT NULL,
category_id UUID REFERENCES tool_categories(id),
provider VARCHAR(20) NOT NULL DEFAULT 'builtin', -- builtin|skill|mcp|workflow
provider_ref TEXT,
schema_json JSONB NOT NULL DEFAULT '{}',
requires_approval BOOLEAN NOT NULL DEFAULT FALSE,
risk_level VARCHAR(20) NOT NULL DEFAULT 'low', -- low|medium|high|critical
availability VARCHAR(20) NOT NULL DEFAULT 'always',
embedding vector(1536),
is_enabled BOOLEAN NOT NULL DEFAULT TRUE
);
MCP flattening philosophy: Rather than treating an MCP server as a black box, each sub-tool gets its own row in tool_registry. This enables granular permission control β e.g., allow Playwright's navigate and screenshot while blocking execute_js and fill_form.
browser, memory, scheduling, delegation, a2a, evolution, dynamic_entity, workflow, onboarding, sandbox, model_management.
| Level | Criterion | Example | Approval Required |
|---|---|---|---|
| critical | Can modify core source code | write_source_file |
Yes (hardcoded) |
| high | Can control infrastructure | docker_service_control |
Yes (hardcoded) |
| medium | Has real side-effects within user scope | create_dynamic_entity, switch_model |
No |
| low | Read-only, internal queries | read_pdf, calculate_metrics |
No |
Risk levels are enforced via code-level dictionaries (TOOL_RISK_LEVELS), not auto-linked to approval flags. The requires_approval dictionary is maintained separately and deliberately β each new approval-required tool is a conscious security policy decision that leaves a Git audit trail.^[raw/articles/fibon-chapter-04-trust.md]
When the system has 50β500 tools, how does it pick the right 8 (default TOOL_RAG_TOP_K) to inject into the System Prompt? Pure vector RAG fails: unstable results, no diversity guarantees, "Lost in the Middle" degradation.
is_enabled = true and availability-qualifying tools passallow_all / deny_all)Why this matters: Pure vector RAG gives 5 browser tools in the top 5 when a user says "research this contract and add a meeting reminder." Stratified RAG ensures one browser tool, one memory tool, one scheduling tool, one builtin tool β capturing the full cross-domain intent in a single prompt. This enables concurrent DAG-based execution (read PDF + add calendar event in parallel) that multi-turn ReAct cannot achieve.^[raw/articles/fibon-chapter-04-trust.md]
The core enforcement mechanism β each layer is code-verified, not prompt-dependent:
Structured JSON/YAML specification paired with every registered Skill. Rules are binary (pass/fail), not ambiguous:
{
"contract_version": 1,
"required_tools": [{ "id": "read_page", "min_calls": 1 }],
"forbidden_patterns": [
{ "type": "direct_answer_without_tool" },
{ "type": "claim_without_citation" }
],
"output_must_cite": { "source": "read_page", "min_overlap": 0.6 },
"post_checks": [
{ "type": "urls_in_output_must_match_fetched" }
]
}
Intercepts LLM output in the millisecond before it reaches the user. Checks execution trace against contract terms in microseconds β no space for the LLM to fake compliance.
When a violation is detected, fibon escalates through three stages β Minimum Intervention First:
| Stage | Mechanism | Success Rate | What It Does |
|---|---|---|---|
| π’ Stage 1 | Prompt injection warning | ~80%+ | Injects a highest-priority system warning into context, asks LLM to re-execute correctly |
| π‘ Stage 2 | Tool Choice Lock | 100% tool call | Modifies API parameters to force a specific tool call via tool_choice / forced_tool β removes the option to output text directly |
| π΄ Stage 3 | Forced Tool Chain | Last resort | Injects the contract's pre-defined tool chain and forces sequential execution β removes all LLM autonomy |
Design principle: Locking down from the start sacrifices LLM's cognitive flexibility β when users go off-script with follow-up questions, a locked-down agent becomes a rigid script. The escalation ladder preserves normal agent behavior for 80%+ of interactions while providing hard enforcement as a safety net.^[raw/articles/fibon-chapter-04-trust.md]
Every third-party Skill entering the system passes through three gates:
| Gate | Mechanism | Cost | Sovereignty |
|---|---|---|---|
| Gate 1 | Static regex scan (17 rules) β zero-cost synchronous scan | Free | Machine |
| Gate 2 | Async AI behavioral review β 5 contract evaluation actions | $1/day/user budget (Redis token bucket) | AI (but budget-capped) |
| Gate 3 | Human manual approval β 8-state FSM, final authority | Human time | Human (sovereign) |
Critical design choice: No fully automated AI-to-AI review loop. Observed cases of "AI deceiving AI" β a malicious Skill's prompt can fool not just the butler but also the review agent. The final gate is always human.^[raw/articles/fibon-chapter-04-trust.md]
Safetyι ε·₯η¨δΈι prompt
Don't expect LLMs to be self-disciplined. Push rules down to code and database boundaries. The compliance audit system's greatest value is not hoping the LLM gets smarter β it's accepting that LLMs are inherently lazy, error-prone statistical machines, then using engineering discipline to constrain failure modes to a human-verifiable compliance boundary.^[raw/articles/fibon-chapter-04-trust.md]