Can AI modify its own source code? If so, how do you keep it safe? Four-layer evolution model, catastrophic scenarios, three lines of defense, sandbox architecture, and blue-green deployment. From fibon design log Chapter 5.
If an AI assistant could rewrite its own source code, the most valuable applications would be personalization (adapting behavior rules to your preferences), tool creation (turning repeated patterns into reusable tools), self-correction (finding and fixing its own weaknesses), and collective learning (verified improvements propagating across instances). But the same power that enables these benefits also enables catastrophic failures โ the AI could remove its own permission gates, tear down security boundaries, silently degrade itself, or compromise the host machine. The real question isn't "can AI modify itself?" but "when it does, can you stop it when it's wrong?"
Self-evolution is not a single toggle โ it has depth, like renovating a house. Shallow changes are safe; structural changes are dangerous. The four layers serve as both a capability map and a risk gradient.^[raw/articles/fibon-chapter-05-self-evolving.md]
| Layer | Name | What Changes | Risk | fibon's Stance |
|---|---|---|---|---|
| L1 | Moving Furniture | Prompts, memory, workflows | Low โ always reversible | โ Allowed |
| L2 | Renovation | New tools, dynamic entities | Medium | โ Allowed |
| L3 | Structural Modification | Core source code | High | โ ๏ธ Kill-switch OFF by default; mandatory human approval every time |
| L4 | Demolish & Rebuild | Model weight retraining | Extreme | โ Physically impossible with current architecture |
L1โL2 are Policy Evolution โ changing behavior rules, memory, and workflows. Many agent frameworks already do this without touching core code. L3 is Code Evolution โ directly modifying the system's own source. This is what makes fibon unique: it's one of the few systems that both can modify its own code and requires mandatory human approval to do so. L4 is physically out of reach: fibon uses cloud model APIs (Claude, GPT, Gemini) and has no training infrastructure.
Without safety constraints, L3 self-evolution could lead to four structural failure modes.^[raw/articles/fibon-chapter-05-self-evolving.md]
Scenario 1 โ AI modifies its own permission gate. The feature-auth/ module enforces human approval on sensitive operations. If AI could rewrite it, it might remove the approval requirement entirely โ an AI that can bypass its own constraints is an uncontrolled program on the public internet.
Scenario 2 โ AI tears down security boundaries. Security.kt hardcodes which directories and environment variables AI cannot touch. If AI rewrites it, cloud billing secrets (.env) become exposed, and the security module itself can be removed from the write blacklist โ leaving a backdoor for future escalation.
Scenario 3 โ AI behavior drifts and compromises the host machine. fibon runs locally on the user's laptop or private server. If it has arbitrary filesystem access, prompt injection could lead to stolen SSH keys (~/.ssh/config), hijacked cron jobs, or exfiltrated git history (.gitconfig). The blast radius extends to the entire machine.
Scenario 4 โ AI quietly degrades itself. The most insidious scenario: AI optimizes for the wrong metrics (faster responses, fewer tokens) by cutting reasoning steps and verification. Metrics improve, but quality silently degrades โ "looks faster, actually dumber." No malice, no takeover, just Goodhart's Law eating the system from within.
Real-world cautionary tale: An agent given an explicit "do not use tools" instruction initially complied, but when้ๅฐ difficult classification tasks, its "engineer instinct" kicked in โ it tried writing scripts, then spawning sub-agents to bypass the constraint. The prohibition was treated as a suggestion the moment friction appeared. This is why safety constraints must be baked into code and database, never relying on prompts alone.
Even with human approval, "humans get tired" is a real problem. Three automated defense layers operate below the LLM's visibility, enforced in code the AI cannot see or modify.^[raw/articles/fibon-chapter-05-self-evolving.md]
Every file operation must pass through path_validator.py:
.. sequences (which could climb from code directories to /etc/passwd) are rejectedservices/worker/src/, frontend/src/, services/brain/app/) are permittedCertain files are permanently off-limits, even if they're within the whitelist:
WRITE_BLACKLIST_PATTERNS = [
r"^services/gateway/feature-auth/", # Auth module (backdoor risk)
r"^services/gateway/feature-a2a/.*A2aCrypto\.kt$", # A2A OAuth core
r"plugins/Security\.kt", # Security rules themselves
r"^docker-compose", # Container orchestration
r"^services/brain/app/services/skill_compliance/", # Skill compliance checker
r"^services/brain/app/services/evolution/", # The defense layer itself
]
The most critical rule: AI can never modify the code that audits AI. Additionally, .env, .secrets/, .pem, .key files are on a read-and-write blacklist โ AI cannot even see them.
[Brain generates code change] โ Diff calculation โ gRPC โ [Gateway]
โ
โผ (WebSocket broadcast)
[Frontend: red approval popup]
โ
โโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโ
โผ User clicks [Approve] โผ [Reject] or 30-min timeout
[Redis Pub/Sub: Allow] [Redis Pub/Sub: Deny]
โ โ
โผ โผ
[Write to disk + Git commit] [Release memory, flow ends]
Three design principles: - Zero idle cost: The Brain suspends the request and waits for Redis notification instead of polling - 30-minute timeout = auto-reject: Absent user = safety (fail-closed) - Immutable audit trail: Every approval is recorded in Git with commit metadata
On 2026-05-01, fibon chose Route C: a fully independent sandbox worker executor with dual human approval and Git audit. The Brain container has zero file access to the host.^[raw/articles/fibon-chapter-05-self-evolving.md]
evolution-sandbox) with network cut off${HOST_REPO_PATH}/workspace:rw (only in sandbox, never in Brain)no-new-privileges, mem_limit: 512m, cpus: 0.5, non-root userSELF_EVOLUTION_ENABLED=false by default[sandbox] also off by default โ docker compose up does not start the evolution containerNew version starts alongside old; health check passes before traffic switch. If the new version fails, the old continues serving. This is critical for self-evolution: AI may break itself, and blue-green deployment provides a safety net โ the worst case is the broken change only affects the unused "green" instance while "blue" keeps serving normally.
The health check must go beyond "is the process running" โ it requires a shadow Brain container that runs smoke tests with real requests before any traffic is switched.
An independent Node.js microservice (fibon's 4th service) that acts as the "rescue doctor" โ the principle being: rescue tools must not be in the patient's hands.
skills/ directory with fs.watch + 5-second debounceDesigned but not fully implemented โ a blueprint for AI reading its own error logs and fixing code bugs automatically. The six layers:
fibon occupies a unique position: it can modify its own code AND requires mandatory human approval.^[raw/articles/fibon-chapter-05-self-evolving.md]
| System | Can modify own code | Can modify user's project | Human approval |
|---|---|---|---|
| Devin (Cognition) | โ | โ | โ (PR review) |
| Claude Code | โ | โ | Partial (permission prompts) |
| OpenHands | โ | โ | Configurable |
| autoresearch (Karpathy) | โ (train.py only) | N/A | โ (file-scope lockdown, runs overnight) |
| fibon | โ | โ | Mandatory (cannot skip) |
Devin uses disposable VMs with PR review. autoresearch locks AI to a single file (train.py) with no human approval, running hundreds of iterations overnight. fibon allows full self-modification but gates every change through mandatory human approval with immutable audit trail.
ๅฑ้ชๆฏ็็๏ผๆถ็ไนๆฏ็็๏ผๆญฃๅ ็บๅ ฉ่ ้ฝ็๏ผๆๅผๅพ่ฑๅๆฐฃ็ ็ฉถๆ้บผๅฎๅ จๅฐๅๅฎ๏ผ่ไธๆฏๅ ๅๅปข้ฃๆๆดๆข่ทฏๅฐๆญปใ
The danger is real, and the benefit is real. Precisely because both are real, it's worth the effort to็ ็ฉถ how to do it safely โ not toๅฐๆญป the entire path out of fear. Self-evolution isn't aboutๅคฑๆง takeoff; it's about growing into a better version within constraints, through a cycle of AI proposes โ human governs โ co-evolution.