Karpathy's Obsidian RAG

Why This Matters for AI Agents

Traditional RAG is amnesiac. Every query starts from zero — re-chunk, re-embed, re-retrieve, re-synthesize. The LLM Wiki compiles knowledge once, then keeps it current. When you ask a question, the answer is already half-synthesized in the wiki. The LLM just reads the relevant pages and connects the dots.

Layer 3: The Schema

The configuration file (like CLAUDE.md or AGENTS.md) that tells the LLM how to maintain the wiki. Defines conventions, page thresholds, tag taxonomy, update policies. This is what makes the LLM a disciplined wiki maintainer instead of a generic chatbot.

📥 Ingest — Knowledge Accumulation

You drop a source (URL, PDF, paste). The LLM reads it, extracts key info, creates/updates wiki pages, adds cross-references, flags contradictions, updates the index and log. A single source might touch 10-15 wiki pages. The human stays in the loop — reads summaries, guides emphasis.

❓ Query — Knowledge Retrieval + Filing

The LLM reads the index, finds relevant pages, synthesizes an answer with citations. Key insight: good answers get filed back as new wiki pages. A comparison you asked for, a connection you discovered — these compound into the knowledge base instead of disappearing into chat history.

🔍 Lint — Knowledge Maintenance

Periodic health checks: orphan pages, broken wikilinks, stale content, contradictions between pages, missing cross-references, tag audit. The wiki stays healthy because the LLM does the maintenance that no human wants to do.

Path A: Project Knowledge Wiki (Simplest)

Add a knowledge/ directory alongside your CLAUDE.md. Instruct Claude via CLAUDE.md to maintain it as a wiki. When you research something, ask Claude to file the findings. When Claude discovers something useful (a tricky API, a library quirk), it files that too.

my-project/
├── CLAUDE.md          # Instructions
├── knowledge/         # ← The wiki
│   ├── SCHEMA.md
│   ├── index.md
│   ├── api-quirks/
│   ├── architecture/
│   └── debugging/
├── src/
└── tests/

Path B: Team Wiki via Git (Medium)

A shared wiki/ repo that the whole team's Claude Code instances contribute to. Add it as a git submodule or symlink. The schema defines who can write what. Claude instances across the team ingest sources, update entity pages, and maintain cross-references. Git handles version history and conflict resolution.

team-wiki/
├── SCHEMA.md
├── index.md
├── raw/
├── entities/
├── concepts/
└── .git/

# In each dev's CLAUDE.md:
# "Also read and maintain the wiki at ../team-wiki/"

Path C: MCP-Powered Wiki (Full Power)

Build or use an MCP server that exposes wiki operations as native tools. Claude Code calls wiki_ingest, wiki_query, wiki_lint as tool calls — no shell commands, no file path gymnastics. The MCP server handles indexing, search, and consistency checks internally.

# Claude Code settings
{
  "mcpServers": {
    "wiki": {
      "command": "npx",
      "args": ["-y", "llm-wiki-mcp-server", "~/wiki"]
    }
  }
}

# Now Claude can natively:
# - wiki_ingest(url) -- process and integrate a source
# - wiki_query(question) -- search and synthesize
# - wiki_lint() -- health check the wiki

📝 llm-wiki-compiler (by atomicmemory)

A Node.js CLI that compiles raw sources into an interlinked markdown wiki. Supports Anthropic, OpenAI, and Ollama backends. Obsidian-compatible out of the box. Inspired by Karpathy's pattern.

npm install -g llm-wiki-compiler
export ANTHROPIC_API_KEY=***
llmwiki ingest https://some-article.com
llmwiki compile
llmwiki query "what is X?"

🔍 qmd (by tobi)

Local search engine for markdown files with hybrid BM25/vector search and LLM re-ranking. Has both a CLI (for agent shell commands) and an MCP server (for native tool integration). Perfect for scaling the wiki beyond the index file.

🧠 MCP Memory Server (official)

Anthropic's reference MCP server for knowledge-graph-based persistent memory. Stores entities, relations, and observations as a graph. Good for structured knowledge but lacks the wiki's narrative synthesis.

npx -y @modelcontextprotocol/server-memory

🔗 Obsidian Integration

The wiki directory works as an Obsidian vault out of the box. Graph View visualizes connections. Dataview plugin enables queries over frontmatter. Web Clipper captures articles to raw/. For headless servers, use obsidian-headless for Obsidian Sync without a GUI.

📚 Hermes llm-wiki Skill (Already Built)

Bob already has the llm-wiki skill — a full implementation of Karpathy's pattern with schema, index, log, three-layer architecture, ingest/query/lint operations, provenance markers, and frontmatter validation. It's ready to use right now.

Step 1: Start a Wiki (10 minutes)

Create a wiki/ directory. Add a SCHEMA.md with your domain and conventions. Add an index.md and log.md. Drop your first source into raw/ and ask your LLM agent to ingest it. That's it — you're running the pattern.

mkdir ~/wiki/{raw/{articles,papers},entities,concepts,comparisons,queries}
# Write SCHEMA.md with your domain
# Drop a source: cp article.md ~/wiki/raw/articles/
# Ask your agent: "Ingest ~/wiki/raw/articles/article.md into the wiki"

Step 2: Add Search (when you need it)

When the index file isn't enough (50+ pages), add qmd for hybrid search. Or just use grep — it works surprisingly well on structured markdown with consistent frontmatter.

# Simple search that works today
grep -r "transformer" ~/wiki/entities/ ~/wiki/concepts/

# Or install qmd for proper search
npm install -g @anthropic/qmd
qmd search "transformer architecture" ~/wiki/

Step 3: Connect to Your Agent

Point your agent's CLAUDE.md (or AGENTS.md) at the wiki. Define ingest, query, and lint workflows. The agent reads the schema at session start and follows the conventions. Knowledge compounds from day one.

# In CLAUDE.md or AGENTS.md:
## Knowledge Wiki
Maintain a knowledge wiki at ~/wiki/.
- Read SCHEMA.md and index.md at session start
- When researching, file findings as wiki pages
- When learning something useful, add to concepts/
- Run lint weekly to check for stale content

🏛️ Vannevar Bush's Memex (1945)

Karpathy's pattern is spiritually connected to Vannevar Bush's Memex — a 1945 vision of a personal, curated knowledge store with associative trails between documents. Bush imagined something private, actively curated, with connections between documents as valuable as the documents themselves.

The part Bush couldn't solve: who does the maintenance? Humans abandon wikis because maintenance burden grows faster than value. LLMs don't get bored, don't forget to update a cross-reference, and can touch 15 files in one pass. The wiki stays maintained because the cost of maintenance is near zero.