RAG — Vector Knowledge Base

Yukkai's RAG (Retrieval-Augmented Generation) engine is a local vector search system that indexes your project documentation, code files, and reference materials so the AI can find relevant information by meaning, not just by keyword match.

What RAG Does

Traditional search matches keywords. If you search for "authentication" and your docs say "login flow," a keyword search won't find it. RAG solves this by using embeddings — mathematical representations of meaning — to find semantically related content.

The Pipeline

Text → Chunking → Embedding → SQLite Storage → Cosine Similarity Search

1. Chunking — Documents are split into meaningful chunks (paragraphs, sections, or code blocks)
2. Embedding — Each chunk is converted into a high-dimensional vector that captures its semantic meaning
3. Storage — Vectors are stored in a local SQLite database alongside the original text
4. Search — When you ask a question, your query is embedded and compared against all stored vectors using cosine similarity. The closest matches are returned.

All of this happens locally on your Mac — no data is sent to external servers.

Two Knowledge Pools

Yukkai's RAG engine maintains two distinct pools, indexed separately:

Agent Pool (yukkai_knowledge_agent)

Technical documentation about how Yukkai itself works — tool specifications, internal knowledge documents, architecture references. This is the pool Yukkai's own reasoning engine searches when it needs to understand how to use a feature or tool.

User Pool (yukkai_knowledge_user)

Product-level knowledge — simple, user-facing explanations of what Yukkai can do. This is the pool searched when answering end-user questions about capabilities, without leaking implementation details.

The RAG engine automatically scopes searches to the appropriate pool. When Yukkai needs technical info to execute a task, it searches the agent pool. When answering your questions about Yukkai's features, it searches the user pool.

Indexing Your Projects

Indexing a File or Folder

rag(action: "index", path: "/Users/me/Documents/GIT/MyProject", source_type: "project")

This indexes all text files in the specified path. Supported source_type values:

Source Type	Description
project	Your project's source code and documentation
document	Standalone documents (PDFs, markdown, text files)
skill	Content indexed as part of a skill's knowledge base

Indexing Raw Text

If you have content you want to index without a file (e.g., pasted text), use:

rag(action: "index_text", content: "Your text here...", title: "Optional Title")

Checking Index Status

rag(action: "status")

Returns database statistics — number of chunks, sources, storage size, etc.

Listing Indexed Sources

rag(action: "sources")

Lists all indexed sources with their IDs. You'll need these IDs to delete specific sources.

Searching

rag(action: "search", query: "How does the authentication middleware work?", top_k: 5)

Parameters:

• query — Your search question (natural language)
• top_k — Maximum number of results to return (default: 5)

The search returns the most semantically relevant chunks from the indexed content, along with relevance scores.

Scoping Searches

You can optionally scope a search to a specific pool:

rag(action: "search", query: "How to use bifrost", source_type: "yukkai_knowledge_agent")

Omitting source_type searches across all indexed content.

Managing Indexed Content

Deleting a Source

rag(action: "delete", source_id: "source_id_from_list")

Use source_id: "all" to purge everything (use with caution).

Reindexing

rag(action: "reindex")

Force-rebuilds Yukkai's internal knowledge base. This purges old yukkai_knowledge_* entries and recreates them from the compiled knowledge store. Useful after an app update that ships new internal documentation.

Workflow: Clean Stale Data

1. sources → List all, identify stale or duplicate IDs
2. delete(source_id: "stale_id") → Remove each one
3. reindex → Rebuild Yukkai internal knowledge
4. status → Confirm clean state

Security

• Local only — All embeddings and vectors are stored in a local SQLite database on your Mac
• Encrypted at rest — The knowledge base is encrypted with AES-256-GCM
• No external API calls — Embedding generation happens on-device
• PathGuard protection — File access during indexing is subject to PathGuard's security rules

Use Cases

• Codebase understanding — Index your project so Yukkai can answer questions about your code's architecture, conventions, and patterns
• Documentation search — Index your team's docs so Yukkai can find relevant procedures and guidelines
• Reference materials — Index API documentation, tutorials, or specifications for quick semantic lookup
• Cross-project search — Index multiple projects and search across all of them simultaneously