The Model Needed a Map, Not a Search Engine

Every team I have worked with has the same quiet problem. The answer to most operational questions already exists, written carefully in a procedure document by someone who knew what they were doing. The trouble is that it lives in a folder somebody else has to remember. Finding it is the slow part, not knowing it.

So I built a small assistant that answers from a wiki of those documents and nothing else. It runs against llama.cpp as its LLM API, which means the prompts and the content stay on hardware I control. It is two services, about ten small Python modules, and exactly one tool the model can call.

The shape of it

A Next.js frontend, a FastAPI backend, and llama-server doing the thinking. The browser never sees the backend address; every call goes through Next.js server routes. Login is checked against a directory service, and threads and sessions are plain JSON files. There is no database and no vector store.

The llama.cpp part turned out to be the most boring part, which is the highest praise I have for any integration. The backend speaks the OpenAI wire format through a plain HTTP client, so pointing it at llama-server is a base URL and nothing more. Leave the API key unset and no Authorization header is sent at all.

base_url = settings.openai_base_url or DEFAULT_OPENAI_BASE_URL
headers = {}
if settings.openai_api_key:
    headers["Authorization"] = f"Bearer {settings.openai_api_key}"

One requirement matters. The model has to handle tool calling well, and llama-server has to be started with a chat template that supports it (the --jinja flag). Test that before you build anything else on top.

The mistake I corrected

My first design looked exactly like what an agent is supposed to look like. Three tools: list_pages, search_pages, read_page. The model would list, search, narrow down, then read. It felt right, and it was quietly bad.

The search matched the whole query as one literal substring. Ask it about a multi-word problem that didn't appear verbatim in a page and it returned nothing, while a single word matched fine. I could have written a better matcher. Instead I asked a different question: why was the model searching at all?

I deleted both discovery tools and put a plain-text catalog straight into the system prompt, one line per page: title, path, a one-sentence summary. Now the model reads the map and goes directly to read_page. A round trip disappeared from the common case, a whole class of empty-search failures disappeared with it, and the relevance judgement moved to the one component that is actually good at it.

A map with two levels

A flat catalog stops working once it grows, and it invites a worse problem: answering from the wrong section of the wiki. So the catalog has two tiers in the same file format. The root index lists the top-level sections, each with a one-line summary of what it covers. Each section folder has its own index listing its pages. Today that is a handful of sections and a few dozen documents.

The model starts by reading a section's index page. If the read succeeds, the backend pins that section to the conversation and re-injects it into every later turn. If it fails, the assistant says plainly that the section doesn't exist, instead of improvising from somewhere else. I considered a dedicated select_section tool and rejected it, because a successful read of an index path is already unambiguous evidence of what was resolved. No second mechanism to keep in sync.

The summaries are written by the same llama.cpp model, and the reindex step hashes each file so only changed pages are re-summarised. The index describes the wiki in the model's own vocabulary, and editing one document costs a couple of short completions, not dozens.

Instructions live next to the content

The documents are YAML, not prose, and each one carries a block called llm_guidance. It tells the model how to behave for that specific procedure. For a device-down runbook, for example, it says to ask for the hardware model first, the software version second, and to recommend the least disruptive action first.

I like this more than I expected to. It puts the behaviour for one situation beside the procedure it governs, instead of growing a global prompt that has to be right for everyone at once. It is the same instinct I have always had with integrations: each system should own the description of itself.

What I would fix

A review pass over the finished code was humbling in the useful way. The section pin is enforced by the prompt, not the tool, so nothing technically stops the model reading another section mid-conversation. The index is cached for the life of the process, so a reindex needs a backend restart to show up. The system prompt file goes through string formatting, so a stray curly brace in it will break every request.

The deeper one is history. Each turn replays the whole conversation, and tool results are not stored, so follow-up questions make the model re-read documents. On a local model with a fixed context window, that is where it will hurt first. None of this is exotic. It is the honest price of flat files and prompt-level rules, and for a small tool with a small audience it was the right trade.

I have spent most of my career connecting systems that each produce something the next one cannot use directly. This was the same problem again: a pile of careful knowledge on one side, a person with a question on the other, and a model in the middle that needed an interface it could reason about. The interface was not a better search. It was a map.

Don't teach the model to search your knowledge. Show it where everything is.

Cover photo: Two colleagues discuss sticky notes on a wall by Vitaly Gariev on Unsplash.