AI knowledge bases: prepare your documents for reliable retrieval

Define the questions the assistant should answer

An AI knowledge base should begin with a bounded job. A support assistant that explains approved troubleshooting steps needs different material from an internal assistant that answers questions about commercial contracts. List the intended users, questions and decisions. Record which questions should be escalated because the available evidence is insufficient or the consequence requires human judgement.

Retrieval-augmented generation, often called RAG, supplies selected external information to a model when it produces an answer. It can make relevant documents available at answer time, but retrieval is not a guarantee of correctness. A system can retrieve the wrong passage, miss an important exception or produce a confident answer that its sources do not support.

Establish an authoritative document set

Give each document an owner, status and review date. Identify duplicates, superseded versions and contradictory guidance before ingestion. If two policies disagree, ask the responsible owner to resolve the conflict rather than expecting the model to decide which one has authority. Preserve the original source location and version so a reviewer can trace an answer back to its evidence.

Choose a manageable initial collection. A smaller set of current, well-structured documents can be easier to evaluate than a large uncontrolled archive. Keep headings, tables and exceptions together where their meaning depends on context. Check extraction quality on scanned documents and complex layouts; a correct source file can still become misleading text after conversion.

Apply permissions before information reaches the model

A user should not gain access to a restricted document simply because it has been copied into a search index. Carry relevant access information through ingestion, retrieval and updates. Enforce authorisation using the authenticated user's permissions at a trusted application boundary, and test the results with accounts that have different roles.

OWASP identifies information leakage, access-control weaknesses and poisoned data among risks associated with vector and embedding systems. [2] In practical testing, include a document that one role may access and another must not. Check search results, generated answers, citations, caches and logs. Permission changes and document deletion must propagate to the systems that can still expose the content.

Treat retrieved material as untrusted input

A document can contain instructions addressed to a model as well as ordinary business information. OWASP's prompt-injection guidance explains that external content can influence model behaviour and that RAG does not fully remove this risk. [1] Keep system instructions separate from retrieved content and avoid treating text found in a document as permission to run tools or disclose information.

For an assistant that can take actions, enforce tool permissions independently of the model's wording. Require appropriate checks for consequential operations. A knowledge answer and a change to a customer record are different capabilities and should have different controls. Start with a constrained answer-only workflow where that is sufficient for the intended use.

Build a small evaluation set before launch

Collect realistic questions from the people who will use the assistant. Include clear answers, ambiguous requests, outdated terminology and questions the documents cannot answer. For each, record the supporting passage or the expected escalation. Evaluate source selection, factual support, completeness and whether the response acknowledges missing evidence.

Check citations by opening them and comparing the answer with the cited passage. The presence of a citation is not proof of support. Repeat the evaluation after changing the model, retrieval settings or document processing. Keep representative failures in the test set so an apparent improvement does not quietly reintroduce a known problem.

Operate the knowledge base as a service

Define how new documents are approved, how old versions are retired and how urgent corrections reach the live index. Monitor unanswered questions and user feedback, while protecting sensitive conversation content. Assign responsibility for investigating recurring errors and give users a clear route to a human when the assistant is uncertain or wrong.

Evo's AI Agents service can help connect a defined business workflow with the right information and controls. Bring sample questions, document owners and permission rules to the project. A useful first release has a bounded scope, traceable answers, an evaluation record and a maintained process for keeping its knowledge current.

Frequently asked questions