How to build an AI Agent knowledge base: The complete process from document preparation to retrieval acceptance testing
When a knowledge base is poor in quality, the Agent often does not “fail to answer entirely.” Instead, it may retrieve an outdated version, confuse the scope of applicability, or use a seemingly relevant passage to produce an incorrect conclusion. The goal of building a knowledge base is not to store more files, but to enable the Agent to find the correct, up-to-date, and authorized supporting information for a specific question.
What is the difference between a knowledge base, persistent memory, and skills?
| Component | Primary content | Question answered |
|---|---|---|
| Knowledge base | Curated business documents, rules, and FAQs | What information is this based on? |
| Persistent memory | User preferences, runtime state, and historical context | What ongoing context applies to this interaction? |
| Skills | Operating procedures, tool calls, and acceptance criteria | How should this task be performed? |
These three components can work together, but they should not be mixed. Stable policies belong in the knowledge base, temporary user preferences belong in memory, and procedures for calling business systems belong in skills.
Step 1: Define the scope based on user questions
First, list the questions the Agent must answer or handle, and then work backward to determine which materials are needed. A customer service knowledge base can initially cover product usage, common issues, and handling policies instead of importing the entire shared drive at once.
- Which authoritative source corresponds to each question?
- Which questions must be declined or escalated to a human?
- Do different regions, products, or customer types have different rules?
- Who is responsible for the content, and how long is it valid?
Step 2: Clean up the materials instead of uploading them directly
- Delete duplicate copies and clearly outdated versions.
- Preserve titles, sections, and lists rather than leaving only unstructured plain text.
- Run optical character recognition on scanned documents first, and manually spot-check the results.
- Label high-risk information such as prices, policies, and permissions with its effective date and scope of applicability.
- Resolve conflicts between contradictory files first instead of asking the model to guess.
Step 3: Chunking and metadata
Each chunk should focus on a topic that can be understood independently. If a chunk is too large, the retrieval result will contain irrelevant content; if it is too small, conditions and context will be lost. Do not begin by pursuing a fixed character count. Validate the chunks with real questions first.
| Metadata | Purpose | Example |
|---|---|---|
| Source | Enables traceability to the original text | Product manual, handling policy |
| Version/date | Prevents references to outdated rules | Effective date, revision number |
| Scope of applicability | Reduces misuse across products | Product line, region, department |
| Permissions | Restricts retrieval visibility | Public, internal, specific team |
| Owner | Facilitates updates and corrections | Content owner |
Step 4: Create a set of acceptance test questions
At a minimum, the test set should cover questions that can be answered directly, questions that require combining multiple passages, questions for which the materials contain no answer, questions involving versions that are easy to confuse, and questions involving unauthorized information. During evaluation, do not assess only the fluency of the wording.
- Does retrieval find the correct passage?
- Is the answer faithful to the passage without fabricating facts?
- Does it identify the supporting information or provide a traceable source?
- When the available information is insufficient, does it clearly say that it does not know?
- When permissions are insufficient, does it refuse to return the content?
Step 5: Incorporate maintenance responsibilities into the process
A knowledge base will continue to age after launch. Every critical document should have an owner, a review schedule, and a retirement method. When a new version is released, determine whether the old version should be retained, deprioritized, or removed.
- The business owner submits an update.
- The reviewer verifies the facts, scope of applicability, and effective date.
- The system creates a new index or updates the record.
- Regression questions are used to validate critical answers.
- The change is recorded, and the old version is handled appropriately.
How do you troubleshoot common issues?
| Symptom | Possible cause | Recommended action |
|---|---|---|
| Cannot find the answer | Missing scope, inappropriate chunking, or different terminology used by users | Add materials, adjust chunking, and include synonymous expressions |
| References an outdated policy | Missing version metadata or old content that has not been retired | Add dates and priorities, and handle the old version |
| Information leaks across departments | Incorrect permission inheritance or indexes that are not isolated | Redesign knowledge domains and access controls |
| The answer seems reasonable but is unsupported | Generation constraints are too weak | Require answers to be based only on retrieved content and allow the Agent to decline to answer |
Frequently asked questions
Is a knowledge base the same as RAG?
No. A knowledge base is responsible for content and its management, while RAG is a type of method that retrieves relevant content from the knowledge base and provides it to the model. Both storage quality and retrieval strategy are essential.
Do you need to retrain the model after updating the knowledge?
Knowledge retrieval usually does not require retraining the model, but you must confirm that the new content has been processed, indexed, and validated through regression testing.
How should SmaugBrain use a knowledge base?
Business materials can be combined with task workflows, allowing the Agent to consult supporting information while performing tasks. The formats, capacity, and retrieval methods actually supported depend on the specific configuration and should not be inferred directly from feature descriptions in outdated articles.
Final checks before launch
- The scope is driven by real questions, not by the number of files.
- Every item of high-risk knowledge has a version, scope, and owner.
- The test set includes questions with no answers and questions involving unauthorized information.
- Answers can be traced back to their sources.
- Processes for updates, rollbacks, and deletion have been clearly defined.