An AI assistant for company documents needs more than good answers

An assistant that answers questions about your company documents sounds straightforward. Connect the files, add a chat window and let people ask. The difficult questions arrive when two policies disagree, someone changes department, or yesterday’s answer is no longer valid.

Retrieval-augmented generation, usually shortened to RAG, searches a knowledge collection and supplies relevant material to a language model. It can make an assistant more useful, but it does not guarantee a correct answer. The quality of the search, the source material and the resulting interpretation still matter. Anthropic’s explanation of retrieval describes why even splitting documents into smaller passages can lose important context.

Here is how we would scope a hypothetical internal policy assistant. This is an illustrative design, not a claimed client implementation.

Start with one question worth answering

Imagine an employee asking, “Can I book a hotel before my manager approves the trip?” The useful result is a short explanation of the applicable policy, its effective date and the relevant passage. It should distinguish requesting permission from actually receiving it.

For a first release, we would choose a small approved collection: the travel policy, expense procedure and booking instructions. HR case files and individual salary records would remain outside that scope. Microsoft’s RAG preparation guidance similarly starts with the business domain, representative documents and answerable questions.

That gives the business something concrete to accept: employees can find the right procedure without treating the assistant as an approval authority.

Permissions belong in the application

The server must establish who is asking and enforce their current access before retrieved text enters the model’s context. A user typing “I’m the finance director” must not change that decision. Permission metadata needs to remain attached to each searchable passage, with tenant boundaries enforced where several organisations share infrastructure.

Microsoft documents security filtering at query time. Its filter pattern is not itself authentication: the application must supply trustworthy identity information and apply the restriction consistently.

In our example, an employee could read the general travel rules without seeing a colleague’s exceptional reimbursement agreement. We would also recheck access when serving a cited document. If authorisation cannot be established, the system should stop rather than ask the model to decide what is confidential.

A citation must support the actual claim

A source link makes an answer inspectable; it does not prove that every sentence is correct. “Manager approval is required” and “approval can happen after booking” are different claims, even if both answers cite the same PDF.

We would retain source identifiers, versions and passage references, then check that the displayed citation exists and supports the statement beside it. Google’s grounding documentation makes the same useful distinction between a claim and the evidence supporting the whole claim. Automated checks help; they still need representative human review.

For the employee, that means opening the relevant paragraph rather than searching a forty-page policy for a sentence the assistant may have misread.

Old answers need an expiry path

Suppose finance replaces the travel policy on Monday. Updating the original PDF is only part of the change. Our design would refresh affected passages and invalidate dependent answer caches. Deleted documents and revoked permissions need equivalent treatment. OWASP’s RAG security guidance covers these derived-data and cache risks.

We would agree a freshness target with the policy owner, record effective dates and surface failed synchronisation. Where currentness cannot be confirmed, the assistant should say so. An apparently precise answer from an obsolete policy can create more work than an honest interruption.

Test the awkward questions before launch

The acceptance set should include an ordinary booking question, conflicting policy versions, an unanswered exception, a revoked user and a document containing malicious instructions. Retrieved text is evidence to inspect, not permission to change the assistant’s behaviour or perform an action.

Measure retrieval and answer quality separately. Microsoft’s evaluation guidance explains why groundedness, correctness and completeness answer different questions. We would also review whether the assistant appropriately declines unsupported requests.

“I cannot confirm this exception; please ask the travel-policy owner” is a useful outcome when the evidence is insufficient. The handoff should explain what is missing without exposing restricted information.

The first delivery should include the agreed sources, access rules, test cases, update ownership and operating costs. Describe the documents and questions your team struggles with, and we can define a focused pilot with clear acceptance criteria.