Start with one business workflow, a defined set of documents and the people allowed to use them. Agree what a correct answer looks like, how access is enforced, and what the assistant should do when evidence is missing. Retrieval-augmented generation (RAG) supplies context; it does not guarantee correct answers.
Choose a question the business needs answered
A first release needs a narrower purpose than searching every company document. Choose a repeatable task, such as helping an internal support team find the current approved procedure. Name the business owner and the users, then collect representative questions before choosing an interface or model.
Decide which decisions remain with a person. Finding a policy, interpreting an exception and approving a customer commitment are different responsibilities. A useful assistant can locate evidence without receiving authority to approve an action.
- Identify the workflow owner and intended users.
- Collect real questions, including ambiguous and unanswerable ones.
- Define actions that require review or remain outside the assistant’s scope.
Make source ownership and freshness explicit
Inventory the sources that actually answer those questions. Record the owner, update process and authoritative version of each collection. Duplicated policies, scanned files and old attachments can make retrieval harder even when an index accepts them successfully.
Plan how revisions, deletions and permission changes reach the search index. Test a changed document and a removed document, not just the first import. The person reading an answer should be able to open its source and understand whether that source is current.
Enforce document access before generating an answer
An assistant should retrieve only documents the requesting user is authorized to access. Azure AI Search describes document-level controls that carry permission information into retrieval and filter results for the caller. The mechanism depends on the data source and supported features; verify it for the chosen architecture.
Test access with two users who have different permissions. A restricted document should not appear in the retrieved context, answer or citation for the unauthorized user. A prompt asking the model to keep secrets is not a substitute for access enforcement.
Agree how answers will be evaluated
Create a small evaluation set from the workflow’s real questions. For each, record the approved evidence, expected answer and conditions that require abstaining or escalation. Include conflicting documents, outdated information and questions with no supported answer.
Review whether retrieval found the right evidence separately from whether the answer used it correctly. Check that citations support the claim, rather than merely linking to a related document. Agree acceptable response time, cost and review requirements with the workflow owner before expanding access.
Plan who operates the assistant after launch
Define who handles ingestion failures, incorrect answers, access problems and model changes. Keep the evaluation cases, configuration and operating notes with the implementation so the team can test a change before releasing it.
A bounded first scope can cover one document collection and one user group, with explicit acceptance checks and a review point. Expand when the workflow works reliably enough for its intended use, not simply because the demonstration produced a convincing answer.
Technical references
Examples and templates are illustrative. Adapt acceptance criteria and operating procedures to your environment.