# Company-document AI evaluation worksheet

By Varveta · Updated 2026-10-02

Evaluate retrieval and answer quality separately. For each question, record the user’s permissions, approved evidence, expected behavior and actual result. These proposed test cases are a starting worksheet, not claims about a tested system.

## 1. Define the test context

Workflow: [task and business owner]. Source collection: [approved documents and owners]. Evaluation date: [date]. Configuration: [model, retrieval settings and deployed version]. Test identities: [users and permitted sources].

Use synthetic or approved non-sensitive data. Keep access tokens and restricted document text out of shared results. Store detailed evidence in a location with appropriate permissions.

## 2. Adapt the test matrix

Specify the expected source and behavior before running each case. A plausible answer is not a pass if the evidence does not support it.

| Case | Expected behavior | Evidence to capture |
| --- | --- | --- |
| Answer exists in one approved document | Answer follows the source and cites the supporting passage. | Retrieved passages, answer and citation target. |
| Requested fact is absent | State the evidence gap; do not invent the missing fact. | Question, retrieved context and answer. |
| Two approved policies conflict | Identify the conflict and follow the agreed review route. | Both sources and escalation behavior. |
| Document is restricted for this user | Restricted content does not enter retrieved context, answer or citations. | Access-test outcome for permitted and restricted identities. |
| Document was updated or removed | Respect the agreed refresh/deletion behavior; avoid presenting obsolete text as current. | Source change time, index state and answer. |
| Retrieved document contains instructions to the assistant | Treat source text as evidence, without granting it authority to change the workflow. | Retrieved passage and observed behavior. |

## 3. Record failures separately

For each run, record the case ID, retrieval result, answer result, citation result, access result and evidence link. Distinguish a missing source from an answer that misuses a correctly retrieved source.

Track response time and cost under the workload you actually expect. Agree acceptance criteria with the workflow owner; this worksheet intentionally does not invent a universal passing score.

## 4. Make a release decision

Decision: [release / hold / limited pilot]. Reviewer: [name and date]. Unresolved failures: [case IDs, consequences, owners and next actions]. Re-evaluation triggers: [source, permission, retrieval, prompt or model changes].

- [ ] Access failures are investigated before expanding access.
- [ ] Unanswerable and conflicting questions have an agreed response.
- [ ] Citations support the claims they accompany.
- [ ] A named owner handles incorrect answers and ingestion failures.
- [ ] The receiving team can rerun the evaluation after a change.

## References

- [Microsoft: RAG with Azure AI Search](https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview)

- [Microsoft: document-level access control](https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview)

Illustrative template. Adapt the checks and operating procedure to your environment. This is not a completed test or incident record.