What you’ll take away
- Run the same question as two project members and check the retrieved passages, answer, snippets, and citation titles against their source permissions.
- Measure what a former project member can retrieve after removal, including responses served from a cache.
- Report the affected path, a reproducible test case, and the evidence required to approve or narrow the rollout.
In this guide
Define what each project team should be able to learn
Two engineers at a fictional consultancy ask their internal AI assistant the same question. Which survey code appears on the current field-release sheet? The Alder engineer should get the Alder code and a link to the Alder sheet. The Birch engineer should get a different code and source. Both can read the company handbook. Each client also treats its drawing titles and revision details as private, so the sources displayed underneath the answer matter as much as the answer itself.
The consultancy uses retrieval-augmented generation, or RAG. When someone asks a question, the application searches connected documents and sends selected passages to a model to help it answer. The release decision is whether employees from both projects can safely use that workflow together. An accurate answer shown during an administrator’s demonstration says little about which passages a project engineer can retrieve.
Start with the people who own the documents. Have the Alder and Birch leads identify what each team can read, what both teams share, and what should happen when someone changes projects. Ask the application owner which surfaces can reveal material, including search suggestions, quoted snippets, citation cards, and conversation history. Record the agreed expectations before running the test so the product team has a clear basis for a release decision.
Follow one document and one identity through the system
Trace the Alder sheet from the source folder through the connector, index, retrieval service, model input, and citation display. A connector might read every project as one service account. At question time, the system still needs the engineer’s effective permissions to select eligible material. Ask the team to show where the signed-in identity becomes an access decision, how project membership is established, and which component applies that decision to a citation lookup.
Microsoft documents one Azure AI Search pattern in which an application includes a user or group identifier in a search filter. The filter compares strings, leaving the application responsible for establishing the caller’s identity. [1] For a system using this pattern, trace a project identifier from its trusted source. Try changing a project selector in an authorized test request and check that an Alder user still receives only data allowed under the source policy.
Large documents often become several searchable passages, sometimes called chunks. OWASP’s RAG Security Cheat Sheet describes carrying source permissions to the chunks and enforcing them before the model receives the passages. [2] Pick a passage well inside the synthetic Alder sheet and trace its document and access information. Also check how a separate lookup turns a retrieved passage into a title or preview. Those steps can use different indexes, credentials, or caches.
Decide which passages may enter the prompt.
Keep each searchable chunk linked to its source document and access rules.
Derive the engineer’s identity and project membership from a verified session.
Only permitted passages reach the model.
Apply the same rule to titles, snippets, and links.
Write the expected results before asking the assistant
Give each project an ordinary test account. In an approved test workspace, create an Alder field-release sheet containing the invented survey code WREN-27 and a Birch sheet containing FERN-84. Put the same question heading and similar wording in each sheet so either would be relevant to a search. Give the shared handbook a different, verifiable fact, such as the location of the field correction log. Keep the answer codes out of the prompts.
Have a document owner verify access directly in the source system under both accounts. Build a small table showing each engineer’s expected access to each sheet and to the handbook. Add expectations for titles, excerpts, and direct links. OWASP’s authorization guidance recommends testing the permissions established in the design and checking authorization on each request. [3] The source checks give the test team an independent reference against which to judge the assistant.
Include an allowed result alongside every important denial. If both engineers receive an empty answer, the index might still be loading, the connector might have failed, or the question might match neither sheet. A successful Birch retrieval establishes that the Birch fixture is available for its intended reader. An Alder run then tests the boundary around the same content. Verify the shared handbook from both accounts to catch a change that blocks legitimate collaboration.
Same question, different permitted sources.
| Synthetic document | Alder engineer | Birch engineer |
|---|---|---|
| Shared handbook | ||
| Alder field-release sheet | ||
| Birch field-release sheet |
Ask each engineer’s account for the survey code. Confirm its own sheet is retrievable and the other project’s sheet stays out of context and citations.
Inspect the answer and the evidence used to produce it
Ask the survey question through the interface employees will use. Then paraphrase it and try a follow-up that asks for the supporting passage. Start with separate, clean conversations for each account, record the workspace and application build, and preserve request identifiers. Next, run the sequence with the other identity first. A mismatch that depends on which user asked first gives the platform team a reason to inspect reused search results or answers.
At each step compare the rendered answer, source list, snippets, and link destination with the expected result. Where the agreed test access permits, correlate those observations with the IDs of passages supplied to the model and the authorization decision for the request. An application could send a restricted passage to the model even when the final answer omits its contents. Conversely, a model might invent a plausible project detail. A distinctive marker, source ID, and matching request trace help determine what actually happened.
State which part of the path the evidence covers. A screen recording can establish that someone saw a private title. Reliable evidence about the exact model input requires visibility into that stage. Synthetic passages and restricted access to the trace let the team investigate without building a second repository of client material. Keep enough information to replay the test after a fix, including the question, fixture version, account memberships, and cache state where observable.
Work through a failed citation test
Consider the following illustrative result for the fictional consultancy. The project owners have restricted both the contents and titles of the field-release sheets. In a fresh Alder conversation, the assistant gives the correct survey code. The source drawer underneath the answer nevertheless reveals a Birch drawing issue. Clicking that citation reaches a source page that denies access. The reader has already learned the private title.
Classify the observed failure precisely. The evidence supports disclosure of Birch metadata through the citation drawer. Determining whether Birch body text also reached the model requires inspection of the corresponding retrieval and model-input evidence. Have the application owner reproduce the case, follow the source-card lookup, and establish whether it reapplies the engineer’s permissions or reuses a result from another request.
Before widening access, fix the responsible lookup or restrict the affected workflow to a separately verified configuration. Rerun the failed case under Alder and repeat the Birch success case. Check the drawer, preview, and direct link on each. If the repair simply suppresses every citation, the original functional requirement for engineers to check an answer against its source still needs attention.
| Record field | Case R-04 |
|---|---|
| Identity and request | Alder-only engineer asks for the survey code on the current field-release sheet. |
| Expected | WREN-27 with the Alder sheet as evidence. All project source cards belong to Alder. |
| Observed | WREN-27 and the Alder citation appear alongside a card titled “Birch field-release sheet, drawing issue B14.” Opening the Birch link returns access denied. |
| Decision and owner | Hold the two-project rollout. Application owner investigates the citation lookup and retests both accounts. |
Measure access after a project member leaves
A boundary can work at setup and change when someone transfers projects. Agree on the required cutoff with the data owner and decide how often the test will sample each path. First confirm that the Alder engineer can retrieve the synthetic Alder sheet. If the assistant caches answers, repeat an allowed request before removal and confirm a cache hit where observable. Remove the engineer’s Alder membership through the approved source and record when the change was made and when a direct source check first denies access.
Repeat the original question and a fresh variation on a recorded schedule. Exercise an existing signed-in session and a new one. Record cache hits or misses where the platform exposes them. OWASP’s Web Cache Security Cheat Sheet describes how cached responses can retain stale authorization state and recommends testing stored results after permission changes. [4] A request that reuses old output and one that performs fresh retrieval can reach different components.
For each path, report the last observed disclosure, if any, followed by the first denial that subsequent samples confirm. Check the recorded permission decision and keep an authorized account retrieving the same sheet to rule out an outage. Record observation times, sampling interval, and the agreed cutoff. The gap between probes limits how precisely you can time the change. A later disclosure reopens the result. If access persists through the final sample, report the open issue. Distinguish a new server response from text already displayed on someone’s screen. Test reopening a saved conversation separately according to the product’s history policy.
Measure the gap after access is removed.
- Source access removed
Record the change time, user, document, and previous permission.
- Last observed disclosure
If a probe still reveals protected text or metadata, record its time and path.
- First confirmed denial
Record the first verified denial on that path. Check an authorized account still succeeds.
Repeat on fresh retrieval and cache-hit paths. Track each path separately; an empty answer alone does not confirm denial.
Test who can influence an answer through a document
A separate question concerns the integrity of the answer. An engineer may legitimately read a project note whose author inserted a sentence directing the assistant to append a made-up approval stamp. OWASP describes indirect prompt injection as a case where externally supplied material influences model behavior in an unintended way. [5] Here the engineer is authorized to see the document. The issue is whether its author can give the assistant instructions by putting them in the source.
Use a harmless marker in a synthetic note and an agreed upload path. Ask for a factual detail from the note, verify that the test passage reached the model, and inspect whether the assistant adds the unsolicited stamp. Compare with a clean copy containing the same factual detail. If the marked passage was never retrieved, record an inconclusive instruction test and adjust the fixture. Document separate observations for passage selection and the assistant’s behavior once it received the passage.
For the consultancy, review who can publish to an indexed workspace, how the application distinguishes source text from trusted instructions, and whether employees can verify claims against the original. An assistant that can also change tickets or publish messages introduces additional actions to scope and authorize. For this release decision, keep the confidentiality case about the Birch citation separate from the integrity case about an Alder author’s influence.
Make the release decision specific to the tested boundary
Give the release owner a short record of the candidate build, sources, identities, query paths, access changes, and outstanding failures. For every material case, show the expected result, what observers saw, who owns the response, and how to verify it. If evidence ends at the user interface, identify any deeper claim about retrieved passages that still needs testing. Keep the supporting request IDs and synthetic fixtures available to the team doing the repair.
For Alder and Birch, an unresolved private citation blocks the proposed rollout across both projects. The release owner could consider a smaller pilot using only the shared handbook after the team independently verifies that both project collections are excluded across retrieval, citations, and cached responses. Assign the decision to the person authorized to approve that scope. Set another review when a connector, permission model, index configuration, cache policy, or citation feature changes.
Source Point Security’s RAG Pipeline & Data Boundary Testing maps the permission model, tests agreed identities and sources, and delivers an expected-versus-observed access matrix, evidence from an instruction-bearing test document, a measured permission-change timeline, and targeted retesting. The broader LLM Application Penetration Test covers the surrounding application, including sign-in, sessions, prompt handling, and output behavior. Bring the source owners, a description of the proposed pilot, and the access decision leadership needs to make.
Test your document assistant before expanding access
Bring the connected sources, project roles, and proposed rollout. We can scope a RAG security test around the access decisions you need to verify.
Discuss RAG Pipeline & Data Boundary Testing