Private-document AI search: retrieve answers without widening access

Fix the knowledge collection before adding chat

Select documents with a clear business purpose and owner. Remove obsolete duplicates from the searchable set through the organisation's approved content process. Record the document identifier, current version, effective date and source link so an answer can point back to something a person can check.

Break long documents into passages that preserve headings and surrounding context. A paragraph about an exception is misleading if separated from the condition that activates it. Test real questions from staff, including acronyms, misspellings and terminology used locally rather than only the formal document titles.

Carry permissions through every processing stage

Create a map from the source document's access rules to its indexed passages. The retrieval service must use the requester's validated identity and apply access restrictions before content is passed to the model. Filtering only the final displayed citations is too late if restricted passages already entered the prompt.

Microsoft's Azure AI Search documentation describes document-level permission approaches and notes that several native capabilities are in preview. Check the chosen service's actual production support, source compatibility and permission-refresh behaviour. Do not assume a connector preserves every inherited permission or that revoking source access instantly updates an index.

• Test a user with no access, partial access and recently revoked access.
• Keep permission metadata attached to each passage.
• Include deletion and permission updates in the indexing process.

Make the answer inspectable

Require the interface to show the sources used and let the reader open the original document. Ask the system to distinguish a direct policy statement from an interpretation. When documents disagree, show the conflict and their dates rather than blending them into a confident answer.

Use a no-answer path for questions outside the collection. For example, an internal travel policy may explain expense approval but cannot determine the current immigration requirements for a destination. Route that question to an appropriate source or owner instead of filling the gap from model memory.

Evaluate freshness as well as relevance

Build test questions with known supporting passages and expected access boundaries. Measure whether the system retrieves the right evidence, whether the answer follows that evidence and whether links still work. Include recently replaced documents, tables and scans with difficult text.

Decide how quickly an approved document change must reach search. Monitor indexing failures and provide a visible content-owner route for corrections. Keep the search collection limited to its intended purpose; a helpful internal assistant does not need unrestricted access to every mailbox or personal folder.