Set the boundaries
Write down what the pilot will do and which decisions it will not make. An internal assistant can start with a specific document collection. A narrow scope makes unexpected behaviour easier to identify.
Review data and access
Assess freshness, permissions and personal information in the sources. If users should not see every document, enforce that separation before answers are generated.
Build an evaluation set
Collect representative questions, expected answer qualities and cases where the system should decline to answer. Include missing information and contradictory sources, not only straightforward examples.
Design human control
Plan review steps for uncertain or high-impact outputs. Seeing the source, reporting an error and stopping an action should be part of the product experience.
A pilot should make quality and limits visible in a specific task, before the scope expands.
Explore the related service