The company assistant gives an answer in the demo. The source is stale, one project is missing, a former employee can still retrieve a confidential page, and nobody knows what happens when sync fails.
The demo worked. The system did not.
A useful company brain needs evidence across six gates: coverage, freshness, permission correctness, retrieval and answer quality, failure recovery, and business value. Test them separately. A high-quality answer cannot compensate for an access failure; an accurate pilot cannot justify a system no one uses.
This is the last part of the guide: it tests what the previous eight parts build. Source authority is part 3, ingestion and freshness part 5, what the assistant may do part 7 and retrieval permissions part 8. The full build path is on the guide page.
The answer in 60 seconds
- Define the business decision or task before selecting metrics.
- Build a representative test set with expected sources and acceptable abstention.
- Test coverage, freshness and permission changes.
- Measure retrieval separately from answer usefulness.
- Inject sync, API, duplicate, timeout and recovery failures.
- Compare observed use and correction effort with a baseline.
- Re-run the gate after material changes to sources, permissions, models, workflows or versions.
Do not ask for one “accuracy” percentage. Ask which gate failed, what evidence proves it, who owns the repair and whether the system should continue, narrow, pause or stop.
In this article
- Start with the business decision
- Build a representative test set
- Test coverage, freshness and permissions
- Separate retrieval from answer quality
- Test failures, recovery and security operations
- Measure value and make a go/no-go decision
1. Start with the business decision
“Help employees find information faster” is too broad to evaluate.
Name one task: prepare a client response from approved delivery documents, find the current onboarding policy, or turn a meeting decision into a draft issue. Record:
- user and moment of need;
- source systems and authority rules;
- acceptable evidence;
- consequence of a wrong, stale or unauthorised answer;
- current manual baseline;
- owner of the operating decision.
NIST’s AI Risk Management Framework organises work around Govern, Map, Measure and Manage. Its Core says organisations should decide whether a system achieves its intended purpose and whether deployment should proceed. That is the right evaluation frame: evidence for a decision, not a decorative dashboard.
The consequence also sets the test depth. A missing internal lunch menu and a wrong contractual answer should not share one acceptance rule.
2. Build a representative test set
Start with questions real users ask, including difficult and negative cases. For each case, capture:
- question and user role;
- expected authoritative source or sources;
- source version and expected freshness;
- content that must not be retrieved;
- minimum useful answer;
- acceptable abstention;
- reviewer and review date.
Include ordinary questions, ambiguous language, old terminology, cross-system conflicts, missing evidence, newly changed documents and permission-denied cases. Keep a stable core for comparison and add cases from real failures.
n8n describes evaluation as running a test dataset through an AI workflow, often with expected outputs. Its quick evaluations support visual comparison of a small set before formal metrics justify their cost. These are useful mechanics. The validity still depends on whether the dataset represents the business task and its failure modes.
NIST’s Generative AI Profile offers voluntary risk-management actions and explicitly notes that not every action fits every actor or use case. Apply the same discipline to testing: use the smallest set that covers the material decision, then expand from observed failures.
3. Test coverage, freshness and permissions
Before judging the prose, verify the evidence layer.
| Gate | Question | Minimum evidence | Stop or narrow when |
|---|---|---|---|
| Coverage | Are the required sources and objects represented? | Source inventory, sampled objects, known exclusions | A required domain is missing or silently partial |
| Freshness | Do changes arrive within the promised operating window? | Source/update timestamps, lag, failed syncs | The system cannot identify stale or failed updates |
| Permission correctness | Can allowed users retrieve - and denied users not retrieve - the same objects? | Role matrix, allow/deny tests, revocation case | Unauthorised text reaches retrieval or model context |
| Retrieval and answer | Does the system find the right evidence and use it honestly? | Expected evidence, citations, reviewer outcome | Confident answers appear without sufficient sources |
| Recovery | Can operators detect, reconcile and restore after failure? | Error path, replay test, final-state check | Duplicate, missing or corrupted state is unresolved |
| Business value | Does observed use improve the named task after correction and operating cost? | Baseline, adoption, handling time, corrections, owner decision | Burden or risk outweighs the observed improvement |
Coverage is not connector status. Freshness is not “the workflow ran”. Permission correctness is not a successful login.
Run known-change tests: edit an authoritative page, remove a user from a group, delete a source and force a connector failure. The company brain should expose the resulting state, abstain where necessary and recover through a defined path.
OWASP’s Vector and Embedding Weaknesses includes cross-context leakage, poisoning, access-control gaps and monitoring among RAG concerns. These belong in the test set, not only the security review.
4. Separate retrieval from answer quality
When an answer is wrong, identify the failing layer:
- source: the authoritative information is wrong, missing or contradictory;
- ingestion: the current object or permission state did not reach the index;
- retrieval: the right evidence was available but not selected;
- generation: the model ignored, distorted or overextended the evidence;
- presentation: citations, uncertainty or next action were unclear;
- workflow: the answer triggered the wrong downstream state.
Evaluate retrieval with expected documents or passages, not only a reviewer’s opinion of the final prose. Evaluate answers for usefulness, grounding, completeness within the available evidence and honest abstention. A fluent answer from the wrong document should fail.
Keep human review where business context matters. Model-based scoring can help compare many runs, but it should not become the sole judge of a consequential workflow. NIST’s Core treats measurement as quantitative, qualitative or mixed, and expects metrics and controls to be reassessed as risks evolve.
5. Test failures, recovery and security operations
Most pilots test the happy path. Production fails in combinations:
- an API times out after a write succeeds;
- a webhook arrives twice or out of order;
- a source returns partial results;
- a credential expires;
- a model or workflow version changes output structure;
- an index update stops halfway;
- an operator retries with the wrong workflow state.
n8n’s execution views expose stored runs and retry options. Error workflows can route failures. Log streaming can send selected events to external monitoring on eligible plans, while OpenTelemetry tracing is documented as under development. Visibility helps; recovery still needs a final-state check and named operator.
Security maintenance belongs in the same operating gate. n8n’s security audit checks listed credential, database, filesystem, node and instance conditions. Its update guide makes version maintenance explicit, and source-control environments can support separated development and protected production on eligible plans.
The 25 February 2026 n8n sandbox-escape advisory and related security bulletin are dated examples of why version inventory, editor trust, isolation and updates need owners. They do not establish that a reader’s instance is vulnerable.
Related: IZZY’s guide to AI agent security controls.
6. Measure value and make a go/no-go decision
Value evidence begins with the manual baseline from section 1. Observe:
- eligible users and actual use;
- successful task completion;
- handling time before and after;
- corrections, escalations and rework;
- failures and time to recover;
- model, infrastructure and operating cost;
- decisions delayed or improved.
n8n Insights reports production execution metrics and can use a configured time-saved estimate. Treat configured time saved as an assumption until sampled against real work.
Illustrative Atlas evaluation
Atlas tests a project assistant. Its answers look good, but the gate finds an unknown update lag and duplicate tasks after a repeated webhook. Atlas does not average these failures into one accuracy headline. It pauses expansion, repairs freshness monitoring and duplicate handling, then re-runs the same cases.
Atlas is illustrative, not an IZZY client case. No result was tested.
Finish each review with one decision: continue, expand, correct, narrow, pause or stop. OWASP’s Excessive Agency is a reminder that more tools and autonomy are not the default reward for passing. Authority should expand only when operating evidence supports it.
Conclusion: evaluate the system, not the demo
A company brain works only when the right people can retrieve current, authorised evidence; the workflow handles failure; and the named task improves enough to justify its burden.
Test all six gates. Keep retrieval separate from answer quality. Inject failures. Re-run after material change. Let the evidence support a go/no-go decision.
Evaluate one company-AI workflow with IZZY
Bring one workflow, its user, source systems, manual baseline, ten representative questions and one known failure. We will map the six gates and define the evaluation.
The outcome may be expansion, a corrective roadmap, a narrower scope - or a stop decision before more budget is spent.
This is exactly the scope of our n8n AI Automation service.
Frequently asked questions
At minimum: source coverage, freshness, permission correctness, retrieval, answer usefulness, abstention, failure recovery and the business task against a baseline.
There is no universal number. Start with a representative set covering ordinary, edge, conflict, stale, missing and denied cases; add real failures and disclose the coverage.
It can support comparison at scale, but consequential decisions still need known evidence, calibrated criteria and human review. Do not let one model-generated score hide a permission or recovery failure.
After material changes to sources, permissions, models, prompts, tools, workflows or versions, and on a cadence proportionate to the consequence and rate of change.
Sources and evidence boundary
Research checked on 28 July 2026 against the primary or official sources linked in the article. The six-gate scorecard and Atlas scenario are IZZY guidance.
No source inventory, user role, evaluation dataset, answer, permission change, workflow execution, duplicate event, restore, n8n instance, configured time saving, cost or business outcome was tested. The article defines an evaluation method; it reports no measured accuracy or ROI. Product documentation and security guidance can change after the research date.