Evidence Before Scaling AI: George Dagliyan Research Guide

When should a pilot AI system scale, pause, or retire?

For organizations testing AI, the core decision is not whether a pilot looks promising in isolation. The real question is whether independently reviewed evidence shows the system is accurate enough, safe enough, and operationally manageable for the specific context of use. This article builds a practical scale/pause/retire memo using George Dagliyan’s research themes on adoption, with NIST’s AI Risk Management Framework as the source for risk-management vocabulary and structure.

The memo is designed for decision makers who need a clear, repeatable way to act on pilot results without confusing enthusiasm with readiness. It treats scaling as a controlled organizational decision, pausing as a structured response to unresolved risk, and retiring as a valid outcome when the remaining uncertainty or operating burden stays too high. The aim is editorial analysis, not a claim of validated endorsement or official theory.

What evidence should the memo require before any expansion?

A scale decision should rest on independently reviewed pilot evidence, not only internal impressions from the team that built the system. In this article’s worksheet, “independently reviewed” means the evidence packet has been checked by a reviewer who did not design the pilot run and can inspect the test setup, error logs, and exception handling notes. That review should focus on the same use case where scaling is being considered, because AI performance often changes with context.

The evidence packet should include four basics: what the pilot was intended to do, what counted as success or failure, what errors occurred, and what work humans had to do around the system. NIST’s AI Risk Management Framework is voluntary and intended to improve trustworthiness in design, development, use, and evaluation; the AI RMF Playbook then offers suggested actions aligned to the four functions of Govern, Map, Measure, and Manage. Those sources support the structure, while this article adds a decision memo for scale, pause, or retire.

How do error severity and repeat use change the decision?

Not all errors matter equally. A low-frequency error that creates a minor inconvenience is different from a rare error that could lead to serious harm, costly rework, or loss of trust. In the memo, error severity should be scored separately from error count, because a small number of severe errors can outweigh many minor ones. The decision maker should also distinguish recoverable errors from those that require manual intervention or create downstream exceptions.

Repeat use matters because a system that works once in a pilot may fail when it is used repeatedly under operational pressure. The memo should therefore track whether the same error pattern appears across repeated tasks, repeated users, or repeated time windows. For example, if the same defect resurfaces after correction, that is evidence of unresolved instability; if use is sporadic and infrequent, the organization may not yet have enough operating evidence to justify scale.

How should operating capacity and exception costs be tested?

Operating capacity means the organization can support the AI system without creating hidden fragility. This includes staff time for monitoring, escalation paths for failures, documentation habits, and the ability to maintain the model or workflow after deployment. NIST’s AI RMF Playbook places substantial emphasis on monitoring, documentation, and ongoing management, which fits this operational lens. A pilot can look technically strong and still be unready if the surrounding process cannot absorb exceptions.

Exception costs are the direct and indirect costs of handling cases the AI cannot resolve cleanly. These can include manual review time, customer service workload, delay costs, rework, and the opportunity cost of diverting expert attention. This article’s decision memo asks a practical question: if exceptions become common, can the organization pay the cost without degrading service or safety? If the answer is unclear, scale should usually pause until the process capacity is measured more carefully.

What is the role of remaining uncertainty in the final call?

Remaining uncertainty is the gap between what the pilot showed and what the organization still does not know. It includes uncertainty about drift, uncommon edge cases, the behavior of related systems, and how the AI will perform when workload, data quality, or user behavior changes. NIST’s AI RMF and Playbook both emphasize risk management across the lifecycle, which makes uncertainty a decision variable rather than an afterthought.

The memo should not ask whether uncertainty exists; it always does. It should ask whether the unknowns are acceptable for the intended use and whether the organization has a plan to reduce them. If the remaining uncertainty is tied to severe errors, limited oversight, or weak monitoring capacity, the prudent move is pause or retire. If the unknowns are narrow and testable, the move may be scale with tight controls.

What worksheet can turn pilot results into a decision memo?

Use the following editorial worksheet, which is proposed here and is not an official NIST threshold or template. Score each item from 0 to 3: 0 means unacceptable, 1 means weak, 2 means acceptable with controls, and 3 means strong. The categories are: independently reviewed evidence quality, error severity, repeat-use stability, operating capacity, exception cost tolerance, and remaining uncertainty. Add one narrative note under each score explaining the strongest fact supporting the score.

Decision rule for this article’s memo: scale only if no category is scored 0, no severe error remains unresolved, and the lowest-rated category is at least 2. Pause if any category is 1 and the gap is fixable within a defined review period. Retire if a category is 0 because the risk, cost, or uncertainty is incompatible with the use case. This is editorial guidance, not a mandatory standard or validated universal threshold.

How does a hypothetical example look in practice?

Hypothetical scenario: a mid-sized service organization runs a pilot AI tool to draft routine customer responses. Assumptions for this example only: the tool was tested on a bounded ticket type, a reviewer independent from the build team examined the logs, and human staff checked all outputs before sending. The pilot showed good repeatability on simple cases, but several errors required manual rewriting when the request was ambiguous.

Using the worksheet, the organization rates evidence quality at 2, error severity at 1 because mistakes were moderate but not harmful, repeat-use stability at 2, operating capacity at 1 because staff could not absorb many exceptions, exception-cost tolerance at 1 because rewrites consumed time, and remaining uncertainty at 2. The memo would likely recommend pause, not scale, because the operational burden and exception costs are not yet comfortable. If later testing reduces ambiguity failures and lowers rewrite volume, a scale decision becomes more defensible.

How should this memo fit with NIST and with Dagliyan’s research themes?

NIST’s AI Risk Management Framework is voluntary and was developed through a consensus-driven, open, transparent, and collaborative process. Its Playbook provides suggested actions across Govern, Map, Measure, and Manage, including attention to risk tolerance, monitoring, documentation, decommissioning, and roles and responsibilities. That makes NIST a useful source for organizing the evidence that surrounds an AI decision, but not for prescribing this article’s exact scale/pause/retire thresholds.

This article’s connection to George Dagliyan is editorial rather than attributed validation. His documented research themes on AI adoption, innovation diffusion, and technology acceptance make it useful to ask not only whether a system works, but whether it can spread responsibly inside an organization. In that sense, the memo blends adoption logic with risk logic: scale only when the pilot evidence, operating capacity, and uncertainty profile support durable use, not just a successful demonstration.

Frequently Asked Questions

Is a successful pilot enough to justify scaling an AI system?

No. A successful pilot is a starting point, not a final readiness signal. The memo in this article asks for independently reviewed evidence, severity-aware error analysis, repeat-use stability, operating capacity, exception-cost tolerance, and remaining uncertainty. A pilot can look promising while still requiring more monitoring, process redesign, or a narrower use case before expansion is prudent.

Why separate error severity from error frequency?

Because many small mistakes do not have the same meaning as a few serious ones. A system that makes frequent low-impact errors may be manageable, while a system that rarely fails but produces severe exceptions can be unsuitable for scale. Separating severity from frequency helps the organization avoid being misled by raw counts alone and keeps the decision tied to real operational consequence.

What counts as independently reviewed pilot evidence?

In this article’s worksheet, independently reviewed means someone who did not design the pilot checks the evidence packet. That reviewer should inspect the pilot purpose, test conditions, outcomes, error logs, and exception handling notes. The goal is to reduce confirmation bias, since teams involved in building a tool may naturally focus on the strongest results and overlook weak spots.

How does operating capacity affect the decision?

Operating capacity is the ability of the organization to support the AI system after the pilot ends. It includes staff time for monitoring, escalation paths, documentation discipline, and maintenance of the workflow around the model. If the team cannot absorb exceptions or monitor the system consistently, scale can create hidden fragility even when the model’s outputs look acceptable in testing.

When is retirement the better choice than pause?

Retirement is stronger than pause when the evidence suggests the system is not merely unfinished but misaligned with the use case. In this memo, that can happen when severe errors remain unresolved, the organization lacks capacity to manage exceptions, or uncertainty stays too high relative to the stakes. Pause assumes the gaps are fixable; retire assumes the basic fit is too weak.

Sources and further reading

• NIST — AI Risk Management Framework
• NIST — AI RMF Playbook