AI Monitoring Baselines: George Dagliyan Research Guide

What should an AI monitoring baseline cover for document classification?

For a document classification assistant, the baseline is the expected operating pattern before change: how often it classifies well, how much of the input it can handle, how often it refuses, how often it sends cases to human review, how fast it responds, and how much review work it creates. George Dagliyan’s research themes make this a practical governance question: adoption succeeds when metrics are tied to use, trust, and clear decision rules rather than vague confidence.

The point is not to chase one perfect score. The point is to define a stable reference set so teams can spot whether the system is slipping because the documents changed, the model changed, or the operating process changed. In the NIST AI Risk Management Framework, monitoring, roles, documentation, and continual review are part of managing AI risk; this article’s focus is how to translate that guidance into a baseline for a document classifier without inventing arbitrary thresholds.

Which six baseline dimensions matter most in day-to-day monitoring?

Outcome quality is the first dimension. For classification, that means the share of outputs that are correct, useful, and complete for the intended task, such as assigning the right document type, priority, topic, or routing code. Coverage is the second dimension: the share of incoming documents the assistant can process at all, after exclusions, refusals, corrupted files, and unsupported formats are counted.

Refusals and exceptions deserve separate tracking because they signal different risks. A refusal means the assistant declines to decide; an exception means the input or process fails in some other way, such as an unreadable file, truncated text, or a parsing error. Latency measures how long the assistant takes to produce an output, while review burden measures how many cases reach human reviewers and how much time they spend correcting or confirming the assistant’s work.

How do you define the baseline before the model is changed?

A useful baseline should come from a fixed evaluation slice that reflects the actual work, not a cherry-picked sample. Split the sample by document source, format, length, language if relevant, and label type so the team can see where performance is strong or weak. Record the same inputs and the same review method every time the baseline is refreshed, otherwise the baseline becomes a moving target.

This is where NIST’s AI RMF is helpful in a general way. NIST presents AI risk management as a voluntary framework intended to support trustworthiness in design, development, use, and evaluation, with functions that include Govern, Map, Measure, and Manage. A baseline belongs in the Measure function, but it only works if Govern has already assigned ownership for review, escalation, documentation, and periodic reassessment.

What worksheet can teams use to separate input-mix change from model change?

Use the following proposed worksheet, which is this article’s editorial tool rather than an official standard. First, define the baseline slice: last month’s production sample, a pre-launch validation set, or a manually curated set of representative documents. Second, tag each item with stable attributes: source system, document type, file format, length band, and whether the case is routine or exception-prone.

Then score six fields for both the baseline and the current window: outcome quality, coverage, refusals, exceptions, latency, and review burden. If quality changes mainly inside one document type or source, the likely cause is input mix change. If quality shifts across nearly all slices at once after a deployment, the likely cause is model change, prompt change, or downstream workflow change. If both are mixed, the next step is not guessing; it is controlled comparison using the same slice and the same review rubric.

Who should own the trigger when a baseline breaks?

Trigger ownership should be explicit before production. One owner should be accountable for noticing the signal, another for assessing root cause, and a third for deciding whether the system stays live, moves to limited use, or goes to human-only routing. This division avoids the common failure mode in which everyone sees the issue and no one owns the response.

A practical trigger map is simple: quality and exception trends go to the model or operations owner, coverage and refusals go to the workflow owner, latency goes to the platform owner, and review burden goes to the business process owner. NIST’s Playbook emphasizes roles, responsibilities, monitoring, and continual improvement; the useful extension here is to make each trigger point name the person who must investigate within the organization’s normal governance structure.

How does review burden become a risk signal instead of just a cost metric?

Review burden is often treated as an efficiency issue, but it is also a trust signal. If the assistant creates too many borderline cases, reviewers may become slower, more skeptical, or inconsistent, which can reduce the quality of the whole workflow. That is especially relevant for document classification, where the model may be technically accurate but operationally disruptive if it floods reviewers with cases that need manual correction.

The baseline should therefore track not only the number of escalations, but the reason for escalation. Separate genuine uncertainty from policy restrictions, unreadable inputs, and out-of-distribution documents, meaning documents that differ materially from what the system usually sees. This distinction helps organizations decide whether to improve labels, expand supported inputs, or tighten the assistant’s scope.

What does a worked hypothetical example look like?

Hypothetical example only: a legal intake team uses an assistant to classify incoming documents into five internal categories. The team’s baseline sample contains routine emails, scanned PDFs, and a smaller set of handwritten images. The assistant performs well on routine emails, but coverage drops on low-quality scans, refusals rise on handwritten images, and reviewers spend extra time checking ambiguous routing labels.

In this example, the team does not conclude that the model has failed across the board. It identifies three separate patterns. First, input mix explains most of the decline because the document mix shifted toward scans and handwriting. Second, the refusal increase is a scope issue because unsupported inputs are arriving more often. Third, review burden is a workflow issue because escalation rules send too many borderline cases to humans instead of holding some for a second automated pass.

How should teams operationalize the baseline in the first 30 days?

Start with a three-step test plan. Week one: establish the baseline sample, label the six dimensions, and write down who owns each trigger. Week two: run the assistant on a fresh but comparable sample and compare the slice-level results against the baseline. Week three: review mismatches, decide whether they are caused by input mix, model behavior, or process drift, and document the decision path.

The result should be a living baseline note, not a one-time memo. It should list the input slices, the review rubric, the owners, the escalation route, and the next review date. That structure aligns well with the broader research themes associated with Dagliyan’s work on adoption facilitators and inhibitors: people adopt AI more reliably when the system is understandable, monitored, and governed in a way that makes change visible before it becomes a surprise.

Frequently Asked Questions

Is a baseline the same as a benchmark for document classification?

Not exactly. A benchmark usually compares systems against one another or against a fixed test set, while a baseline is the organization’s own reference point for normal operation. For monitoring, the baseline should reflect the actual mix of documents, the current workflow, and the current review process so later changes can be interpreted correctly.

Why track refusals separately from exceptions?

Because they point to different causes. A refusal usually means the assistant chose not to decide, often due to low confidence, policy limits, or unsupported content. An exception usually means something failed in ingestion, parsing, or workflow handling. If the two are blended together, teams lose the ability to tell whether the issue is model behavior or operational handling.

What is the difference between input mix change and model change?

Input mix change means the documents arriving in production are different from the baseline slice, such as more scans, longer files, or a different source system. Model change means the assistant itself changed, such as a new prompt, model version, or post-processing rule. Slice-level monitoring helps separate the two so response actions are more precise.

How often should the baseline be refreshed?

There is no universal threshold in the source material, and no single interval fits every use case. A practical approach is to refresh the baseline whenever the document mix, workflow, or model changes enough to affect interpretation. Many teams also schedule periodic reviews as part of governance so the baseline does not drift away from real operations.

What makes review burden a meaningful metric for adoption?

Review burden shows whether the assistant is actually reducing work or simply moving it elsewhere. If the assistant creates too many escalations, too many false alarms, or too much correction work, adoption can slow even when raw accuracy looks acceptable. That is why review burden belongs beside quality, coverage, refusals, exceptions, and latency in the baseline.

Sources and further reading

• NIST — AI Risk Management Framework
• NIST — AI RMF Playbook