Compare AI With a Non-AI Baseline: George Dagliyan Guide

What is the clearest way to compare a generative ticket summarizer with the current method?

The clearest comparison is a paired test: take the same ticket set, run each ticket through both the generative summarizer and the current search-and-template process, then review outputs against the same quality criteria. This article’s editorial analysis, informed by George Dagliyan’s research themes on AI adoption and trust, focuses on controlled comparison rather than enthusiasm for the tool.

The point is not to prove that the AI version is “better” in the abstract. It is to test whether it performs well enough, with fewer exceptions or less effort, for the organization’s actual ticketing context. NIST’s AI Risk Management Framework frames this kind of work as part of voluntary AI risk management, and NIST’s Playbook organizes related work under Govern, Map, Measure, and Manage.

What should be matched so the test is fair?

Match workload first. Use the same ticket types, the same time window or sample period, the same source fields, and the same expected output format for both methods. If one method sees short password resets and the other sees long incident narratives, the comparison becomes misleading. Keep language, channel, and ticket complexity as similar as practical.

A useful worksheet for matching is simple: ticket category, length band, urgency level, required systems, and whether any sensitive details must be redacted. Pair one AI output and one baseline output to each ticket, and keep the reviewer blind to which method created which summary. Blind review reduces bias from brand effects, novelty effects, and expectations about AI quality.

How should the paired test be structured step by step?

Start with a small, clearly bounded sample of tickets that reflect real work, not only the easiest examples. Split the process into intake, generation, review, and exception handling. For each ticket, create two summaries: one from the current search-and-template method and one from the generative summarizer. Label them only as Version A and Version B before review.

Next, define one review form for both versions. Reviewers should score each summary on the same criteria, such as factual completeness, clarity, actionability, and policy fit. They should also note whether the summary omits a critical field, adds unsupported content, or creates confusion. Keep revision rules identical: if one version is allowed one human edit and the other is not, the comparison is no longer clean.

Which exceptions matter enough to treat separately?

Exceptions should be designed before the test begins. Typical exceptions include tickets with missing source data, conflicting notes, highly sensitive information, nonstandard language, or cases where a summary could cause operational harm if wrong. These should not be hidden inside the main score because they represent different risk conditions, not ordinary variation.

A practical exception log should record the ticket ID, the exception type, whether the ticket was excluded from the paired comparison, and whether it required escalation. This connects to NIST’s emphasis on documentation, monitoring, and risk management in the AI RMF and Playbook. The goal is not to punish the AI for edge cases, but to see where human review remains essential.

What quality criteria should reviewers use?

Use criteria that are observable and tied to the ticketing task. Four useful criteria are: correctness of facts, completeness of essential fields, usefulness for next action, and readability for the intended worker. A fifth criterion can cover compliance with internal wording or privacy rules if the organization has them. Keep the scale simple, such as pass, partial, or fail, so reviewers can apply it consistently.

To avoid vague judgments, define each criterion in plain language. For example, “complete” can mean the summary captures the issue, affected system, user impact, and required next step when those items exist in the source ticket. “Useful” can mean another team member could act without rereading the full ticket immediately. These definitions turn a subjective preference into a reviewable standard.

How should the results be interpreted without inventing statistics?

Interpretation should be decision-oriented, not numerically inflated. If the generative summarizer matches or outperforms the baseline on most core criteria, and exceptions remain manageable, it may be suitable for limited use with human oversight. If it performs well on speed but poorly on factual accuracy or omission risk, the baseline may remain the safer default.

A simple decision rule can be used: proceed only if the AI version meets all non-negotiable criteria for accuracy and privacy, and does not increase exception severity. That rule is an internal decision aid, not a NIST threshold. NIST’s framework is voluntary and supports organizations in tailoring risk management to their own priorities and tolerance, so the pass line should reflect local operational risk.

What does a worked hypothetical example look like?

Hypothetical scenario: a support desk wants to compare 30 tickets from one service queue. Assume the queue contains routine access issues, a few application failures, and some tickets with incomplete notes. The current method uses search plus a fixed template, while the AI version generates a draft summary from the same source text. Two reviewers, blind to method, score each version using the same form.

In this hypothetical example, one ticket lacks a key system name, so both versions are flagged. The exception is logged rather than forced into the main score. Another ticket contains a long chain of emails; reviewers judge whether each method preserves the final action item without adding speculation. The exercise ends with a side-by-side review of strengths, weak spots, and the kinds of tickets that should stay on the baseline.

How does this test connect to broader AI adoption decisions?

This paired test does more than compare output quality. It reveals whether the new tool fits existing work habits, review capacity, and tolerance for ambiguity. That is where George Dagliyan’s research themes on AI adoption, innovation diffusion, and technology acceptance become relevant in editorial analysis: adoption depends not only on model capability, but also on the organizational conditions that support or inhibit use.

The most practical outcome is a controlled recommendation for role fit. An organization may decide the generative summarizer is appropriate for first drafts, but not for final customer-facing text. It may decide the current search-and-template process remains the default for exceptions or high-risk queues. Either way, the comparison creates an evidence-based path for measured adoption instead of a broad launch.

Frequently Asked Questions

Why compare the AI summarizer against the current search-and-template method instead of testing it alone?

A standalone test shows whether the AI output looks acceptable, but it does not show whether it is better than what staff already do. A paired comparison measures change against the real baseline, which matters when the existing process already works reliably for some ticket types. It also helps avoid adopting a tool that is faster but less accurate in practice.

What makes a blind review important in this kind of test?

Blind review keeps reviewers from favoring the AI version because it is new or expecting the baseline to be safer because it is familiar. When reviewers see only Version A and Version B, they focus on the text itself. That improves the fairness of the comparison and makes quality judgments more defensible inside the organization.

Should all tickets be included in the paired test?

Not necessarily. Tickets with severe data gaps, unusual legal sensitivity, or other special conditions may need separate treatment. The important step is to define those exceptions before the test begins and log them consistently. Excluding a ticket from the main comparison should be a documented decision, not an informal judgment made after the fact.

What if the AI summarizer is better at speed but worse at some quality criteria?

Speed gains alone are not enough if the tool increases the chance of factual errors, missing action items, or privacy problems. In that case, the result may still support limited use, such as drafting only, but not full replacement of the baseline. The right decision depends on which criteria are non-negotiable for the task and which can be reviewed or corrected by humans.

How should a team decide whether to expand use after the test?

Expansion should follow the organization’s own risk tolerance and review capacity. A sensible next step is limited rollout in the ticket categories where the AI matched the baseline most closely and produced few exceptions. Keep monitoring, retain a fallback process, and revisit the decision if the ticket mix, source quality, or staffing model changes.

Sources and further reading

• NIST — AI Risk Management Framework
• NIST — AI RMF Playbook