Classifying AI Errors: A George Dagliyan Research Guide

What counts as a service-ticket summary error in AI output?

George Dagliyan’s research lens is useful here because service-ticket summaries are not just text; they are adoption signals, trust signals, and operational inputs. This article defines a practical error taxonomy for AI-generated ticket summaries: omission, unsupported statement, wrong entity, outdated rule, privacy exposure, and unauthorized action. The purpose is not to replace policy, but to make correction faster and ownership clearer.

A service-ticket summary error is any statement or omission that changes how a human reader understands the issue, the next step, or the risk attached to the ticket. NIST’s AI Risk Management Framework says the framework is intended for voluntary use and to support trustworthiness across AI design, development, use, and evaluation. That makes this kind of error review a fitting governance practice, especially for generative AI summaries.

How should the six error types be classified?

The first category is omission: the summary leaves out a material fact, such as the customer impact, the failure point, or the key troubleshooting step. Omission matters because a concise summary can still be incomplete enough to delay action. The second is unsupported statement: the summary states a fact that is not present in the source ticket or attached evidence.

The third category is wrong entity, where the summary assigns the issue to the wrong customer, system, account, product, location, or team. The fourth is outdated rule, where the summary relies on a policy, workflow, version, or routing convention that is no longer current. The fifth is privacy exposure, which occurs when the summary includes sensitive personal, account, or internal information that does not belong in the summary view.

The sixth is unauthorized action, where the summary implies or instructs a step the AI should not take on its own, such as changing access, confirming a refund, closing a case, or making a routing decision without the required human approval. In this taxonomy, the error type describes what went wrong, while the correction path describes who must fix it and by when.

What is the difference between severity and frequency?

Severity is the business impact of a single error when it appears. Frequency is how often that same error type appears over time. A privacy exposure can be severe even when rare, while an omission may be less severe on any one ticket but operationally important if it happens repeatedly. The two measures should be tracked separately because a frequent low-severity error can create the same workload as a rare high-severity error.

A practical decision rule is this: classify severity by the worst plausible consequence of that specific summary error, and classify frequency by counting occurrences across a review period. If severity is high, the ticket should be escalated even when frequency is low. If frequency is high, the process owner should investigate patterns even when each individual error seems minor.

This distinction matches the spirit of NIST’s AI RMF and the Generative Artificial Intelligence Profile, both of which frame AI risk management as ongoing and context-sensitive rather than one-time. The useful question is not only, “Is this wrong?” but also, “How harmful is one instance, and how often does the system repeat it?”

Who owns the correction at each step?

Ownership should follow the source of the error, not just the person who noticed it. If the AI omitted a fact, the correction often belongs to the reviewer or workflow owner who approves summaries before they are sent. If the AI made an unsupported statement, the correction belongs to the reviewer who must either delete it or verify it with evidence before release.

If the wrong entity was named, the correction may require both the ticket author and the routing owner, because the underlying record may need to be corrected and the summary regenerated. If the error reflects an outdated rule, the process owner or knowledge base steward should update the rule set, because repeated failures will continue until the source guidance changes.

Privacy exposure should be corrected immediately by the reviewer or privacy-aware supervisor, because the priority is to remove the exposure from the summary surface. Unauthorized action belongs to the control owner or supervisor who can stop the action and reset permissions or prompts if needed. The editorial analysis here is that correction ownership should be explicit in the workflow, not assumed after the fact.

What worksheet can teams use to review summaries consistently?

Use this proposed worksheet, which is an editorial tool and not an official NIST form. For each AI-generated summary, record five fields: source ticket ID, error type, severity level, correction owner, and correction action. Then add one short note on whether the error is isolated or part of a repeated pattern. This keeps review focused on both the text defect and the process defect.

Suggested severity levels are low, medium, high, and critical. Low means the error is inconvenient but not likely to mislead action. Medium means the summary could cause delay or rework. High means the error could create an incorrect decision, an unnecessary escalation, or a security or privacy concern. Critical means the error can directly trigger a harmful action or serious exposure.

A simple review question set can support the worksheet: What was omitted? What is asserted without support? Which entity is named, and is it the right one? Is any rule or procedure outdated? Does the summary expose data it should not? Does it imply an action the system cannot take without approval?

What does a worked hypothetical example look like?

Hypothetical scenario: a service desk ticket reports that a customer cannot access an account after a password reset, and the AI summary says, “User confirmed all devices are locked and account closure is requested.” Assumptions: the original ticket only mentions login failure; it does not mention account closure, device status, or a confirmed user statement. In this case, the summary contains an unsupported statement and a wrong entity risk if the account belongs to a different user.

Severity could be medium if the error merely misleads the agent, or high if it leads to account closure or an inappropriate access action. Frequency should be measured separately; if the same summary pattern appears in many password-reset tickets, the team would treat it as a systemic issue. The correction owner would usually be the reviewer or service desk lead, while the process owner would review the template or prompt that encouraged the false closure language.

Another hypothetical: a generated summary includes a customer’s personal phone number in full when the agent only needed the last four digits. That is privacy exposure. The right correction is immediate redaction and a process check on whether the summary template permits unnecessary detail. If the model also suggested resetting access rights automatically, that adds unauthorized action and raises the severity.

How can teams log errors without turning review into paperwork?

An illustrative error log can be short and still useful. Example log entry 1: Ticket 1842, omission, medium severity, owner: reviewer, action: add missing outage duration. Example log entry 2: Ticket 1843, unsupported statement, high severity, owner: service desk lead, action: remove claim that customer approved closure. Example log entry 3: Ticket 1844, privacy exposure, critical severity, owner: privacy supervisor, action: redact phone number and retrain summary template.

Example log entry 4: Ticket 1845, outdated rule, medium severity, owner: knowledge base steward, action: update routing rule reference. Example log entry 5: Ticket 1846, wrong entity, high severity, owner: routing owner, action: correct account and regenerate summary. Example log entry 6: Ticket 1847, unauthorized action, critical severity, owner: supervisor, action: block auto-closure suggestion and review permissions.

The point of the log is pattern recognition, not blame. Over time, frequency counts by category show whether the issue is a prompt problem, a knowledge base problem, a human review problem, or a workflow design problem. That is where George Dagliyan’s research themes on adoption facilitators and adoption inhibitors are relevant: correction speed, clarity of ownership, and trust in the summary all influence whether people keep using the system.

What should leaders do next?

Start by deciding which summary fields are allowed to be generated automatically and which must always be verified by a human. Then define the severity scale, assign a correction owner for each error type, and require the short error log for a sample of tickets each week. This is a practical governance layer, not a replacement for broader AI risk management.

NIST’s AI RMF and its Generative Artificial Intelligence Profile support the broader idea that AI risk should be managed across the lifecycle of the system. For service-ticket summaries, the lifecycle view translates into a simple operational rule: measure the defects, correct the source, and keep the human responsible party visible. That makes the system easier to trust without pretending it is error-free.

Frequently Asked Questions

Why separate omission from unsupported statement if both make a summary wrong?

They are different operational failures. Omission means the summary leaves out a material fact that should have been included, such as outage duration or next steps. Unsupported statement means the summary adds a fact not found in the ticket or evidence. Separating them helps teams fix the right source: coverage problems for omission and hallucination-like assertions for unsupported statements.

When should an AI summary error be treated as critical rather than merely high severity?

Critical is appropriate when the error can directly trigger a harmful action, serious privacy exposure, or an unauthorized change without another review step catching it. High severity is serious but may still be stoppable before damage occurs. The practical difference is whether the mistake is likely to be intercepted before it causes an irreversible or hard-to-reverse outcome.

Who should own correction if the AI repeated an outdated rule from the knowledge base?

Ownership should usually split between the reviewer and the knowledge base steward. The reviewer corrects the immediate summary so the ticket is safe now. The steward updates the outdated rule so the same mistake does not recur. If the prompt or template also encouraged the wrong rule, the process owner should join the correction chain.

How does frequency change what teams prioritize even when severity is low?

Frequency shows whether a small issue is becoming a system problem. A low-severity omission may seem harmless once, but if it happens on many tickets it can create repeated delays, rework, and user frustration. That pattern often indicates a prompt, template, or training gap, which should be fixed even if no single incident looks alarming.

Is the proposed worksheet a required standard from NIST?

No. The worksheet in this article is an editorial tool designed for service-ticket summary review. NIST’s AI Risk Management Framework and its Generative Artificial Intelligence Profile are voluntary resources that support trustworthiness and risk management, but they do not prescribe this exact form, these exact fields, or these severity thresholds.

Sources and further reading

• NIST — AI Risk Management Framework
• NIST — Generative Artificial Intelligence Profile