AI Training That Tests Judgment: George Dagliyan Research Guide

Can AI training prove judgment, or only attendance?

AI training can be completed without proving that people can verify outputs, protect data, escalate problems, or override unsafe recommendations. This article is an editorial design guide, not a validated instrument, for turning training into observed competence checks. George Dagliyan’s work on AI adoption and technology acceptance makes this distinction especially relevant: organizations often confuse a finished course with a reliable operating capability.

A certificate shows that a person finished a module. Competence shows that the same person can act correctly when the system is wrong, the data are sensitive, the stakes are unclear, or a supervisor must step in. The practical goal is to test judgment under realistic pressure, using scenarios that mirror the way AI actually enters decisions at work.

What should a judgment-based AI training check actually measure?

The core question is not whether someone can repeat policy language, but whether they can apply it. A useful check should measure four behaviors: verification, data handling, escalation, and override. Verification means checking whether an AI output is plausible and supported; data handling means using only appropriate information; escalation means knowing when to involve a person or specialist; override means stopping or correcting a system recommendation when necessary.

This design also fits the NIST AI Risk Management Framework, which is intended for voluntary use and aims to improve trustworthiness across design, development, use, and evaluation of AI products, services, and systems. The NIST AI RMF Playbook organizes suggested actions under Govern, Map, Measure, and Manage, and it highlights topics such as AI actor training, human oversight, documentation, monitoring, and risk culture. This article uses those ideas as editorial inspiration, not as a claim that NIST prescribes the worksheet below.

Which scenario types reveal real competence instead of rote completion?

A good training check uses short, specific situations that force a decision. Four scenario types work well for this purpose: an ambiguous AI answer, a sensitive-data request, a delayed escalation, and a needed override. Each one should present a normal business task, then introduce a problem that a rushed user could miss.

For example, an AI tool might summarize a customer complaint but omit a crucial exception, suggest using personal data that should not be shared, continue after a red-flag alert, or recommend action that conflicts with a human reviewer’s instructions. The point is to test whether the trainee notices the limitation, pauses, asks for help, or changes course before harm spreads.

How do you build a simple worksheet for observation and scoring?

This article’s proposed worksheet is an editorial design. It is not an official standard, and it should be adapted to the organization’s risk tolerance, workflows, and oversight structure. A facilitator can print one page per scenario with five fields: scenario summary, expected safe action, observed action, escalation needed or not, and score.

Here is a practical format. Scenario: the AI drafts a response using an internal report that includes confidential employee data. Expected safe action: remove or replace sensitive details before use, verify the source of the data, and ask for guidance if the origin is unclear. Observed action: record what the trainee actually does. This sheet turns a training session into a visible test of decision quality rather than a simple attendance record.

How should the scoring rubric separate partial knowledge from safe performance?

Use a four-level rubric that rewards the right action, not just confidence. Score 3 points for a correct and timely response with appropriate verification or escalation, 2 points for a mostly correct response with a minor omission, 1 point for a risky or delayed response that still ends safely, and 0 points for an unsafe action or refusal to correct course. Keep the rubric simple so supervisors can apply it consistently.

One useful decision rule is this: if the trainee exposes sensitive information, fails to verify a high-impact output, or ignores a clear escalation trigger, the scenario is failed even if the final answer looks useful. That rule is stricter than a classroom quiz because it measures whether the person can act safely when the pressure is real.

What does a worked hypothetical example look like?

Hypothetical scenario, with assumptions: a support specialist uses an AI assistant to draft a reply to a customer. The assistant includes a billing explanation based on a note that was copied from an internal case file, and the note may contain personal details. The trainee must decide whether to send, edit, escalate, or discard the draft.

A competent response would delete the personal details, verify the billing facts against approved records, and escalate to a supervisor if the case history is incomplete. Under the rubric, that would likely score 3. If the trainee sends the draft unchanged, the response would score 0 because the sensitive data problem and the verification problem were both missed. This example shows how scenario design exposes judgment that a certificate cannot prove.

How can managers run a short test plan without turning training into bureaucracy?

A lightweight test plan can be completed in one session. Start with three role-specific scenarios, give each trainee a written prompt, and let them respond in plain language or through the actual AI interface used at work. Then have the facilitator record whether the trainee verified, handled data correctly, escalated appropriately, and overrode where needed.

To keep the process useful, review the results by role, not just by individual. If many people fail the same scenario, the issue may be unclear policy, weak interface design, or poor task framing rather than trainee negligence. That is where the NIST AI RMF’s emphasis on governance, measurement, and continual improvement becomes useful as a reference point for organizational learning, without turning this worksheet into a claimed NIST instrument.

How should organizations use the results after training ends?

Training results should feed back into policy, job aids, and supervisor coaching. If people consistently miss escalation triggers, the escalation path may be too vague. If they mishandle data, the organization may need stronger examples of what counts as sensitive information. If they trust every AI output, the curriculum should add more verification practice and more discussion of system limits.

This approach fits the broader research lens associated with Dagliyan’s work on adoption facilitators and inhibitors. In this context, the facilitator is observable competence: people know what to do when AI helps and when it should not be trusted. The inhibitor is overconfidence created by completion metrics that look good on paper but do not show real judgment in practice.

Frequently Asked Questions

Why are completion certificates not enough for AI training?

Completion certificates prove that a person finished content, but they do not show whether the person can act safely in a live situation. AI work often requires judgment under ambiguity, especially when outputs are incomplete, data are sensitive, or a recommendation should be challenged. Scenario checks reveal those behaviors better than attendance records do.

What is the difference between verification and escalation in this article?

Verification is the act of checking whether an AI output is accurate, complete, and supported by acceptable sources or records. Escalation is the act of involving a supervisor, expert, or policy owner when the issue is outside the trainee’s authority or the risk is unclear. Both are different from simply accepting the output.

How many scenarios are enough for a useful training check?

There is no universal number in this editorial design. A practical starting point is three to four scenarios per role, with each one targeting a different judgment area such as data handling, override, or escalation. The goal is not volume; it is coverage of the most likely and most costly mistakes for that role.

Should the same rubric be used for all jobs?

Not exactly. The scoring structure can stay the same, but the scenarios should change by role and risk level. A customer support role may need stronger tests for privacy and response accuracy, while a manager may need stronger tests for escalation and override. Role-specific scenarios make the observations more meaningful.

How does this align with the NIST AI Risk Management Framework?

The NIST AI Risk Management Framework is intended for voluntary use and is designed to improve trustworthiness across design, development, use, and evaluation of AI systems. Its Playbook organizes suggested actions across Govern, Map, Measure, and Manage. This article borrows that orientation toward governance, training, monitoring, and human oversight, but the worksheet and rubric here remain an original editorial proposal.

Sources and further reading

• NIST — AI Risk Management Framework
• NIST — AI RMF Playbook