Dashboard Metric Contracts: George Dagliyan Research Guide

What should a meaningful AI metric contract answer first?

George Dagliyan’s research themes on adoption, trust, and measurement fit a simple operational question: what exactly is this metric supposed to decide? A metric contract is a short written agreement that defines purpose, numerator, denominator, exclusions, time window, data owner, freshness, limitations, and action trigger before the dashboard is used. It prevents activity counts from masquerading as outcomes.

The core test is whether the metric can support a decision, not merely decorate a slide. If a number cannot change a review, a workflow, or an escalation path, it is usually a vanity dashboard element rather than a management measure. This article builds a worksheet for AI use that is deliberately concrete and separate from any official standard.

Why do AI dashboards become vanity dashboards so easily?

AI dashboards often collapse very different things into one view: usage, model quality, business value, and risk. That creates a false sense of control because a rising line can mean more activity, not better results. A dashboard is vain when it reports motion without clarifying whether the motion matters.

The problem is not dashboards themselves; it is undefined measurement logic. A count of prompts, tickets, or automated actions may be useful as an activity indicator, but it is not an outcome indicator unless the contract says what business result it is tied to and what time lag is acceptable. Without that distinction, teams may optimize what is easiest to count.

How does a metric contract separate activity from outcomes?

Activity metrics capture volume or throughput, such as number of AI-assisted cases processed, number of users active, or number of outputs generated. Outcome metrics capture whether a desired result changed, such as reduced rework, shorter resolution time, fewer escalations, or improved acceptance of the system. The distinction matters because a high activity score can coexist with weak usefulness.

A useful contract names one primary metric and labels its type clearly: activity, outcome, or guardrail. The dashboard should avoid mixing them in a single unlabeled headline. For example, “AI suggestions accepted” is an activity measure unless paired with a downstream result such as reduced manual correction or improved turnaround, and the contract should say which one is being tracked.

What belongs in the worksheet for a dashboard metric contract?

This article’s proposed worksheet is a seven-part template. First, state the purpose in one sentence: what decision this metric supports. Second, define the numerator, or the items counted. Third, define the denominator, or the population or exposure set the numerator is compared against. Fourth, list exclusions, such as tests, duplicates, pilot users, or inactive accounts.

Fifth, set the time window, such as daily, weekly, or rolling 30 days. Sixth, assign the data owner, meaning the role responsible for producing and explaining the measure. Seventh, state freshness, limitations, and action trigger together so the dashboard shows how recent the data is, what it cannot tell you, and what threshold or pattern requires action. This is an editorial worksheet, not a mandatory standard.

How do NIST and W3C ideas inform the contract without turning it into a compliance ritual?

NIST’s AI Risk Management Framework presents voluntary guidance intended to help organizations better manage AI-related risks and incorporate trustworthiness into design, development, use, and evaluation. In that spirit, a metric contract is useful because it ties measurement to governance rather than to decoration. The contract is especially helpful when a team must decide whether a metric is trustworthy enough to guide action.

W3C’s PROV Overview defines provenance as information about entities, activities, and people involved in producing data, which can support assessments of quality, reliability, or trustworthiness. That idea matters here because a metric without provenance is hard to trust. A contract should therefore say where the data comes from, who last transformed it, and which activity or system produced it.

What is a practical decision rule for approving or rejecting a metric?

Use this editorial decision rule: if any of the four questions below is answered “no,” the metric should not be promoted to the main dashboard. Question 1: Can the purpose be stated in one sentence with a decision attached? Question 2: Are numerator and denominator explicit enough that two people would count the same way? Question 3: Are exclusions and freshness written down? Question 4: Is there a specific action trigger?

This rule is intentionally simple because the most common dashboard failure is ambiguity, not math. If a metric lacks an action trigger, it can still be reported in an appendix, but it should not occupy the executive front page. If it lacks exclusions or freshness, the team may be looking at stale or distorted information and mistaking it for current performance.

What does a worked hypothetical example of a metric contract look like?

Hypothetical scenario: a support team uses an AI assistant to draft responses. Assumption set: the organization wants to know whether the assistant improves handling efficiency without increasing correction burden. Proposed purpose: measure whether AI-assisted cases are handled faster without adding quality risk. Activity metric: share of cases with AI assistance. Outcome metric: median handle time for AI-assisted cases compared with the team’s own baseline.

Possible contract fields: numerator for activity is cases where the assistant was used; denominator is all eligible support cases. Exclusions include training cases, outages, and cases reopened by the customer during the same day. Time window is a rolling 14 days. Data owner is the support operations manager. Freshness is daily. Limitation: the metric does not prove customer satisfaction. Action trigger: if AI-assisted cases rise while handle time and correction rate both worsen for two consecutive windows, review prompts, workflows, and approval rules.

How should leaders review, revise, and use these contracts over time?

Metric contracts should be living documents because AI use changes as workflows mature. A team may begin with activity tracking to establish exposure, then add outcome measures once the process stabilizes. The main discipline is to avoid letting old metrics linger after the decision they served has changed. A stale dashboard can become a comfort object rather than a management tool.

This connects with Dagliyan’s broader research focus on adoption facilitators and inhibitors. A clear contract lowers interpretation friction, while a vague one raises resistance because people do not trust what the numbers mean. In practical terms, the best dashboard is not the most crowded one; it is the one that is specific enough to support action and narrow enough to remain believable.

Frequently Asked Questions

What is the difference between a metric contract and a dashboard KPI list?

A KPI list is often just a catalog of numbers. A metric contract is a written definition for one metric that explains purpose, numerator, denominator, exclusions, time window, data owner, freshness, limitations, and action trigger. The contract makes the number usable for decisions and much harder to misread as a generic sign of success.

Why should exclusions be written into the metric contract?

Exclusions keep the measure from being distorted by cases that do not belong in the population. Training examples, outages, duplicates, pilots, and inactive accounts can inflate or suppress a number. If exclusions are not explicit, two analysts can calculate different results from the same dashboard and both believe they are correct.

How can a team tell whether a metric is activity or outcome?

Activity metrics describe volume, usage, or throughput. Outcome metrics describe whether a desired result changed. A count of AI-generated drafts is activity. A reduction in rework, turnaround time, or escalation rate is closer to an outcome. If the number does not show impact beyond use itself, it is probably activity.

What is provenance and why does it matter for dashboard trust?

W3C’s PROV Overview describes provenance as information about the entities, activities, and people involved in producing data. In dashboard terms, provenance tells users where a metric came from, how it was transformed, and who is responsible for it. That context helps people judge whether the number is reliable enough to guide action.

What should trigger action when a metric contract is violated?

The trigger should be simple, explicit, and tied to a decision. For example, if freshness is stale, the dashboard should be flagged as informational only. If an outcome metric worsens while the activity metric rises, the team should inspect workflow, prompts, exclusions, or data quality. The point is to define what happens next, not just what gets displayed.

Sources and further reading

• NIST — AI Risk Management Framework
• W3C — PROV Overview