Technology-Driven Performance Metrics

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

AI field note: my word of the year is 𝔼𝕍𝔸𝕃: celebrating the art and science of rigorous measurement of AI performance, progress and purpose. (1 of 3) This year delivered a wealth of new AI models, architectures, and use cases - all united by one thread: evaluation. Model benchmarking, evaluation, or just "eval" has evolved from a simple, singular measure to a more complex blend of stats, metrics, and measurement techniques. Today's evals help discerning practitioners make pragmatic, informed technology decisions and measures improvements as AI systems are tuned. With AI innovation accelerating, staying up to date on evals ensures informed trade-offs when building intelligent systems, agents, and applications. Let's start by looking at measuring "performance"; the best way we know how to compare model behaviors, and find the right fit-for-purpose. Defining 'good performance' now involves a sophisticated suite of metrics across diverse dimensions. ⚙️ Task eval - beyond raw performance numbers. Today's evals measure how models perform across diverse scenarios - from basic comprehension to complex reasoning, reliability, consistency, and nuanced evaluation of reasoning paths, output quality, and edge case handling. 👛 Token economics - balancing cost, efficiency, and operation. Understanding token costs - both input and output - was essential last year, but evals have evolved beyond raw price per token, to understanding efficiency patterns, batching strategies, and the total cost of operation. ⏲️ Time-to-first-token. Speed is a feature, as they say, and while streaming responses have improved user experiences, this metric has become particularly crucial as models are deployed in production environments where user experience directly impacts adoption. 🔥 Inference compute: The amount of compute used for prediction shapes what problems a model can solve. More compute enables greater complexity but increases costs and latency - making it a pivotal benchmark for 2024. For some light holiday reading to explore this further: Service cards (OpenAI, Amazon), Meta's Llama 3 paper, and Anthropic's evaluation sampling research (links below).

From Candlelight to Smart Grids: Why AI-Ready #ITSM Leaves Traditional Service Management in the Dark Imagine running #IT like a world lit by candlelight. You react only when something flickers or burns out. You wait. Then you scramble. That’s traditional ITSM. Reactive, manual, and effort-heavy. Now imagine a smart power grid: ⚡Lights adjust before they dim. ⚡Energy reroutes to prevent outages. ⚡Issues are predicted, prevented, resolved before anyone notices. That’s AI-powered Service Management. It’s not just about responding to issues. It’s about predicting, preventing, and empowering work to flow at the speed of business. If your org still measures success by SLA compliance or ticket closures, you’re optimizing candlelight in a world powered by neural grids. As you strategized in an AI-ready ITSM organization, these are the metrics tracked, not just because we can, but because they drive speed, automation, and business value: AI-Ready, Automation-Driven ITSM Metrics: 1. First Predictive Alert Time (FPAT): How early AI detects and alerts potential issues before users report them. 2. Mean Time to Auto-Resolution (MTTAR): Average time incidents are resolved via AI/automation. 3. AI Recommendation Utilization Rate: How often agents follow AI-suggested resolution paths. 4. Digital Agent Containment Rate: % of requests handled end-to-end by virtual agents. 5. Proactive Deflection Rate: Tickets avoided due to proactive alerts/self-healing. 6. Knowledge Intelligence Score: How well AI matches KB articles to intent + outcome. 7. Sentiment-to-Resolution Correlation: The impact of customer sentiment (captured by AI) on resolution speed and satisfaction. 8. AI Learning Velocity: Rate at which the AI models improve based on feedback loops from incident outcomes. 9. Employee Downtime Avoidance Rate: Work hours saved by preemptive fixes. 10. Business Service Resilience Index: Stability of services under AI-assisted ops. 11. Automation Potential Realization (APR): % of manual tasks converted into automation. 12. Innovation Throughput: Capacity is freed from incident firefighting to innovation. 13. Shadow IT Discovery Rate: AI-led detection of unauthorized capabilities, helping with governance and cost optimization. 14. CX and EX Alignment Score: AI-powered ITSM aligns employee and customer experience outcomes with business KPIs. 15. Cost-to-Serve Reduction via AI: Tracks how AI reduces cost per ticket/user/service. Bottom line? AI in ITSM isn't about doing IT faster. It’s about making business better. Organizations that invest in AI-powered ITSM today are not just solving tickets, they’re building intelligent, adaptive digital experiences that unlock exponential value. Still stuck optimizing candlelight or ready to plug into the grid of intelligent service, explore #ServiceNow Predictive Intelligence and Agentic AI? Repost if this resonates with you. #AIinITSM #DigitalTransformation #AIOps #Automation #ITStrategy #EX #CX #FutureofIT

The prevalent Productivity metrics to measure return on Digital Investment is inherently flawed. This single-metric approach misses crucial value dimensions. 💡 However, as per Deloitte, 81% of companies are still doing just that...using productivity as their key metric for digital transformation ROI. This approach overlooks critical connections between: - traditional financial KPIs and - purpose metrics like sustainability, diversity, and organizational strategic fit. Therefore, a balanced scorecard approach covering a holistic set of financial, non-financial, and technical measures is needed. 📊 Organizations using holistic measurement frameworks (spanning financial, customer, process, workforce, and purpose metrics) are 20% more likely to report higher enterprise value from their digital transformations. ⚠️ 73% of leaders cite the "inability to define exact impacts or metrics" as their top measurement challenge. As per the below study by Deloitte, the most successful digital leaders share four traits: ↳ They use a comprehensive framework of KPIs various value categories ↳ They avoid over-indexing on just a few metrics ↳ They treat measurement challenges as solvable ↳ They share a strategic growth mindset What metrics beyond productivity are you tracking for your digital initiatives? #DigitalTransformation #BusinessStrategy #LeadershipInsights #DataDrivenDecisions

When I worked in civil construction doing business intelligence and reporting, one of the most important metrics I was asked to track was asset utilization. Once I took a closer look, it became obvious why. These large pieces of equipment represented significant capital investments, and by monitoring their usage, we could optimize preventative maintenance, accurately account for depreciation, ensure the right projects were being charged for the right amount, reduce downtime, and gain insights on which assets to acquire or sell. Ultimately, this data-driven approach led to smarter business decisions and improved profitability. Similarly, in construction technology, I believe this same metric is extremely important, yet it is vastly underutilized in the ConTech space. Examining the utilization of your technology stack unveils a multitude of valuable insights, such as natural growth and product traction, training and SOP needs or gaps, compliance, and decisions around scaling or discontinuing certain tools. Regardless of how you use utilization data, it ultimately opens the door to identifying potential problems or successes you might not have been aware of, enabling you to allocate support and resources more effectively. #Construction #ConstructionIndustry #AEC #ConTech #ConstructionTechnology #ConstructionInnovation

How well does your product actually work for users? That’s not a rhetorical question, it’s a measurement challenge. No matter the interface, users interact with it to achieve something. Maybe it’s booking a flight, formatting a document, or just heating up dinner. These interactions aren’t random. They’re purposeful. And every purposeful action gives you a chance to measure how well the product supports the user’s goal. This is the heart of performance metrics in UX. Performance metrics give structure to usability research. They show what works, what doesn’t, and how painful the gaps really are. Here are five you should be using: - Task Success This one’s foundational. Can users complete their intended tasks? It sounds simple, but defining success upfront is essential. You can track it in binary form (yes or no), or include gradations like partial success or help-needed. That nuance matters when making design decisions. - Time-on-Task Time is a powerful, ratio-level metric - but only if measured and interpreted correctly. Use consistent methods (screen recording, auto-logging, etc.) and always report medians and ranges. A task that looks fast on average may hide serious usability issues if some users take much longer. - Errors Errors tell you where users stumble, misread, or misunderstand. But not all errors are equal. Classify them by type and severity. This helps identify whether they’re minor annoyances or critical failures. Be intentional about what counts as an error and how it’s tracked. - Efficiency Usability isn’t just about outcomes - it’s also about effort. Combine success with time and steps taken to calculate task efficiency. This reveals friction points that raw success metrics might miss and helps you compare across designs or user segments. - Learnability Some tasks become easier with repetition. If your product is complex or used repeatedly, measure how performance improves over time. Do users get faster, make fewer errors, or retain how to use features after a break? Learnability is often overlooked - but it’s key for onboarding and retention. The value of performance metrics is not just in the data itself, but in how it informs your decisions. These metrics help you prioritize fixes, forecast impact, and communicate usability clearly to stakeholders. But don’t stop at the numbers. Performance data tells you what happened. Pair it with observational and qualitative insights to understand why - and what to do about it. That’s how you move from assumptions to evidence. From usability intuition to usability impact. Adapted from Measuring the User Experience: Collecting, Analyzing, and Presenting UX Metrics by Bill Albert and Tom Tullis (2022).

AI in Healthcare: What We Measure Determines What We Scale In healthcare, innovation isn’t just about what we build. It’s about what we measure. Because what we choose to measure is what gets resourced, defended, scaled, and institutionalized. Too often, we fall in love with performance metrics without asking whether we’re solving the right problem or whether the benefits actually reach patients and providers in the real world. Here’s how I break down the four stages of responsible AI adoption and the metrics that matter most at each: IDEA – Does the problem matter? We often over-index on technological possibilities and under-index on problem clarity. Key metrics here aren’t precision or recall. They are: • Problem significance (How big is the gap or harm?) • Workflow relevance (Is this aligned with real clinical or operational bottlenecks?) • Strategic fit (Does it support institutional goals or health equity outcomes?) PROOF OF CONCEPT (PoC) – Can it work technically and operationally? At this stage, metrics help reduce uncertainty: • Model performance: sensitivity, specificity, AUC • System integration: latency, uptime, backend compatibility • Early user signals: perceived usefulness, usability, acceptability PoC tells us if it can work, not if it should. PROOF OF VALUE (PoV) – Does it matter enough to justify adoption? This is where many projects stall. And rightly so, because the bar gets higher: • Clinical impact: outcomes improved, risks reduced • Operational value: time saved, throughput increased • Economic justification: cost-effectiveness, ROI • User experience: trust, burden, intent to reuse • Equity: Does it serve diverse populations equally? If PoC is about internal validity, PoV is about external consequences. MAINSTREAMING – Can it scale safely, sustainably, and equitably? Scaling AI isn't a technical task. It’s a systems leadership challenge. Key metrics shift toward: • Implementation fidelity • Training and adoption rates • Safety triggers and override behavior • Equity audits: performance across demographics, comorbidities, language • Governance readiness: procurement, documentation, feedback loops Mainstreaming means moving beyond what works in pilot to what survives and improves in practice. As a clinician trained in medicine (MBBS), public health (MPH), and business strategy (MBA), I’ve come to see metrics not as technical detail but as ethical choice. We don’t scale what’s possible. We scale what we measure and what we reward. What metrics have helped you decide when an AI tool was ready to move forward or when to walk away? #AIinHealthcare #PoC #PoV #Mainstreaming #ClinicalAI #HealthInnovation #MBBSMPHMBA #HealthEquity #DigitalHealth #Enneagram5 #INTP #StrategicDesign #ResponsibleAI #HealthSystems #InnovationGovernance