Outcome Evaluation Techniques

Is it Time to Move Beyond Kirkpatrick & Phillips for Measuring L&D Effectiveness? Did you know organisations spend billions on Learning & Development (L&D), yet only 10%-40% of that investment actually translates into lasting behavioral change? (Kirwan, 2024) As Brinkerhoff vividly puts it, "training today yields about an ounce of value for every pound of resources invested." 1️⃣ Limitations of Popular Models: Kirkpatrick's four-level evaluation and Phillips' ROI approach are widely used, but both neglect critical factors like learner motivation, workplace support, and learning transfer conditions. 2️⃣ Importance of Formative Evaluation: Evaluating the learning environment, individual motivations, and training design helps to significantly improve L&D outcomes, rather than simply measuring after-the-fact results. 3️⃣ A Comprehensive Evaluation Model: Kirwan proposes a holistic "learning effectiveness audit," which integrates inputs, workplace factors, and measurable outcomes, including Return on Expectations (ROE), for more practical insights. Why This Matters: Relying exclusively on traditional, outcome-focused evaluation methods may give a false sense of achievement, missing out on opportunities for meaningful improvement. Adopting a balanced, formative-summative approach could ensure that billions invested in L&D truly drive organisational success. Is your organisation still relying solely on Kirkpatrick or Phillips...or are you ready to evolve your L&D evaluation strategy?

"There is a need to reconstruct how we use existing measurement tools, techniques, and methodologies so that they capture the complexity of the environment in which an intervention or change occurs." - Siddhant Gokhale and Michael Walton (2023) This superb and extensive guide by United Nations Population Fund (UNFPA) introduces adaptive evaluation - an approach designed for complexity. In increasingly turbulent, uncertain, novel, and ambiguous environments traditional evaluation methods often fall short. "In a complex system, we cannot predict what will happen. What will happen depends on the (evolving) interactions between actors and changing external conditions." This guide provides the tools and mindset needed to embrace complexity, foster learning, and adapt in real time. In this guide, you'll find: 1️⃣ Approaches, methods and techniques - What to do and how to do it 2️⃣ Attitudes, believes and values to make it work - The mindset At 105 pages, this resource offers a wealth of insight. The authors have categorised this insight to align with 6 key challenges: 1️⃣ Methods to foster evaluation use 2️⃣ Methods for learning and adaptation in real time 3️⃣ Methods to capture complexity 4️⃣ Methods to capture contribution in unpredictable environments 5️⃣ Leadership roles in adaptive evaluation 6️⃣ The adaptive evaluation mindset I can see myself coming back to this resource time and time again throughout 2025. "Evaluative thinking is not synonymous with evaluation. As IllumiLab says, “Evaluation is the doing, while evaluative thinking is the being”. Evaluation is a set of activities, while evaluative thinking is an approach and a way of thinking."

Lessons From the Frontlines: Evaluating RAG and LLMs for Real-World Results 1. 🎯 Anchor Everything in the Business Problem Teams often rush to optimize technical metrics, but the real question is: does this solve a business pain? Early in our journey, we celebrated high retrieval scores—until we realized customers still struggled to get clear answers. The lesson: define success with stakeholders before building. If the outcome doesn’t move a business metric, it’s the wrong target. 2. 🔄 Evaluation Is a Continuous Cycle The temptation is to treat evaluation as a milestone. In practice, it’s a loop. Each deployment reveals new user behaviors, new edge cases, and new opportunities for improvement. The most valuable insights come after launch, not before. The process: deploy, observe, learn, refine—repeat. 3. 🧩 Choose Metrics That Reflect Reality Technical metrics are necessary, but not sufficient. For RAG, the focus is on: Are we retrieving the right information? Are generated answers grounded in facts? Are users satisfied and returning? Is there measurable business impact? If a metric doesn’t tie to a business outcome, it’s deprioritized. 4. 🧪 Balance Offline and Online Evaluation Offline tests (precision, recall, latency) are fast and safe, but they don’t capture the full picture. Real-world use always surfaces surprises. One deployment looked flawless in the lab, but failed when users asked questions outside our test set. Only live data revealed the gaps. The approach: validate offline, but trust online feedback. 5. 🗣️ Communicate With Clarity and Context Data without context is noise. When sharing results, the focus is on the story: what changed, why it matters, and how it impacts the business. If a technical metric drops but user engagement rises, that’s a win. The narrative always ties back to business goals. 6. 🔁 Iterate Relentlessly The first version rarely gets it right. User feedback is the most valuable input—sometimes uncomfortable, always instructive. When users flagged irrelevant answers, the team didn’t defend the model. We adjusted retrieval, retrained, and improved. Each cycle brought us closer to the mark. Feedback is not a threat; it’s a guide. 7. 📝 Document the Journey Institutional memory is fragile. Every experiment, every decision, every lesson is logged. 8. Outcome-First Development (OFD) means starting every feature or model change with a clear plan for how success will be measured, not treating evaluation as an afterthought. This approach aligns engineering, product, and business teams, ensuring everyone builds with the end in mind. OFD helps catch issues early and keeps progress visible and actionable for all stakeholders. This discipline prevents repeated mistakes, accelerates onboarding, and ensures progress is cumulative, not circular. #RAG #LLM #AILeadership #AIEvaluation #BusinessImpact #MachineLearning #ProductStrategy #LessonsLearned

I've analyzed hundreds of L&D programs. If your L&D metrics stop at "completion rate," you're running a compliance factory, not a development program. Top L&D leaders measure this instead: Development outcomes in the form of behavior change. Here’s an example of a training outcome vs. a development one: Training outcome: "98% of staff completed food safety training." Development outcome: "Food safety incidents decreased 42% quarter-over-quarter after implementing our new training approach." See the difference? One is about checking boxes. The other is about changing behaviors that impact the business. The most effective learning leaders I work with: 1. Start with the business problem they're trying to solve 2. Identify the behaviors that need to change 3. Design learning experiences that drive those behavior changes 4. Measure the impact on actual performance This isn't just about better metrics—it's about repositioning L&D from service provider to strategic business partner. When you can walk into an executive meeting and talk about how your programs are moving business metrics rather than just completion rates, everything changes.

Choosing the Right Type of Evaluation: Developmental, Formative, or Summative? Evaluation plays a critical role in informing, improving, and assessing programs. But different stages of a program require different evaluation approaches. Here’s a clear way to think about it—using a map as a metaphor: 1. Developmental Evaluation Used when a program or model is still being designed or adapted. It’s best suited for innovative or complex initiatives where outcomes are uncertain and strategies are still evolving. • Evaluator’s role: Embedded collaborator • Primary goal: Provide real-time feedback to support decision-making • Map metaphor: You’re navigating new terrain without a predefined path. You need to constantly adjust based on what you encounter. 2. Formative Evaluation Conducted during program implementation. Its purpose is to improve the program by identifying strengths, weaknesses, and areas for refinement. • Evaluator’s role: Learning partner • Primary goal: Help improve the program’s design and performance • Map metaphor: You’re following a general route but still adjusting based on road conditions and feedback—think of a GPS recalculating your route. 3. Summative Evaluation Carried out at the end of a program or a significant phase. Its focus is on accountability, outcomes, and overall impact. • Evaluator’s role: Independent assessor • Primary goal: Determine whether the program achieved its intended results • Map metaphor: You’ve reached your destination and are reviewing the entire journey—what worked, what didn’t, and what to carry forward. Bottom line: Each evaluation type serves a distinct purpose. Understanding these differences ensures you ask the right questions at the right time—and get answers that truly support your program’s growth and impact.