Thousands Of Flashlights: How AI Is Lighting Up Dark Corners
In my previous two articles in this series, we explored the streetlight effect, that is, when we tend to look for something where it is convenient (under the streetlight) rather than where it is (the dark park). In the case of Learning and Development (L&D), it means measuring easily available metrics that are in our control (course completions, time spent in training, and satisfaction). This last article will shed light on (pun intended) the darker corners of measuring learning impact, and show how one can think of AI as a force multiplier in L&D analytics.
Why Is It Difficult To Measure The Impact Of Learning?
Aligned with several studies and my own experience, the latest ATD research on the future of learning evaluation revealed the same barriers and challenges [1]:
- Lack of time and resources
- Lack of access to data
- Lack of skills
- Lack of buy-in and support from stakeholders
Sounds familiar? No wonder L&D stays in the well-lit area of the LMS. In order to measure impact on the job, we have to get out of the LMS bubble and work with the business, IT, talent acquisition, etc. We need many flashlights for the dark corners. Up to now, scalability due to the lack of time and resources seems to be one of the biggest barriers.
Technology will not remove all your barriers. Culture, lack of clarity, broken processes, unclear goals and responsibilities, lack of accountability, etc., needs to be addressed by humans before Artificial Intelligence (AI) can help.
The ability to shine a light in many places at once, quickly and intelligently, to see the whole picture of impact, is what data analytics, AI, and automation promise.
6 Ways AI As Force Multiplier Can Help With Measurement And Evaluation
1. Assisting With Strategy
Prioritization, trade-off analysis, backward design, and ROI calculations are some of the examples where automation and AI can provide you with guidance on what to take on in the first place and how success should be measured.
2. Assisting With Design
Even before measurement and evaluation, you can use AI to help you with assessment writing, for example. For learning designers, I built an AI bot that analyzes your assessment questions and provides detailed scoring, suggestions, and feedback on the approach. These assistants are now evolving into agents with the ability to execute, not just suggest, actions.
3. Turning Satisfaction Surveys Into Performance-Focused Questions
Turning backward-looking satisfaction surveys into performance-focused questions producing actionable data insights is another area where AI can help. Another AI service was trained on the learning-transfer evaluation model (LTEM) and performance-focused survey question design to help create more actionable data [2].
4. Analyze Data At Scale And Depth
AI in L&D measurement can help us gather and analyze data at a scale and depth that was impractical before. Where a human analyst might struggle to correlate training data with six different performance metrics spread across three systems, an AI-driven tool can crunch those numbers in seconds and spot patterns. For example, AI can track learning data alongside performance metrics over time to identify correlations, compare groups (who took the training vs. who didn’t), and even parse qualitative data (like open-ended survey responses or work product samples) to see how learners are applying skills [3].
5. Lack Of Time And Resources
Open-text responses, real-time chat interactions, or small break-out group conversations can now be analyzed to summarize insights, find patterns, categorize responses, predict sentiment, etc. Lack of time and resources? Solved.
6. Immersive Dialogue And Built-In Measurement
At the ATD TechKnowledge conference in February 2025, I shared a prototype of a 3D adventure where users could interview persons of interest based on a given rubric of best practices. The AI characters interacted in real time, and they had their short-term and long-term memory. They shared facts about the world, but also had an awareness of each other. At the end, the AI coach provided a detailed analysis of the interviews. All this I built within a month. My prediction is that this type of immersive activity will soon be available on all decent learning platforms.
One 2025 industry report noted that advanced AI enables more sophisticated approaches to link learning and performance—instead of just tracking completions, AI-powered analytics can evaluate things like understanding, application, and behavior change, which are “the real drivers of business performance” [3]. This means AI isn’t dazzled by the streetlight: it’s actively looking for the glow of impact in the dark.
Predictive Analytics To Provide Actionable Insights
Moreover, AI can predict and prescribe. Through predictive analytics, AI might highlight which employees are likely to benefit most from a particular training (so L&D can target interventions better). It can also help identify if a performance issue is emerging that training could help with, essentially alerting L&D to a need before the business even asks. In our metaphor, AI might not only shine a light where the keys are, but even predict where you should look first (“based on past patterns, keys are usually dropped near the park bench”).
And finally, privacy and ethics cannot be ignored—shining a light everywhere should not mean spying on employees or violating trust. The goal is to illuminate impact, not intrude on privacy.
We have the technology to truly measure what we’ve always cared about: actual behavior change and business results in a scalable, real-time way. Think of AI as a force multiplier of your impact in the new world rather than a threat to your job in the old.
A Bright Future: Measuring What Matters Across All L&D Roles
Stepping out of the streetlight’s narrow circle and into a broader, well-lit landscape of measurement isn’t just a nice-to-have, it’s the future of L&D. And it requires a culture shift that touches every role in the L&D field:
For Instructional Designers
It means designing with measurement in mind. Embrace models like LTEM to ensure your learning solutions include opportunities to demonstrate application.
For L&D Program Managers And Facilitators
It’s about reinforcing the learning on the job and following up. You may need to partner with line managers to gather feedback on behavior change, or set up post-training touchpoints (like refreshers or coaching sessions) that both boost transfer and yield insights on progress. Instead of declaring success when the class ends, you’ll see your role extending into the workplace: guiding managers on how to support new behaviors, and maybe doing light measurements like sampling work outputs or conducting focus groups to hear how folks are applying (or not applying) the training.
For L&D Leaders
This is about strategy and culture. Lead the charge in aligning learning to business goals. Advocate for the tools and resources (perhaps investing in an LRS, or analytics talent, or AI platforms) that allow your team to measure what matters. It will also fall on you to educate stakeholders. Set expectations with executives that L&D will report on business outcomes, not just activity, and then deliver on that promise. Why not use the measurement rubric and prioritize project requests where stakeholders are willing to collaborate on measuring real impact?
For Learning Analysts Or Data Scientists
Your skills in analytics and facility with AI tools will help translate raw data into meaningful stories. You’ll experiment with different methods (A/B tests for training, predictive modeling, etc.) to truly understand causation, not just correlation.
Conclusion: AI As Force Multiplier
Ultimately, avoiding the streetlight effect in L&D measurement means having the courage to seek the truth, even if it’s in murky, difficult places. It means trading the immediate comfort of an easy metric for the more rewarding payoff of a meaningful metric. Yes, it’s harder to measure how a new software training affected productivity than to count how many people opened the training video. But which one would you rather bring to your CEO? Which one actually tells you if the training succeeded?
References:
[1] The Future of Evaluating Learning and Measuring Impact: Improving Skills and Addressing Challenges
[2] Learner Surveys and Learning Effectiveness with Will Thalheimer
[3] Measuring What Matters: Connecting Learning Outcomes to Business Results with AI