Learning Transfer Evaluation Model (LTEM)

Q: Tell us about LTEM, the Learning Transfer Evaluation Model, its core principles and how it differs from other approaches to measurement.

A: LTEM is an eight-tier learning evaluation framework originally published in 2018. LTEM was designed because I’ve been working in the field a long time and there have been many concerns from practitioners about the Kirkpatrick four level model.

I often find myself calling it the Katzell-Kirkpatrick model because the Kirkpatrick work was built on top of the earlier work of Raymond Katzell. Kirkpatrick’s version of the four-level model was easy for people to understand and adopt. I think both gentlemen deserve credit.Researchers in the field have criticized the four-level model as ignoring 40 years of research on human learning and leads to a checklist approach. Critiques can sometimes be valid and sometimes over the top. My thinking is more moderate – all models have strengths and weaknesses.When I speak to conference audiences or at workshops on learning evaluation, I discuss the Kirkpatrick model. It nudges us and sends some good messages, encouraging us to think about work performance and organizational results. It’s beneficial that it prioritizes these elements. However, it suggests that learner perceptions aren’t as important, assigning them a lower priority.One significant issue with the model is its silence on the aspect of remembering. The model puts all learning into a single category—Level Two—which doesn’t capture the nuances between different types of learning. Learning can range from simple recall of facts to the development of meaningful skills and competencies.This broad spectrum is often reduced to mere knowledge checks, which we know are not sufficient. In essence, there is no learning wisdom baked into the four level model. Learning is put all in one bucket level two, and that’s just raw.For example, you know, you can measure, learning with the regurgitation of trivia, or the recall or recognition of meaningless information. Or you can measure use of the meaningful information, or decision making competence, or task competence or skills. There’s a big distance between trivia and skills.Relying on an aggregate level called learning can lead to using a knowledge check which measures recall. And we know knowledge alone is insufficient. That’s a big problem in the four level model. While it encourages us to do some things (i.e., measure learning) a good model should also discourage us from doing things that aren’t good enough (i.e., knowledge check).The LTEM model went through 11 iterations before I published it in 2018. Throughout its development, I got feedback from a lot of smart people—learning evaluation experts like Rob Brinkerhoff and Gregoire Lopez, learning experts like Julie Dirksen and Clark Quinn, and a bunch of smart practitioners and made it better in each of those 11 iterations.One of the biggest things that’s important about LTEM is that it separates learning into three tiers: knowledge, decision making competence, and task competence. And that has a lot of implications. When people use LTEM, they’re thinking at a much higher level than just knowledge checks.In fact, Dr. Elham Arabi did her doctoral dissertation on LTEM. Her hypothesis was that if you introduce LTEM to a learning team, not only will they do better learning evaluation, but also they’re going to be inspired to do better learning designs. And she introduced this at a hospital with nurse professional development. And that’s exactly what she found.It makes sense. If you start thinking about measuring decision-making competence, they will think about their design and build in more decision-making practice. That’s what the scientific research and learning says we need—realistic practice and decision making and doing tasks that are we have to do on the job.

Q: You help your clients with presentation science, and I imagine it’s morphed some over the last few years with more organizations operating virtually. What are the top two or three things that people seem to not already know about presentation science?

A: Recently, I attended a conference filled with intelligent individuals presenting their ideas. As I observed the presentations, it struck me how often we’ve experienced the sheer boredom sitting through countless PowerPoint slides, accompanied by a monotone delivery and an endless parade of bullet points. To combat this, I recommend a few strategies to revitalize presentations and maintain audience engagement.Transform Bullet PointsInstead of traditional bullet points, turn them into visual objects. For example, you could replace the text after a bullet with text inside a round circle, which can make the content on the slide stand out. Understand Cognitive PatternsRecognize that human cognition is wired to habituate, meaning we get used to and less interested in repeated patterns like bullet points. Changing the format can help maintain the audience’s interest.Be Mindful of Eye MovementPeople may not be familiar with the research showing how our eyes dart around a visual field, scanning rather than fixating. When designing slides, it’s crucial to be strategic about where logos or decorative graphics are placed, as these elements naturally attract attention. These should be used sparingly and with intent.Use WhitespaceLeaving a significant amount of space on the slide without objects or content to pay attention to helps people focus their attention where it belongs—on your brilliance. Sequential RevealDon’t present an entire slide all at once. Reveal one point or part at a time to guide the audience through your argument or explanation, supporting their cognitive process. Use a ClickerTo smoothly navigate through your presentation, use a clicker. This allows you to move through slides without awkward pauses or the need to physically interact with your laptop. Prioritize ECRAIn my approach to presentations, which focuses on them as learning events, I advocate for creating sessions that facilitate engagement, comprehension, memory retention, and action—a model I refer to as “ECRA.” By emphasizing these elements, you help the audience not just to listen but to learn and be ready to act on the information presented.Remember, while these tips can significantly improve the effectiveness of your presentations, they are part of a larger field of presentation science, which includes emotional and motivational aspects, among others. Viewing presentations through the lens of learning science can offer extensive insight and techniques to enhance the experience for both the speaker and the audience.

Q: What have you noticed has been changing in the soft skills needed, both by individual contributors and people leaders?

A: The core soft skills required by both individual contributors and people leaders have largely remained foundational over time. My grandfathers, who worked in large companies and managed teams long ago, dealt with many of the same interpersonal dynamics that we see today. Despite changes in workplace traditions, the essence of facilitating a productive team endures.In my own experience—especially in the context of leadership development—I’ve had the opportunity to collaborate leadership trainers for the US Air Force. Together, we developed survey questions to assess leadership development training. Through this work and my subsequent research into the science of leadership, I identified several key attributes of effective leadership.

1. Caring for the Team: A good leader should genuinely care for the members of their team.
2. Decision-Making: Leaders must be adept at making decisions, which involves being situationally aware of when to decide independently and when to involve the team or delegate.
3. Trust: Building trust is crucial. Leaders should earn the trust of their team by demonstrating they have the team’s best interests at heart.
4. Emotional Intelligence: Today, there’s a heightened awareness of emotional intelligence, which is the ability to be cognizant of and sensitive to others’ emotions and needs.

I’ve observed that many professionals, particularly those with a background in technical fields like engineering or programming, often excel in systematic thinking but may lack in the people skills department.They’ve honed their technical expertise at the expense of developing soft skills. Therefore, part of our focus in leadership training is to help these individuals become more attuned to the interpersonal aspects of leadership. It’s a gradual process, as people typically excel in areas where they’ve invested their time and energy. Our goal is to bridge that gap and enhance their sensitivity to the ‘people side’ of their roles.While some nuances of soft skills may evolve, their fundamental importance in leadership remains constant. It’s about balancing technical acumen with emotional intelligence and interpersonal effectiveness to lead teams successfully.