The Seven Applications

By Matthew S. Richter

Myfriend, colleague, and podcast partner, Will Thalheimer, created the greatest instructional design and learning evaluation tool I have ever used. He teases me that it is the only one I have ever used. But, that isn’t true. Now, I say all this with no exaggeration, nor any attempt to blow smoke his way. In fact, I would rather Will not see this article and know how highly I think of him and this phenomenal resource he has given to the L&D community.

LTEM, or the Learning-Transfer Evaluation Model, was developed initially as a response to the blind faith and adoption of the Kirkpatrick-KatzellFour-Level Model many of us (me, too) have been applying for years. Will, and others, have written much about the needs for an alternative to Kirkpatrick, so I won’t go there. Will has introduced LTEM as an evaluation tool. But, as you will see, I find it so much more useful than just that. For me, I use it as an instructional design planning tool, a coaching reference, and more. This article will give you a brief introduction to LTEM (and, yes, I have Will’s permission to reprint it here), and then share with you the varied applications I have used it.

LTEM focuses on the different stages of learning and the transfer of that learning to an applied environment—if at all. It has eight tiers, with the lower tiers identifying what has traditionally been the focus of corporate training—short term retention; the middle tiers describing longer term learning and understanding; and finally the top tiers illustrating where learning shifts into full application and integration on the job.

The trick with learning is we can easily be fooledinto thinking it has occurred based on different ways in which participants engage with its overall process. Meaning, if learners engage in activities, we can often think that they have therefore learned. But engagement doesn’t equal learning. In fact, most of what we business folks have thought of as learning isn’t learning at all.

Ok… let’s take a step back… what does it mean to have learned something? The answer to this question is pretty complicated and it’sthat complication that makes LTEM so much more useful. Learners are often evaluated based on just attendance, participation in activities, and their own perceptions of their learning. Indeed, these can be viewed as beginning points, but they don’t even begin to get at what learning is. First, we have to ascertain whether a learner understands the concepts of the studied topic. But understanding doesn’t often lend itself to application. So, comprehension is merely in the middle of the pack when it comes to the levels represented by LTEM. For example, as a boy, I had a great desire to become a wide receiver for the Dallas Cowboys. So, I went out andread every book I could find on the mechanics of catching a ball. I studied trajectories and the physics of curves. I bought into the notion that equipment like good gloves would make me a better catcher. I read everything I could find, including Dallas Cowboy playbooks. If I had been given a paper and pencil test, I would have scored a 110%. One day, my father came into my room and suggested we go out and he would throw me some passes. I had to think about this. There was no book for that! But, my studies had prepared me. The first pass went over my head. When it came to making play decisions and modifications on the fly, I failed miserably. I turned left when Dad threw right. I forgot how many steps to go out for the button hook. The sixth pass jammed my finger. The eighth one broke my nose. So much for retained knowledge. Oops. I had learned to a point, but not to the level of transfer to application. LTEMthoroughly describes the different levels of learning. Now, how can I apply this in the work I do? I’m glad you asked.

EVALUATION. Well, this was Will’s purpose for designing LTEM. And, it is great. I love holding up my designs next to it and seeing what the activities and performance tests lead to on the chart. So, it is a great evaluative reference before I deliver anything. I also like to reference it during program implementation. I can query where my participants are literally as a result of my activities. And, finally, I can use it afterwards to evaluate where learners are after the program is complete. To paraphrase my business partner and mentor, Thiagi, there are only three times one can use LTEM as an evaluative tool— before, during, and after learning. But only those three times.
INSTRUCTIONAL DESIGN. This is the obvious one. As a designer, I use it to quickly ascertain the best activities and performance test appropriate for the desired tier. As a consultant doing client designs, I take LTEM and sit down with my customers. I ask them to pinpoint in the model where they want their learners after the intervention. Traditionally, before LTEM, I would simply ask, “What do you want participants to do differently as a result of the program?” The problem with that question was not the question, or even the answers I would get. No. The problem was in managing their expectations of what they would espouse with what they would expect delivered. I would hear, “I want my employees to deliver feedback effectively at the right time.” More clarifications, and I would have a fairly decent understanding of what they wanted. But, then, I would explain that to get there, we need to do “x” and it would take 2-3 days to achieve a result that would then require “y” efforts back on the job. This usually would get them laughing at me with, “Well, what can you do in two hours?” LTEM gives me a graphic to share what outcomes can be achieved and how investing more time and effort can yield better outcomes. While evaluation tools are not in of themselves new, LTEM is conceptually unique. It is the only tool I can find that separates learning into knowledge, decision-making, and task competence. The framework gives me a better language and visual to explain to clients the implications of what they ask for versus what they want. It makes it easier to manage those scoping conversations.
TRAINING GAME DESIGNER. Similar to my applications in the above bullet, when designing activities, I set a goal using LTEM. Then I design the activity to that goal. When designing activities, there must always be an instructional purpose. Too often, the risk is we can design a cool game absent an appropriate goal. Or, the goal is peripheral. LTEM helps me validate the activity is on target.
COACH. As an executive coach, I like to set goals with my clients. But, often those goals are fairly large in scope and ill defined behaviorally. I know, I am not supposed to say that, but for example, often I get hired to work with an executive who doesn’t manage others well, or doesn’t play well with others. The key metric is afterwards, he does do those things. That’s actually easy to measure and keeps me employed. Where is can be a bit nebulous is setting smaller goals as we get there. Lately, I’ve been pulling out LTEM as a coaching tool and building a development plan with the coacheeto reflect the model. This has worked extremely well and makes me seem brilliant. Little do they know.
PERFORMANCE CONSULTANT. Well, ditto, to all the above. LTEM has become a structure for managing expectations with clients and also serves as an evaluative tool for the intervention.
KEYNOTER/ PRESENTER. I’ve even taken to using LTEM as a guide for my large group presentations. I certainly cannot attain Tiers 5-8 usually during a 75 minute keynote, but I can make sure I am mastering Tiers 1-4. Depending on the topic, we might even delve into Tier 5. The key is LTEM has become a rubric I use to ensure my presentations are not just pablum, or empty infotainment.
SALES / BUSINESS DEVELOPMENT. I know this one might be a stretch and Will might kill me for over-extrapolating. But, when I teach sales, LTEM becomes a template I use to teach value-propositions. Rather than the paradigm of learning, I talk about customer reactions/ applications and integration. In other words, I don’t use the discourse or vocabulary for learning, but I use the tiers to describe client experience. For example, Attendance = showing up for the call. Activity = I get to write a proposal, or I get a second meeting. Learner Perception = they like me and are considering hiring me. They may buy… once. They may not yet fully understand how or why we can help, but get enough of a feel to buy. Knowledge = they fully understand how our product works and see an application for it. Decision Making Competence = they can use our product properly. It hasn’t been integrated into their business, but they can mechanically use it. Task Competence = They can use the product in their business. Transfer = they can modify the product and adapt it to their business needs. Effects of Transfer = They quantitatively see a return on their investment from a productivity, financial, and quality perspective.

The bottom line is I love LTEM and I strongly believe there are countless other applications for it. Will never asked me to write any of this. I had to ask him for permission. I think he was a bit in shock at my adoption rate. It is more complex than Kirkpatrick’s Four Levels. Yep. But, it is not more complex than necessary. And, it certainly isn’t too simple so as to be banal. LTEM is practical. And, as I continue to use it with my clients, it is utterly comprehensible to them. Take a look. Call Will with any questions.

CLICK HERE TO DOWNLOAD A PDF OF THIS ARTICLE.

To learn more about LTEM and Will Thalheimer’sother work, go to: http://bit.ly/Thalheimer