Every industry is becoming more and more data-oriented. Decision makers are looking to charted numbers to make big decisions about where to put funding. The corporate training industry is feeling it, too, and rapidly so – research from Leo Learning indicates a 71% rise in the number of training managers being pressured to produce numbers and watch learning metrics to justify their budget.
This is frustrating for many learning and development (L&D) professionals – of course learning is important! And it’s important in a way that’s hard to reduce to statistics. Learning is a journey that can impact all areas of an employee’s work, and therefore, can positively transform a company. Learning bonds people, boosts confidence, increases engagement in ways that research may never put a pin in. But, stakeholders without the passion of learning and development want to hear numbers.
Unfortunately for the L&D department, according to Brandon Hall, nearly 30% of companies say they only capture very basic learning metrics, if anything. And 45% say they are not at all effective at measuring informal learning – which, as many L&D professionals know, might be one of the most productive types of learning.
Turn it Around and Talk About Results First
Often, learning metrics turns into something people think about at the last minute. Last-minute metrics are bad metrics because you haven’t been tracking data proactively and you haven’t examined whether the data that you do end up with is actually meaningful.
L&D professionals should be talking about the goals of their project anyways – open that conversation to include the metrics you want to address. We’ll address this more later, but even if every learning goal you have can’t be directly tied to a metric, you can still devise learning metrics that you can tie to at least some of the goals you want to achieve.
You should also ask yourself how you plan to actually use these learning metrics down the road. This will help you pick out metrics that make sense for driving future business decisions. It may also help you decide which metrics don’t make sense to measure. Having good data that no one looks at it is just a waste of time, and ineffectual data that takes time to gather is a waste of resources as well.
Create a Sustainable Evaluation Model
One downfall that companies sometimes fail to consider is the expense and time that evaluation, all by itself, requires. Coming up with an extensive plan is great – but is it sustainable? Make sure that when you’re narrowing in on the methods of evaluation that would be most effective for your company, they are not so involved that you can’t afford to do them repetitively over time.
For example, a plan that includes input from multiple people or departments – especially those
including c-level executives – may not be sustainable long term. Schedules are always packed, priorities can change rapidly and often, and data collection/formation will likely be inconsistent and require formatting.
Similarly, don’t settle on the first evaluation model you think up. It’s important to test out a variety of learning measurement methods so that you can figure out what is most appropriate for your organization.
Kirkpatrick and Phillip’s Models
According to Brandon Hall, the most commonly used learning evaluation models in the industry are Kirkpatrick and Phillip’s models.
Kirkpatrick has become a standard evaluation model within the industry, so most learning and
development professionals are familiar with the four-tier process – including reaction, learning,
behavior change, and business impact.
Phillip’s model expands on that by including ROI as the fifth level of evaluation. He argues that each level leads to the next in a chain reaction.
That may be true, but don’t fall into the trap of believing that every level requires that same amount of attention, time, and money. The evaluation models are a tool and guide, not the end-all-be-all. Depending on the project, not every level needs evaluation. For example, in the case of compliance training, the learning and behavior change levels are more important to evaluate than ROI. Therefore you should allocate evaluation resources to those two levels.
CHOOSE USEFUL LEARNING METRICS
One of the more obvious downfalls surrounding learning metrics is if you pick the wrong metrics to measure. We’re not all data scientists, so sometimes it’s hard to know which piece of data will be meaningful enough to draw useful, scientific conclusions. This is especially true when there are possibly hundreds of metric options. Here are some questions to ask yourself when picking out learning metrics.
Is there enough data for this metric to be scientific? If only 15% of your employees responded to a survey, there aren’t a lot of accurate results you can draw. Make sure that you have enough data to draw scientific conclusions. You don’t want to send yourself on metric wild goose chases.
Is this data source stable enough to draw from again and again? In order to properly judge progress of a learning metric over time, you have to be able to draw from your data source at the beginning of a project, once it’s implemented, in six months, in a year, and so on and so forth. If your data source is not always available or won’t be there for very long, maybe try looking at different sources of data.
Is this a metric I can influence with my project? Learning and development projects, when they come together, can feel like a miracle – and they are! But you have to remember that any initiative will have limitations on where they can make waves. Ask yourself if your project can actually influence the metric you’re looking at. For example, a brand new course teaching updated customer service skills may increase customer happiness and satisfaction – but it’s unlikely to increase worker engagement, or increase the amount that customers spend.
Does this metric matter? Sure, your new course might be able to make your workers better dancers or create world peace, but if your metric doesn’t have any bearings on business operations, employee engagement, or any other areas that your stakeholders will care about, then reconsider using it. Learning metrics help you determine how to move forward with this or similar projects in the future – so make sure these metrics can help impact those decisions when you’re choosing them.
USE THE RIGHT MEASUREMENT TOOLS
There are plenty of conventional learning measurement tools that are still useful in an online
environment and could be made more efficient through technology. This includes surveys, tests
and quizzes for assessment, pre- and post-interviews and questionnaires, and on-the-job training checklists.
E-learning technology also gives us new tools to work with. LMSs can record and collate data into reports. The best LMS reporting features give you access to a wide range of metrics and allows you to customize reports to reflect your evaluation methods. They can also automate reports to save you time and head space.
Another up-and-coming tool is the learning record store (LRS). These are especially useful for gathering informal learning data, which uses xAPI/Tin Can to expand the learning environment. It gathers data about social learning and can also record videos and materials consumed outside the LMS.
Technology in measurement reduces human error and increases efficiency. Survey data can only be so accurate – people’s estimates about how much time they spend on learning can be wildly off. Using technology can even automate much of the data collection, and even go as far as to automate regular reports – emailing themselves to the appropriate people without anyone having to think about it. The Towards Maturity 2015 Benchmark™ research indicated that companies using technology to record learning perform better than those that don’t.
Learning metrics are, understandably, one of the trickiest parts of the elearning process. The sheer volume of learners can be especially hard to track. But the best LMS software has tools available to help you collect and process metrics.
If you do your part and think about results first, create a sustainable evaluation model, choose useful learning metrics, and use the right tools, you’ll be well on your way to a measurement system that works for you and your company – and convince stakeholders to continue investing in your project.