A few years ago, a group of senior learning and development (L&D) and operational leaders were sitting in a conference room with a consultant evaluating the success of a recently launched gamified learning platform. The engagement looked good, really good. High participation numbers, employees climbing the leaderboard, time in the system was high. The executives where pleased with the results and started patting themselves on the back. The consultant let them self-congratulate for a few moments and then quietly asked, “Are sales up? Is performance improving? Has that engagement translated into performance?”

Those senior leaders struggled to find an answer.

One of the advantages of a gamified system is the ability to track employee actions and activities and collect a ton of data. The trick to managing the data and successfully leveraging the power of gamification is to ensure you understand the types of data that can be collected and what that data means to the organization.

A helpful way to make sense of gamification data is to divide gamification metrics into three distinct categories, or levels:

• Level 1 focuses on initial engagement. This level looks at whether employees are logging into the gamified system and whether they are interacting with it on a regular basis.
• Level 2 examines whether learning occurs as a result of the gamified approach. Here you want the metrics to give you evidence of knowledge acquisition or skills development.
• Level 3 is the most important level, which considers whether those changes translate into improved performance on the job.

Looking at gamification metrics through these three levels helps organizations move beyond simple engagement measurements to more helpful performance measurements.

Level 1 Engagement Metrics

The first level is what many people refer to when they ask if employees are “engaged.” This level looks at how (and if) employees are interacting with the gamified learning platform. These metrics capture activity within the system and help determine whether the experience is drawing and maintaining interest.

These metrics often include:

• Login frequency or number of logins
• Number of active users
• Session durations (time in system)
• Challenges complete
• Points earned
• Badges collected
• Leaderboard participation rates

While these metrics will not move the needle on organizational performance, they should not be ignored. If employees are not using the gamified platform or immediately log off the system after checking it out or find the gamified learning system to be patronizing or juvenile, then you’ll never get performance results.

If you are getting poor metrics at this level, you should look at the amount of managerial support, the design of the system, the system launch process, level of perceived relevance by the employees, feelings of big brother watching, technical glitches or complexity of design. If an organization doesn’t get this level right, the system will never achieve even partial success. This level of failure is a back-to-the-drawing board level of failure.

Fortunately, this level of failure is rare. Careful planning, a focus on relevance, a thorough needs analysis, focus group interviews with employees and identification of performance issues before implementation all can help make this level a success.

Level 2 Learning Metrics

The second level of metrics focuses on whether employees are actually learning anything. This level looks beyond merely logging in and experiencing the gamified learning platform. This level examines the learning results that occur once an employee engages with the gamified experience.

These metrics include:

• Quiz and assessment scores
• Score improvement over time
• Completion of knowledge-based missions
• Mastery badges earned after demonstrating proficiency
• Accuracy in scenario-based decisions
• Progress through skill-based levels

These measurements reflect action learning by determining whether employees actually learn from participating in the gamified experience. Gaining knowledge is an important prerequisite to performance, so it’s essential to understand whether learning has occurred as a result of the gamified solution.

If you see poor metrics at this level it means that the instruction provided by the gamified system is inadequate or poorly design. If employees are engaged at Level 1 but failing at Level 2, re-examine the design of the instruction, the questions and any distractors in the quizzes and knowledge checks, as well as whether the quizzes are aligned with the content being presented. Failure here is not a gamification failure, it’s an instructional failure, and the instruction contained within the gamified system needs to be fixed.

Level 3 Performance

At this level, you want to know if the gamified learning platform is actually impacting workplace performance. After all, this is the entire goal of a gamified approach. If on-the-job performance is not increasing or improving, you need to rethink the entire project.

While these metrics at this level need to be tied to organizational key performance indicators (KPIs) or performance goals, here are some general performance metrics that are typically measured at this level in a gamified learning system:

• Increased sales conversions
• Improved customer satisfaction scores
• Reduced safety incidents or compliance violations
• Faster task completion times
• Improved quality metrics or reduced error rates
• Higher productivity levels
• Improved employee retention or reduced turnover

These metrics should be tied to the business. They are the needles that the organization wants to move. The more closely these metrics are tied to the success of the business, the more accurately they’ll measure how well the gamified system is impacting the performance of the business.

If the other two levels are successful but this level is not, then there are elements to consider.  One might be a misalignment between the learning objectives being taught and the required on-the-job knowledge, performance and activities. You want to look to see if the designed learning activities are mirroring what is happening on the job and if they are providing enough practice for skills transfer. Just because you know something, it doesn’t mean you can actually apply that knowledge. The learning activities in Level 2 should be geared toward skills transfer and practice of on-the-job tasks.

At this level, the failure might also be caused by issues not related to training. Lack of performance can be a result of things like conflicting priorities, limited authority to act, misaligned incentive programs or even lack of managerial support. In these situations, employees may know the correct behavior but feel unable or discouraged from implementing it. No gamified program, regardless of how well-designed and executed, can overcome non-training related problems with performance.

Conclusion

The amount and types of data and metrics that gamified systems produce can seem overwhelming. It can sometimes be difficult to determine what metrics are truly important and what metrics are not helping improve the organization. By evaluating metrics across all three of these levels, learning leaders can move the conversation beyond engagement and toward the questions executives really care: “Are sales up? Is performance improving? Has that engagement translated into performance?”