Measuring training effectiveness

A waste of time and money

In the days when newspapers and magazines were printed on actual paper, advertisers used to complain that half their budget was wasted, but they didn’t know which half. We used to think this very funny - but remember YouTube and cat videos hadn’t yet been invented. It remains a reminder, however, that a lot of time and money can be wasted if you can’t or don’t measure how effective something is. 

But things move on. Paper has gone the way of papyrus, and advertisers can not only measure how effective an advertisement is, but also predict how well a campaign is likely to perform.

In the world of learning and development, things are also changing. Organizations have huge amounts of data that can be used to analyse training effectiveness. The Azure data platform, Microsoft Fabric, and Microsoft Power BI are all accessible and cost-effective ways to manage and analyse data. And AI capabilities help to spot trends that were previously available only to specialists. This means that measuring training effectiveness by matching results to learning activities is now available to all organizations. For companies whose competitive advantage is their people, this is a game-changer.

Where to start? 

Whilst accurate training data is important, it’s crucial to first consider what your people are trying to achieve. Having clear objectives by role provides a roadmap for data collection. 

Some roles are easier to measure than others. Those in sales often have highly measurable objectives including sales closed, or meetings scheduled. In contrast, the output for knowledge workers can be more challenging to quantify. But how can you plan training, or measure its effectiveness, if you don’t know what you are trying to improve?

Data-led training

Modern, data-led organisations no longer see learning and development as different to any other part of the organisation. Data-led training is both possible and valuable. By leveraging tools such as Microsoft Fabric with Power BI, the Azure data platform, together with their AI capabilities, companies can now link learning activities to results. 

With clear role objectives, a well-structured data collection plan, together with learning and development data, the process of measuring training effectiveness starts to take shape. You can then sit back and listen sympathetically as your competitors complain that they don’t know which half of their learning and development budget is wasted. 

We are a Microsoft Partner with a depth of experience in data for learning and development. If you would like to find out more about how Microsoft data technologies can save you time and money, get in touch for an initial chat. 

L&D in the age of AI: what’s the point?

What’s the point of learning and development (L&D)? It is self-evident, isn’t it? Better trained people do better work. That’s true, and backed by research, but L&D has become a source of competitive advantage. 

In discussion with clients, I know that measuring training is hard. Asking about objectives always provokes quizzical looks and defensive answers. Providing the training people want is hard enough, without having to define measurements. 

L&D departments are trying to provide a good service for less money, with poorly defined objectives. These anecdotal findings are backed by research. Deloitte’s Global Human Capital Trends reports found only 3% of respondents claiming to be extremely effective at defining or measuring employee value. And if you can’t measure when someone is creating value, how will you know whether L&D has made a difference? 

But L&D is changing. Training has become online learning. Microlearning is real, and not a sign of being unable to concentrate. And knowledge workers go into the office when they can be more productive, not just because they are expected to. 

Then there’s the new kid on the block – generative AI or ChatGPT. Everyone’s new friend is reshaping how we learn and what we learn. Not only can it generate content, but it can also quiz you on whether you’ve understood the subject matter, something that traditional training often missed. 

Training that gets measured with a smiley or a frown will soon be as nostalgic as the golf ball typewriter or phones that were tied to the wall. (Look them up, they really did exist.)

So, in the age of AI and microlearning, what is the point of L&D? 

More and better data will lie at the heart of L&D. Data from a variety of sources will help us define what we expect from people, and what they need to know to do better work. Better metrics will provoke thinking in new ways at a time when options are arriving faster than we can imagine how to take advantage of them. Without clear objectives, you cannot measure the success or failure of any learning programme.

If we want to move learning out of the classroom and into the boardroom, we have to think like leaders. And increasingly, leaders are thinking about data.

Find out more about how Microsoft Fabric, Power BI, and the Azure intelligent data platform can position your L&D efforts to take advantage of AI.

Data-Driven OKRs: Unlocking Google's Success Secrets with Microsoft Power BI

Despite the appeal of systems like Objectives and Key Results (OKRs), many companies struggle to implement the concept. Like many seemingly simple ideas, OKRs are not as obvious as they look. 

How hard can it be to set an objective, and then define some measurable key results? I mean, what could go wrong?

In his book, Measure What Matters, John Doerr describes the OKR system as he learnt it under Andy Grove at Intel. It’s a good read, and Doerr brings out an A-list of celebrity organizations to illustrate his point. Doerr describes how he introduced OKRs to Google, and how it was Google who greeted his OKR message with the most enthusiasm. In the Plex author, Steven Levy, says “Doerr had Google at metrics”. Not quite as romantic as Jerry Maguire, but we get the message. Levy goes on to say “OKRs were an elastic data driven apparatus for a free-wheeling data-worshipping enterprise”.

And there’s the kicker. Google was already a data driven company. OKRs didn’t change that, John Doerr just gave them a powerful management wrapper around their data driven culture. And they took the idea and knocked it out of the park.

But the very fact that Google were already a data driven company goes some way to explain why so many companies struggle with OKR’s or other objective and measurement systems such as the Balanced Scorecard. Data, analysis, and good communication lie at the heart of the concept. Which is exactly why you can’t “set and forget” an objective. 

It’s not that visionary leaders don’t believe in their objectives, or that their objectives are unimportant, more that they struggle to get good data, and to make the data work for them. And without good data and a solid way to communicate it, no one pays attention. 

To make things even harder, as soon as people are scattered in multiple locations, such as working from home, or on different continents, whiteboards and water cooler discussions break down. Remote Team meetings go some way to fill the gap, but the data and the communication must be rigorous and intentional, to quote Andy Grove. Without a good platform to help, it all becomes too difficult.

Fortunately, in the intervening years data technologies have become more powerful and easier to use. Microsoft Power BI can take data from a wide variety of sources to support all types of objectives. And powerful visuals can communicate progress in different ways, from summary scorecards to more graphical gauges and dials.  

Unlike specialist apps that support specific objectives, Power BI supports a wide range of data sources and data analysis. It’s this width and breadth of situations that make Power BI so impressive. Like Excel, it’s a tool that can tackle the simplest or the most complex of problems and do it very well.

The fact that Power BI Desktop can be downloaded for free, makes measuring and communicating key results more attainable for many more leaders. It may not be the end of their data journey, but it certainly could be the beginning.

Microsoft Power BI and the whiteboard

The room is noisy with people on the phone, while others are shouting, unaware of the racket they are making. There’s laughter, banter, and plenty of energy. 

I notice the large whiteboard on the wall. The writing is messy, but at the top there's each team’s sales goals written large and in red. The smaller numbers beneath are difficult to read because they keep getting rubbed out, so the new numbers can be written in. These are their results. 

This is a focused, albeit noisy, bunch of people. And they all have their eyes on the prize.

While I wait for my meeting to start, I stare at the whiteboard. I know the company is growing fast, and while they won’t win an award for neatness, I wonder how much of their success is due to their ability to focus their team on their objectives. 

This is the world of telesales a decade or two ago, in one hugely successful company. Since then, the world has changed at lot, but the ability to keep people focused on an important objective is just as powerful. 

Arguably a manager’s work has become a lot more complicated, including new challenges such as:

• Keeping everyone focused when people work from home and come into the office at different times.

• Creating objectives for complex projects that cannot be reduced to one number.

• Focusing people’s energy and ambition on work that contributes to objectives, even when everyone is working on different things.

Whilst some things have changed, other things remain the same. Effective managers still want to:

• Unite their team behind a common objective – including complex objectives.

• Keep people focused and engaged with the objective.

• Update personal and team results, so everyone can see and analyse progress. And see where their contribution fits in. 

• Encourage discussion about how to adjust when things are going well, or when you are falling behind.

What’s needed is an electronic whiteboard that is highly graphic, and that updates results without all that rubbing out. Oh, it lets managers to write as large as they want, in whatever colour they want. 

Power BI is a genuinely multi-purpose package that is indispensable once you get to know its capabilities. You could say it’s a bit like a whiteboard, only neater, and a lot more powerful. And easier to clean.

Microsoft Power BI is great at communicating objectives, the progress towards those objectives, and creating engagement. Why? Because it’s not really a whiteboard (you might have guessed that). It will connect to existing and new data sources. It will clean and transform data and join data together from different systems. You can create visuals that communicate what you are trying to achieve and show progress. Because just like that whiteboard on the wall, it needs to be at-a-glance easy to communicate what needs to be done. 

But here’s where the analogy breaks down. A whiteboard may be great at keeping a roomful of people focused, but Power BI can bring large disparate teams and departments together. Wherever in the world they are located. You can go from prototype to enterprise wide, and Power BI will grow with you. And you can use in-built AI capabilities to learn more about your successes and failures. Unlike a whiteboard where you start from scratch every day, Power BI allows you to generate and analyze data about what went well, and what didn’t go so well. This is powerful business intelligence, that is available to everyone.

If you’re trying to achieve something important with your team, Power BI could be just what you need to succeed. You can download it for free, and we provide in-depth data services to help you. Whether you want to clean your existing data, use several different data sources together, or model your data, we can help. We can also help you design dashboards. If you are curious, then get in touch for a no obligation chat. There’s nothing we like more than to help people succeed with what’s important to them.

The Microsoft Power BI KPI Visual in Practice

The KPI Visual is a summary that compares an actual value to a target value, together with the percentage variance. It displays in red when below target, and in green when above target. The background shows the trend of the actual value over time.

 

The KPI visual requires two values, a target and actual value, plus a time interval. These three values mean you have to be clear about how to:

•          Express your goal as a single number.
•          Measure on-going progress.
•          Have a system to record accurate data.

Although simple, these values go to the heart of a goal, and how you intend to achieve it.

Many different numbers may contribute to achieving the goal, but you must choose the one which best represents the target. Jim Collins would argue that being able to simplify complexity into a single number is a leadership skill that unifies people, and guides decisions. In his book, Good to Great, he talks of choosing the right economic denominator to drive profitability. The concept is the same as distilling a destination, or goal, into a single metric that can be tracked. Collins is not the only fan of headline numbers, John Doerr in Measure what Matters makes the case for clear goals with clear metrics. Marissa Mayer famously said, “It’s not a key result unless it has a number.”  

That so many great minds have felt the need to say anything at all indicates that this is neither obvious nor easy.

Having decided on what you will measure, the question of how to measure arises. Again, not obvious nor without its pitfalls. Can it be measured automatically, using IOT perhaps from a production line or a smart phone? If not, do people have to input the data, in which case how accurate will it be, and how onerous on those that have to do this work? Unless the process is made super easy, estimating, sloppy categorising, and all sorts of other shortcuts get taken. Including the final shortcut – not recording the data.

All of these things happen, all too commonly simply because people are already busy with the job they were hired to do.

Antoine de Saint-Exupery said, apparently, that “A goal without a plan is just a wish.” Many others have pointed out that “the road to hell is paved with good intentions.” Both pointing out that just setting a goal is insufficient, regardless of the talent or motivation of those setting the goal.

Hitting difficult goals is more about planning, monitoring, and adjusting actions accordingly. In other words, behind every serious goal is a system for monitoring progress. Once you have that, you have a very real possibility of putting meaningful numbers into the KPI visual.

The good news is that you may well have several systems that already contain the information you need. Most companies have data in databases and spreadsheets, some of which can be used to develop and monitor your KPIs. Or you may need to adapt or add a new system.

Power BI has enormous flexibility in allowing you to use data from various sources and manipulate it into something you can use. And once you’ve done that, you have a meaningful KPI that can be communicated to stakeholders.

If you’ve looked at the KPI visual and decided it’s not for you for any reason, think again. It’s simplicity is its power, and the hard work required to get the data it needs will be repaid many times over.

If you’d like to find out more about using your existing data to report on KPIs or other reporting, get in touch.