MEASURING IMPACT IN LEADERSHIP DEVELOPMENT PROGRAMS
The compelling evidence for measuring mindset and behavioural change in face-to-face leadership development programs
How metrics rooted in science helped prove that a highly-customised leadership development program really did exceed expectations
What makes a successful leadership program? The energy in the room, the chemistry amongst the delegates, the relevance of the material delivered and even the charisma of the facilitators all play a part in reaching a high NPS score and positive feedback at the end of the day.
In a changing world of work, when many organisations have invested in online learning platforms, L&D budgets for face-to-face learning are being seriously squeezed. Often senior management, many of whom have attended leadership programs themselves, are cynical about the value of face-to-face learning. Today’s leadership programs need to prove that they are providing relevant knowledge and creating lasting changes in attitudes and behaviours long after the feedback sheets have been handed in and the end-of-program group photo is shared on social media.
When we were approached by Capgemini – world leaders in the IT services industry – to discuss how we could help them further develop growing talent in one of their regions, we knew that we needed to invest as much time and care into designing how its impact would be measured as we did into designing the leadership program itself. This would be a pilot program and the results key to demonstrating that lasting change in behaviours through face-to-face mindset and leadership competency training is not only possible, but recommendable.
Our brief was clear, we would work with 24 leaders from Capgemini’s cloud infrastructure business in India to help them develop the global mindsets that would prepare them for VP-level positions and make them stronger candidates for promotion.
Pune, India: Photo by Atharva Tulsi/ Unsplash
Breaking the Cultural Glass Ceiling
India is Capgemini’s largest block of “off-shore” employees – 100,000 people and growing leadership talent. These leaders needed help breaking through the “cultural glass ceiling” and preparing them to deliver on the organisation’s goal to excel as a tier one tech consulting firm.
Capgemini’s culture is strongly rooted in Europe. We needed to harness the positive traits characteristic of Indian business culture such as their strong work ethic, their attentiveness to the client and their ability to learn new things quickly. To thrive in a global leadership role, they would need to become more comfortable learning to say “no” and understand that they were not letting people down but managing expectations. Likewise, learning to lead in situations where a “command and control” style is not appropriate or learning to turn outcomes into a set of questions to empower others to perform would make them more effective. They needed help acquiring the behaviours and the emotional intelligence to drive leadership performance and personal excellence that would translate to Capgemini’s global environment.
Capgemini’s Powerful Leadership Program
During the 5-day, face-to-face “Powerful Leadership Program”, the delegates would explore and develop personal peak performance strategies and beliefs.
They would learn how to influence and build trust and how to empower others to work more effectively. They would gain clarity around personal values and priorities as well as acquiring a framework for success based on personal responsibility and accountability.
The program is designed to make a profound impact on leadership and culture with a strong focus on driving performance. The foundation for the program was rooted in the Cultural Iceberg Model (E. Hall,1976) and in Open Water’s experienced approach to changing individuals’ thinking patterns and aligning their personal values and beliefs through increased emotional intelligence.
Working closely in partnership with the L&D leaders responsible for the initiative was critical. Capgemini have a well-documented leadership competency framework with six pillars or ‘Leadership Dimensions’.
These include the ability to build profitable businesses, developing others, and embodying the values of the organisation. We mapped these leadership dimensions to show how they related to the program outcomes and the methodologies used. This helped demonstrate the relevance of the program to the organisation and gave us an understanding of what behaviours and attitudes were required.
Creating an Impact Map
Open Water’s methodology starts with client diagnostics that enable us to build out an impact map. We start by plotting the organisational goals against the program goals to make sure we are aligned both in content and context. An important part of Capgemini’s vision is ‘to excel as a tier one global tech consultancy’.
We built out a list of organisational goals that our 24 leaders would need to accomplish in order to make it happen. We were then able to identify the job-critical behaviours they would require in order to achieve those goals. Having pinned down the desired behaviours, we were able to determine the program’s specific learning outcomes – and the desired impact.
As any L&D leader will know, it’s all very well designing a program that appears to achieve the desired outcomes. The real test is the extent to which a learning program can actually change behaviours and how long and deep those changes are embedded. The bedrock to Open Water’s approach is the use of highly relevant and experiential methodologies and frameworks to make that kind of change happen.
Empowering Continued Learning
Key to this, is the ability to hand over ownership of continued learning to the participants themselves. We wanted to have them interacting with each other in order to deepen the learning and further embed new behaviours. Using Open Water’s social learning platform OWL, we custom-designed a continuous-learning offering that would kickstart the learning process before the face-to-face program and continue to engage and assist learning, long after the face-to-face program had ended. To this end, we created five additional modules with extra content, discussions, exercises and assignments on a customised platform. We also used webinars to bring delegates together at different points in the process.
Measuring Success
How much can you actually change an individual’s behaviour in order to drive a cultural change? To measure the success of the program we used a highly validated self-assessment process – more appropriate than a 360 assessment given the learning’s heavy focus on personal beliefs and values, self-awareness and self- management.
We used the learning outcomes derived from the impact map to develop the questions we would use to measure cultural transformation in terms of knowledge, attitude and behaviours (KAB).
The evaluation process has three stages:
Pre-test: where we evaluate knowledge, attitudes and behaviour before the program in order to have baseline results to compare with future results.
Post-test: where we evaluate knowledge and attitude (or ‘learning’) immediately after completion of the face-to- face program.
Job-test: where we evaluate knowledge, attitude and behaviours three months after program completion in order to measure the impact and how well the learning is being applied.
The system that Open Water uses to measure program success has been developed over 25 years. It is a robust, highly-validated tool, resulting in 90% accuracy.
As a self-assessment tool it is efficient and has far greater accuracy thanks to the inclusion of critical incidence thinking as well as specific hierarchical taxonomies for the most accurate measurement of knowledge and attitude learning.
Powerful Results
So what of our pilot program results? With the pre-test scores providing the baseline, our participants exceeded the benchmark targets scoring a high 87% in ‘Learning’ of Knowledge and Attitude in the post-test results. This demonstrates an impressive positive shift of 42% Learning directly after the program. The results also showed that the existing learning gap had closed by 77%, which was also above the targets we had set.
In assessing how much new knowledge and attitude was gained and sustained during the Powerful Leadership program, results were again above target, with an impressive 91% of new knowledge and attitude being retained three months after the program ended.
However, the most significant result, and perhaps the most critical one when it comes to assessing the success of the program, is the development of participants’ job- critical behaviours. These increased from 31% to 77%. This uplift of 46% indicates just how relevant and effective the program was, resulting in a major move in desired job-critical behaviours, within just three months.
Making the Case
Given the nature of the program’s content – mindset & behavioural change, shifts in beliefs and perspectives – these results have far exceeded expectations and create compelling evidence of what can be achieved with a well- designed, well-measured program.
Naturally, there are other factors external to the learning and development component that will help continue to embed the change and ensure leaders are more effective. Organisations need to ensure that KPIs and rewards are incentivising the right kind of individual behaviours and attitudes; coaching has to become intrinsic to line management and every effort made to support the cultural transformation.
Ultimately, if organisations truly want to make lasting change happen, they need to understand that results are affected by what people do, and what they do is affected by emotion and belief. And these results can, and must be measured.
Ben Shoshan
DOWNLOAD THE PDF ARTICLE HERE