Part 3 – Leadership Development Evaluations And Limitations
There are a lot of question marks regarding development measurement. Perhaps there is a mismatch between the actual outcome of development and organizational expectations, or perhaps the knowledge of appropriate measurement is not well-known yet. Either way, there is a clear divide between what science knows and what is expected by many organizational leaders regarding development efforts. This article covers only a few of the many limitations of development evaluations, but it helps to demonstrate the complexity of the matter and provides insight into appropriate development expectations and outcomes.
The struggle to properly evaluate development efforts presents a clear limitation for organizations seeking quantitative data. Even with effective instruments, the proper evaluations of leadership development can be difficult. This difficulty, mixed with contorted, biased, or false perceptions regarding expected outcomes, along with a heavy dose of circular reasoning, all contribute to various misconceptions about development, and they often impede leadership development’s greater adoption. These complexities may also create a false perception that leadership development efforts have failed when they have not.
Many measurements of development can be unreliable. Decision-makers need better contexts to accurately gauge their return on investment regarding their development efforts. However, aside from the complexities of development measurement, considerations must also include the variety of programs available, issues that result from organizational adherence to the status quo, and the lack of time often needed to measure. While seemingly unrelated, each of these points is interconnected and hinders accurate perceptions of development results.
Differences In Approach
The problem begins with the idea that different practitioners approach leadership and organizational development differently. Research by Parker and Carroll acknowledges the ‘crisis of leadership’ and ponders the lack of consensus regarding leadership constructs, the ongoing debate and controversy within the discipline of leadership regarding the various philosophies that underpin leadership development, and the questions regarding leadership development’s best practices (2009). Indeed, there is a lack of consensus regarding many aspects of leadership and its development. This variety complicates measurement standards. The good news is that despite the variety of available programs, the data continues to demonstrate that leadership development works when conducted by leadership-educated professionals (Robertson, 2022).
Circular Reasoning, Fear, and the Status Quo
Examining any development effort without also examining organizational norms leads to a contortion of perception. Specifically, if a true understanding of development effectiveness is to be found, evaluators must examine development efforts, compare that to development outcomes, and then contrast both to any potential organizational adherence to the status quo. Having this lens provides an opportunity to see the greater truth. However, that truth is often masked by circular reasoning.
For example, many organizations find comfort in the status quo. This comfort is often rooted in predictability. However, some organizations understand that challenging the status quo is essential once they recognize that the status quo holds the organization back. Leadership development often seeks to challenge the status quo. Unfortunately, workers rarely challenge the status quo at all (Llopis, 2017). Reasons vary, but nearly half of workers say they regularly feel the need to conform to entrenched and sometimes toxic organizational norms (Gino, 2020).
Regardless of development status, “feeling the need to conform” alludes to an organizational culture of fear that pressures employees to adhere to the status quo. This is an important consideration when attempting to measure the success or failure of any development program. To be clear, it is not a leadership development failure when organizations pressure conformity to toxic organizational norms. Instead, it is an organizational failure. However, some organizations ignore this truth and double down on compliance and fear tactics that only further solidify the status quo that they say they wish to change. Harvard Business School professor Francesca Gino insists that fear will not foster the shift organizations need and that if workers fear the aftermath of pointing out issues or offering solutions to organizational problems, the organization will suffer a sharp decline in engagement, productivity, and innovation (2020).
Almost paradoxically, and if an organization truly desires change, professional leadership development investments may help. However, organizational leaders must understand that learning to challenge the status quo via leadership development would be of little value for organizations demanding adherence to the status quo. It is the power of the purse. In other words, for organizations that prefer the status quo, any development program or evaluation is likely going to be a waste of time and money because workers will likely choose the check over the change. If the organization is willing to change, the organization will need to adopt a new mindset and development culture and then develop employees accordingly.
Measurement Difficulties
Another common reason for the perception of leadership development failure may be an inability to reliably measure results (Hickney, 2021). Again, another circular issue has presented itself. It is true that traditional measurements of newer development models may suggest a development failure. However, this supposed failure usually resides in a contorted perspective and faulty measurement, not a program failure.
Leadership and its development often emphasize elements of vision, challenging the status quo, and shaping or chasing possibilities for the future. These are sometimes referred to as leadership intangibles. Such intangible elements of leadership can be challenging, if not impossible, to measure. They also take time to alter. This presents a particular problem for organizations using outdated instruments to measure development outcomes and for those expecting an immediate return. However, the provided list are just a few of the many intangible elements covered in newer development programs, and organizations must understand that exploring intangibles has become common in both leadership science and practice.
While these elements may be challenging to measure, this shift in direction is not bad, and scholars have begun to support the transition. Researchers such as Kennedy, Carroll, and Francoeur note the industry’s shift to focusing on the intangible elements surrounding mindset as a good thing (2013). For example, a common theme in many newer development programs is the emphasis on developing an innovation and growth mindset. The shift makes sense when considering the organizations that profess a desire for innovation and growth. Unfortunately, a qualitative approach is currently the only hope of measuring such a mindset.
Of course, a related problem resides in organizations unprepared for innovation and growth. After all, the predictability of the status quo may be the antithesis of innovative growth in most situations. Once again, if the status quo of the organization does not encourage, adopt, or appreciate the new innovation and growth mindset, and if the organization demands adherence to the status quo, the investment in leadership development is pointless, and the status quo will continue.
Organizational leaders must understand three things regarding this topic:
- The science demonstrates that the intangible elements of leadership, such as mindset and behavior, contribute to high-performing organizations
- Those tasked with evaluations must accept that thinking, feeling, and behavior are measured intuitively, not quantitatively
- Substantial quantitative measurements of these skills might never be possible
(Berman, 2018; Morse, 2013; Sheryn, 2020).
Experts such as Watkins, Lysø, and deMarrais echo the preceding by suggesting that as leadership development programs become more unceremonious and experimental, traditional evaluation methods are less likely to measure the outcomes accurately (2011). Accordingly, many traditional and quantitative assessments will result in expensive failures due to the immeasurable and intangible outcomes of the various development programs (Kennedy et al., 2013). The warning here is that organizations unfamiliar with these truths may affix blame incorrectly while wasting both time and money in their attempts to either develop or measure.
Time to Measure
Time is another concern worth talking about. While innovations in measurement are likely necessary, the investment of time is critical if any meaningful measurement is to be had. Leadership development is a cause-and-effect process where the qualitative benefits, such as mindset and behavioral change, will impact both the student and the organization over time (Hayward, 2011). The change or ultimate effect of such development does not happen overnight, and instant gratification is generally not possible. Therefore, evaluators must measure accordingly (longitudinal) to see the result for what it truly is or may become. If they do not, they will not have a clear picture of the result.
The Organization as a Whole
Employees are impacted and influenced by their organization. Accordingly, evaluators must assess leadership development as an interconnected personal and organizational learning process. Organizational contexts are crucial to individual development evaluations, and the literature is beginning to suggest that a misunderstanding or misconception regarding an individual’s interconnectedness with the organization they serve can result in ambiguous and faulty conclusions regarding the development effort (Hanson, 2013). Therefore, a systemic element must always be considered during any such evaluation. Of course, such insight presents a potential issue for toxic or entrenched organizations that are unwilling to admit to their cultural issues.
Organizational cultures can be complicated, and so too can the leadership be within such cultures. Complexity theory helps explain the issue, viewing leadership as a dynamic and evolving series of complex interactions between individuals within the organization that ultimately shapes organizational strategies, culture, and power structures (Ardichvili & Manderscheid, 2008). However, the preceding also demonstrates that leadership, and its development, are not simple constructs to be shoehorned into simplistic or outdated quantitative evaluations. New methods are necessary but so is expertise. In their study on leadership development evaluations, Jarvis, Gulati, McCririck, and Simpson point to an emerging and growing body of literature that suggests that evaluations should look beyond traditional quantitative materials to understand the broader picture by employing a variety of methods to examine the answers to a more contextually informed set of questions (2013).
A successful example of the preceding may be found in a recent study titled, A Study of the Relationship Between Leadership Development and Employee Job Satisfaction and Engagement. The study’s unique approach measured the outcomes of professional leadership development using a variety of instruments while gaining longitudinal insight by including a time element. The study compared employees who had received development several years back with those who recently received development and those without any development. The results suggested that not only did employee engagement increase with leadership development, but it generally remained or improved over time. The study supports previous findings suggesting a benefit of professional leadership development as far as two years or more post-development (Hayward, 2011). However, such insights would not be possible when attempting to use simplified snapshots or outdated and narrowly-focused quantitative instruments. Furthermore, such evaluations would likely not be possible for those with only a loose understanding of these truths.
Emerging Development Models
Another consideration might be the pressure placed on development practitioners by organizations to address the various modern organizational demands. Edwards and Turnbull point to gaps between leadership development literature and practice and suggest that leadership development has had to innovate creative ways to address organizational culture’s complexity (2013). There is little doubt that such pressures vary by industry and location and have only increased in recent years. Therefore, it becomes essential that evaluators consider the pressures placed on leadership development practitioners to experiment with new methods and that experimental approaches will further complicate measurement. Similarly, and as a result, it should be understood that not all leadership development programs are going to be equal, and new leadership development practices emerge all of the time (Ardichvili et al., 2016). Moreover, there is no doubt that such pressures will continue to force the evolution of the leadership discipline, which will only add to the complexities of measurement.
To be clear, outdated or traditional leadership development evaluations will likely not capture the learning’s broad-based outcomes or objective interpretations of newer mindset-oriented programs. Therefore, evaluators may want to consider a more qualitative approach in determining the effectiveness of such programs because these types of evaluations can capture more of the missing pieces by factoring in organizational culture, politics, and training contexts, including training design and delivery (Ardichvili et al., 2016; Watkins et al., 2011). However, as previously mentioned, such evaluation would likely require a considerable amount of time because the suggested benefits of development can happen as far as two to three years later as both individual behaviors and organizational cultures begin to shift (Hayward, 2011).
Contortions and Confusions
Of course, any evaluation would be highly flawed if the premise of the evaluation is unsound. For example, if the organization or evaluator is unclear or confused about differences between human resources, management, and leadership, is unclear about how to measure the intangible outcomes of the program, is unaware of the root of organizational toxicity, or is unaware of the importance of the practitioner’s expertise, then evaluations of that program’s effectiveness must be questioned or deemed unreliable. At the same time, evaluators must also understand that hard to measure does not necessarily equate to failure.
For leadership programs taught by leadership-trained and educated professionals, Martineau and Patterson suggest that a proper evaluation of a leader’s development is rooted in gaining a strong understanding of the context of the organization or community, identifying and specifying the desired results or outcomes, determining the individual competencies and collective capabilities needed, and exploring all possible solutions (Velsor et al., 2010). These points can provide a deeper understanding of what occurred during the development process. Furthermore, they suggest that evaluators must understand the organization or community’s context, what the organization does well (or does not do well), and consider the organization’s unique value to employees (Velsor et al., 2010).
Conclusion
As demonstrated, the topic of development evaluations can be convoluted. Frankly, it is a confusing mess that leadership scholars are proactivity trying to sort out. The fact that many of the issues are circular and connected makes this task more difficult. Alarmingly, this article only discussed a few of the many measurement issues that are currently known.
The key takeaway of this article might be that, despite the difficulty of measurement, the data continues to demonstrate that leadership development works (Robertson, 2022). Therefore, evaluators should be patient and measure with a level of intuition. If a measurement is absolutely essential, then evaluators should consider using a variety of instruments that capture the qualitative aspects of the process because most traditional quantitative instruments are substantially less effective at capturing the intangibles of leadership development. However, just because one cannot currently measure an intangible does not mean that such intangibles are not vitally important to either the employee or the organization, and it does not mean that a failure has occurred.
With all that said, the inability to effectively measure development efforts presents a strong and legitimate question of how an organization can possibly trust newer development efforts. There is a way. However, gaining the necessary trust requires a better understanding of the leadership industry and those providing development services. A focus on the development practitioner is essential. In Part 4 of A Leadership Development Reality Check, the research demonstrates why choosing the wrong practitioner could be detrimental to both the individual and the organization and explains what organizations can do to increase the odds of having a positive leadership development experience.
_________________________
The Articles in this Series:
- Part 1: Organizational Development Needs
- Part 2: The Reality of Leadership Development
- Part 3: Leadership Development Evaluations and Limitations
- Part 4: The Right Trainer & The Novice Factor
- Part 5: Management vs. Leadership
- Part 6: Organizational Culture
_________________________
Use of this work is permitted with proper citation.
Acknowledgments & Actions
- Thank you for examining this research. Be sure to share if you find value in it, and be sure to check out the other parts of this series.
- This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
References
Ardichvili, A., & Manderscheid, S. (2008). Emerging practices in leadership development: An introduction. Advances in Developing Human Resources, 10(5), 619–631. https://doi.org/10.1177/1523422308321718
Ardichvili, A., Natt och Dag, K., & Manderscheid, S. (2016). Leadership development: Current and emerging models and practices. Advances in Developing Human Resources, 18(3), 275–285. https://doi.org/10.1177/1523422316645506
Berman, G. J. (2018). Measuring behavior across scales. BMC Biology, 16, 23. https://doi.org/10.1186/s12915-018-0494-7
Edwards, G., & Turnbull, S. (2013). A cultural approach to evaluating leadership development. Advances in Developing Human Resources, 15(1), 46–60. https://doi.org/10.1177/1523422312467144
Edwards, G., & Turnbull, S. (2013). Special issue on new paradigms in evaluating leadership development. Advances in Developing Human Resources, 15(1), 3–9. https://doi.org/10.1177/1523422312467147
Gino, F. (2020). Rebel talent: Why it pays to break the rules at work and in life. William Morrow and Company.
Hanson, B. (2013). The leadership development interface: Aligning leaders and organizations toward more effective leadership learning. Advances in Developing Human Resources, 15(1), 106–120. https://doi.org/10.1177/1523422312465853
Hayward, S. (2011). Connecting leadership development to bottom line benefits. Strategic HR Review, 10(1), 28–34. https://doi.org/10.1108/14754391111091788
Hickney, K. (2021). This is why training programs are failing your company. BetterUp. http://www.betterup.com/en-us/about-us/blog/this-is-why-training-programs-are-failing-your-company
Jarvis, C., Gulati, A., McCririck, V., & Simpson, P. (2013). Leadership matters: Tensions in evaluating leadership development. Advances in Developing Human Resources, 15(1), 27–45. https://doi.org/10.1177/1523422312467138
Kennedy, F., Carroll, B., & Francoeur, J. (2013). Mindset not skill set: Evaluating in new paradigms of leadership development. Advances in Developing Human Resources, 15(1), 10–26. https://doi.org/10.1177/1523422312466835
Llopis, G. (2017, August 12). 5 reasons leaders are afraid to challenge the status quo. Forbes. https://www.forbes.com/sites/glennllopis/2017/08/12/5-reasons-leaders-are-afraid-to-challenge-the-status-quo/?sh=f63f6626fe53
Morse, B. J. (2013). Measuring the stuff of thought: Psychology and its discontents. Bridgewater Review, 32(2), 4–7. http://vc.bridgew.edu/br_rev/vol32/iss2/4
Parker, P., & Carroll, B. (2009). Leadership development: Insights from a careers perspective. Leadership, 5(2), 261–283. https://doi.org/10.1177/1742715009102940
Robertson, D. M. (2022, May 17). The reality of Leadership Development. LinkedIn. Retrieved May 22, 2022, from https://www.linkedin.com/pulse/reality-leadership-development-dr-david-m-robertson-msl/
Sheryn, C. (2020, September). Measuring the immeasurable – metrics and measurement for a continuous improvement culture. PA. https://www.paconsulting.com/insights/measuring-the-immeasurable–metrics-and-measurement-for-a-continuous-improvement-culture/
Velsor, E. V., McCauley, C., & Ruderman, M. (2010). The Center for Creative Leadership handbook of leadership development. (3rd ed.). Jossey-Bass.
Watkins, K. E., Lysø, I. H., & deMarrais, K. (2011). Evaluating executive leadership programs: A theory of change approach. Advances in Developing Human Resources, 13(2), 208–239. https://doi.org/10.1177/1523422311415643