Alan B. Knox

Helping Adults Apply What They Learn

ATD developed the talent development framework and puzzle to help practitioners understand the different components of talent development and how organizations can build their own frameworks to address their unique needs. This blog series explores each component of the talent development framework, why it’s important to the field, and what resources ATD offers to practitioners who want to learn more.Evaluating Learning ImpactEvaluating learning impact involves the use of learning metrics and analytics to measure the effectiveness of learning solutions. In Utilization Focused Evaluation: The New Century Text, Michael Quinn Patton defines evaluation as “the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.” Evaluating learning impact has long been considered a daunting task. And, as a recent ATD study shows, only 35 percent of talent development professionals “reported that their organizations evaluated the business results of learning programs to any extent.” A different approach is needed. First talent development professionals who want to evaluate learning impact must know what specifically they want to measure and how they want to measure it. According to learning expert Patti Shank, the biggest mistake many make in evaluating the impact of learning is just testing recall. Just because people have learned something does not necessarily mean that they’ll apply it or understand its importance on the job. Real assessment comes from observation, providing feedback, and evaluating results. There are several models to evaluate learning, such as Kirkpatrick’s Four Levels and Phillips’ Evaluation Model. Kirkpatrick’s Four Levels is likely the most commonly known evaluation model, with a four-level program that follows a path of reaction, learning, behavior, and results. The fourth step is the most important, because the results show outcomes and overall effect of the learning in practice. Regardless of the evaluation strategy used, when collecting data, it is also important to use a variety of tools such as surveys, interviews, and analytics to get a full picture. To be competent at evaluating learning impact, the ASTD Competency Study (Updated now to ATD Talent Development Capability Model™)  identifies skills that talent development professionals need to exhibit:• Identify customer expectations.• Select appropriate strategies, research design, and measures.• Communicate and gain support for the evaluation plan.• Manage data collections.• Analyze and interpret data.• Apply learning analytics.• Make recommendations to aid decision making.Measuring learning impact encourages higher-quality work, resource prioritization and allocation, and accountability. Measurement data are invaluable to organizations that are strategically using talent development efforts to achieve growth and efficiency objectives. Competency in evaluating learning impact is imperative to becoming a real business partner in the organization.We’ve assembled some additional resources to explore:• Use analytics to improve your organization’s evaluation of learning. • Discover new ways of measuring learning. • Learn how improving performance management can help learning impact. • Measure learning using evaluation and not testing. • Consider a new way to look at Kirkpatrick’s four levels. • ATD Research: Evaluating Learning: Getting to Measurements That Matter•Evaluating the Impact of Training by Scott B. Parry• For more information on training delivery and other major components of talent development, check out the ASTD Handbook, 2nd Edition (Updated now to the TDBok™ Guide, 2nd Edition). • Take a deeper dive into evaluation with ATD’s Evaluating Learning Impact Certificate Program.

The ATD Talent Development Framework: Evaluating Learning Impact

This is the second post in a three-part series examining leadership.  (See post 1 and post 3.) In the present business climate, the leaders need to become champions of the “learning organization” as designers, teachers, and stewards (Senge 1990), rather than position themselves as the traditional charismatic hero (Steyrer 1998).  With that in mind, leading others requires constant learning and development (L&D) and designing of knowledge-creative workplaces that foster growth. Agile Skills Required Enter “design thinking.” Research reveals that design-driven companies, such as Apple, Coca-Cola, Ford, IBM, Intuit, Nike, P&G, Starbucks, Starwood, Target, Walt Disney, or Whirlpool, have outperformed America’s S&P 500 large companies by an extraordinary 228 percent over 10 years (DMI 2014). Likewise, being smart about agile working, agile workforce, and agile environments and cultures should be on every leader’s list of priorities. In fact, a 2014 CIPD survey of HR leaders and employees found that improving “leadership and management capability” (64 percent of respondents) and “learning and development” (54 percent of respondents) are top tactics to deliver organizational agility. Meanwhile, respondents in the CIPD research ranked “reward management” (15 percent of respondents) and “increasing workforce diversity” (17 percent of respondents) much lower. In the 2014 report, Change Agents: The Role of Organizational Learning in Change Management, ATD Research concurred that organizations and their leaders should be more adaptive to change. Indeed, the research study found that 61 percent of organizations experience three or more changes per year, but they are not prepared to manage that change. Only 17 percent of respondents rate their enterprises as highly effective at managing change, and a mere 30 percent of organizations have change management teams. The good news is that there is clear evidence that an adaptive, generative, and knowledge-productive learning and development (L&D) approach to leadership development can provide the infrastructure for numerous progressive methods designed to produce adaptive leaders. Coaching and self-managed learning are clear examples of this approach. Why Coaching Works Through my own research into leadership development, I have found that coaching is about building and sustaining high-quality links with others. Extremely good coaches energize, ask, listen, invite, and equip others to develop themselves and achieve their personal and professional best. How does this work? The coach energizes the relationship, while the coachee empowers it. As a result, the link re-energizes the coachee, so that he or she is better able to discern, ideate, transform, and deliver (Tkaczyk 2014). More importantly, numerous coaching skills can be applied routinely with clients, peers, or direct reports. They include setting the stage, bringing forth positive leadership presence, tracking information, identifying issues and key factors, playing back exchanges and checking-in, giving and receiving constructive feedback. In addition, one of the most important tools when working with others is the skill of positive inquiry (Tkaczyk 2014). Meanwhile, Stanford’s 2013 Executive Coaching Survey revealed that nearly 100 percent of CEOs enjoy the process of coaching and receiving leadership advice from professional coaches and outside advisors. Although CEOs are receptive to making changes based on these interventions, the Stanford study found that a majority (66 percent) do not receive leadership advice or coaching. To be sure, the benefits of coaching for developing leadership skills are tangible. ATD Research’s report, The Coaching Approach: A Key Tool for Successful Managers, concluded that, in reality, coaching improves communication (69 percent), raises engagement (65 percent), improves skills-to-performance transfer (63 percent), and stimulates productivity (61 percent). Becoming Self-Managed Learning Leaders As leaders grow, they need to become “L&D crafters” (Tkaczyk 2014). To maximize their leadership presence and learning agility, rooted in design thinking (Austin and Devin 2003, Martin 2009) and positive organizational scholarship (Cameron at el_._ 2003, Cameron and Spreitzer 2012), such L&D crafters may routinely use a “continuing developmental portfolio” comprised of two components: Continuing Executive Development Check-In: A disciplined L&D system, self-managed by an individual who constantly “checks in” with themselves by asking reflective questions regarding any critical and opportunistic incidents that happened in the workplace or outside, daily or in the past week.Design: A natural and drilled L&D mechanism that is self-guided by the leader, in which they constantly “design” their own L&D by asking developmental inquiries regarding what it is that they still want to learn and develop, and how exactly they will execute their development plan. International research conducted by the Learning Entrepreneurs project from September 2011 through September 2013 found that such self-managed learning produces personal gains: upgrades your skills and helps monitor your learning in an evidence-based way (91 percent) and boosts your credibility as a reflective and creative leader (90 percent). It also offers organizational benefits: unlocks HR potential (80 percent) and builds “knowledge-creative” workplaces (73 percent). Interestingly, participants in this global research project represented diverse geographies, including Africa, Europe, Latin America, Middle East, and North America. Still, more scientific research is needed, especially as Asia/Pacific leaders were not represented in the study (Tkaczyk 2014). For more on “self-coaching” as a self-managed leadership development method, see my August 2014 TD article, “Daily Check-Ins Stimulate Self-Improvement.” Finally, other approaches for developing leaders for future success can include:cognitive or relational job re-designjob enlargementsposition transfersjob rotationssabbaticalsexternships—professional development through a full-time temporary position at another organizationself-assessment inventories360-degree feedbackpsychological profiling and strengths-findingleaderless group discussions in which a team of multiple leaders is assigned a complex challenge and must work together to work it out within a certain time periodacting and storytellingconcept mappingrepertory gridscritical incident technique, as utilized in job analysis,mentoring·one-on-one Personal Management Interview (PMI) programs, namely purpose-finding and role-negotiation sessions—approximately 60-120 minute each, and preferably twice per month, depending on life cycle rhythm of work. The final installment in this leadership series will address future vistas in leadership.References ATD Research, 2014. Change Agents: The Role of Organizational Learning in Change Management, Association for Talent Development (ATD), Alexandria, VA.  ATD Research, 2014. The Coaching Approach: The Key Tool for Successful Managers, Association for Talent Development (ATD), Alexandria, VA.  Austin, R. and Devin, L., 2003. Artful Making: What Managers Need to Know About How Artists Work, Prentice Hall, Upper Saddle River, NJ.  Cameron, K., Dutton, J., and Quinn, R. (eds.), 2003. Positive Organizational Scholarship: Foundations of a New Discipline, Berrett-Koehler, San Francisco, CA.  Cameron, K. and Spreitzer, G. (eds.), 2012. Oxford Handbook of Positive Organizational Scholarship, Oxford University Press, New York, NY.  CIPD, 2014. HR: Getting Smart About Agile Working, Chartered Institute of Personnel and Development, London.  DMI, 2014. “Design-Driven Companies Outperform S&P by 228% Over Ten Years—The DMI Design Value Index,” Dialog, Design Management Institute (DMI), Boston, MA.  Martin, R., 2009. The Design of Business: Why Design Thinking Is the Next Competitive Advantage, Harvard Business School Press, Boston, MA.  Senge, P., 1990. The Fifth Discipline: The Art and Practice of the Learning Organization, Century Business, Random House, London.  Stanford, 2013. 2013 Executive Coaching Survey, Stanford University/The Miles Group.  Steyrer, J., 1998. “Charisma and the Archetypes of Leadership,” Organization Studies, Vol. 19, No. 5, pp. 807-828.  Tkaczyk, B., 2014. “Critical Clarification on Coaching,” International Coach Federation (ICF).   Tkaczyk, B., 2014. “Positive Inquiry Skills: Crafting Vital Coaching Questions," Association for Talent Development (ATD).  Tkaczyk, B., 2014. “Crafting Continuing Learning and Development: A Positive Design Tool for Leadership Development,” Development and Learning in Organizations, Vol. 28, No. 4, pp. 5-8.  Tkaczyk, B., 2014. “Daily Check-Ins Stimulate Self-Improvement,” TD, Vol. 68 No. 8, pp. 72-73.