ATD Footer Logo

Alan B. Knox

Helping Adults Apply What They Learn

ATD developed the talent development framework and puzzle to help practitioners understand the different components of talent development and how organizations can build their own frameworks to address their unique needs. This blog series explores each component of the talent development framework, why it’s important to the field, and what resources ATD offers to practitioners who want to learn more.Evaluating Learning ImpactEvaluating learning impact involves the use of learning metrics and analytics to measure the effectiveness of learning solutions. In Utilization Focused Evaluation: The New Century Text, Michael Quinn Patton defines evaluation as “the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.” Evaluating learning impact has long been considered a daunting task. And, as a recent ATD study shows, only 35 percent of talent development professionals “reported that their organizations evaluated the business results of learning programs to any extent.” A different approach is needed. First talent development professionals who want to evaluate learning impact must know what specifically they want to measure and how they want to measure it. According to learning expert Patti Shank, the biggest mistake many make in evaluating the impact of learning is just testing recall. Just because people have learned something does not necessarily mean that they’ll apply it or understand its importance on the job. Real assessment comes from observation, providing feedback, and evaluating results. There are several models to evaluate learning, such as Kirkpatrick’s Four Levels and Phillips’ Evaluation Model. Kirkpatrick’s Four Levels is likely the most commonly known evaluation model, with a four-level program that follows a path of reaction, learning, behavior, and results. The fourth step is the most important, because the results show outcomes and overall effect of the learning in practice. Regardless of the evaluation strategy used, when collecting data, it is also important to use a variety of tools such as surveys, interviews, and analytics to get a full picture. To be competent at evaluating learning impact, the ASTD Competency Study (Updated now to ATD Talent Development Capability Model™)  identifies skills that talent development professionals need to exhibit:• Identify customer expectations.• Select appropriate strategies, research design, and measures.• Communicate and gain support for the evaluation plan.• Manage data collections.• Analyze and interpret data.• Apply learning analytics.• Make recommendations to aid decision making.Measuring learning impact encourages higher-quality work, resource prioritization and allocation, and accountability. Measurement data are invaluable to organizations that are strategically using talent development efforts to achieve growth and efficiency objectives. Competency in evaluating learning impact is imperative to becoming a real business partner in the organization.We’ve assembled some additional resources to explore:• Use analytics to improve your organization’s evaluation of learning. • Discover new ways of measuring learning. • Learn how improving performance management can help learning impact. • Measure learning using evaluation and not testing. • Consider a new way to look at Kirkpatrick’s four levels. • ATD Research: Evaluating Learning: Getting to Measurements That Matter•Evaluating the Impact of Training by Scott B. Parry• For more information on training delivery and other major components of talent development, check out the ASTD Handbook, 2nd Edition (Updated now to the TDBok™ Guide, 2nd Edition). • Take a deeper dive into evaluation with ATD’s Evaluating Learning Impact Certificate Program.

The ATD Talent Development Framework: Evaluating Learning Impact

This is the second post in a three-part series examining leadership.  (See post 1 and post 3.) In the present business climate, the leaders need to become champions of the “learning organization” as designers, teachers, and stewards (Senge 1990), rather than position themselves as the traditional charismatic hero (Steyrer 1998).  With that in mind, leading others requires constant learning and development (L&D) and designing of knowledge-creative workplaces that foster growth. Agile Skills Required Enter “design thinking.” Research reveals that design-driven companies, such as Apple, Coca-Cola, Ford, IBM, Intuit, Nike, P&G, Starbucks, Starwood, Target, Walt Disney, or Whirlpool, have outperformed America’s S&P 500 large companies by an extraordinary 228 percent over 10 years (DMI 2014). Likewise, being smart about agile working, agile workforce, and agile environments and cultures should be on every leader’s list of priorities. In fact, a 2014 CIPD survey of HR leaders and employees found that improving “leadership and management capability” (64 percent of respondents) and “learning and development” (54 percent of respondents) are top tactics to deliver organizational agility. Meanwhile, respondents in the CIPD research ranked “reward management” (15 percent of respondents) and “increasing workforce diversity” (17 percent of respondents) much lower. In the 2014 report, Change Agents: The Role of Organizational Learning in Change Management, ATD Research concurred that organizations and their leaders should be more adaptive to change. Indeed, the research study found that 61 percent of organizations experience three or more changes per year, but they are not prepared to manage that change. Only 17 percent of respondents rate their enterprises as highly effective at managing change, and a mere 30 percent of organizations have change management teams. The good news is that there is clear evidence that an adaptive, generative, and knowledge-productive learning and development (L&D) approach to leadership development can provide the infrastructure for numerous progressive methods designed to produce adaptive leaders. Coaching and self-managed learning are clear examples of this approach. Why Coaching Works Through my own research into leadership development, I have found that coaching is about building and sustaining high-quality links with others. Extremely good coaches energize, ask, listen, invite, and equip others to develop themselves and achieve their personal and professional best. How does this work? The coach energizes the relationship, while the coachee empowers it. As a result, the link re-energizes the coachee, so that he or she is better able to discern, ideate, transform, and deliver (Tkaczyk 2014). More importantly, numerous coaching skills can be applied routinely with clients, peers, or direct reports. They include setting the stage, bringing forth positive leadership presence, tracking information, identifying issues and key factors, playing back exchanges and checking-in, giving and receiving constructive feedback. In addition, one of the most important tools when working with others is the skill of positive inquiry (Tkaczyk 2014). Meanwhile, Stanford’s 2013 Executive Coaching Survey revealed that nearly 100 percent of CEOs enjoy the process of coaching and receiving leadership advice from professional coaches and outside advisors. Although CEOs are receptive to making changes based on these interventions, the Stanford study found that a majority (66 percent) do not receive leadership advice or coaching. To be sure, the benefits of coaching for developing leadership skills are tangible. ATD Research’s report, The Coaching Approach: A Key Tool for Successful Managers, concluded that, in reality, coaching improves communication (69 percent), raises engagement (65 percent), improves skills-to-performance transfer (63 percent), and stimulates productivity (61 percent). Becoming Self-Managed Learning Leaders As leaders grow, they need to become “L&D crafters” (Tkaczyk 2014). To maximize their leadership presence and learning agility, rooted in design thinking (Austin and Devin 2003, Martin 2009) and positive organizational scholarship (Cameron at el_._ 2003, Cameron and Spreitzer 2012), such L&D crafters may routinely use a “continuing developmental portfolio” comprised of two components: Continuing Executive Development Check-In: A disciplined L&D system, self-managed by an individual who constantly “checks in” with themselves by asking reflective questions regarding any critical and opportunistic incidents that happened in the workplace or outside, daily or in the past week.Design: A natural and drilled L&D mechanism that is self-guided by the leader, in which they constantly “design” their own L&D by asking developmental inquiries regarding what it is that they still want to learn and develop, and how exactly they will execute their development plan. International research conducted by the Learning Entrepreneurs project from September 2011 through September 2013 found that such self-managed learning produces personal gains: upgrades your skills and helps monitor your learning in an evidence-based way (91 percent) and boosts your credibility as a reflective and creative leader (90 percent). It also offers organizational benefits: unlocks HR potential (80 percent) and builds “knowledge-creative” workplaces (73 percent). Interestingly, participants in this global research project represented diverse geographies, including Africa, Europe, Latin America, Middle East, and North America. Still, more scientific research is needed, especially as Asia/Pacific leaders were not represented in the study (Tkaczyk 2014). For more on “self-coaching” as a self-managed leadership development method, see my August 2014 TD article, “Daily Check-Ins Stimulate Self-Improvement.” Finally, other approaches for developing leaders for future success can include:cognitive or relational job re-designjob enlargementsposition transfersjob rotationssabbaticalsexternships—professional development through a full-time temporary position at another organizationself-assessment inventories360-degree feedbackpsychological profiling and strengths-findingleaderless group discussions in which a team of multiple leaders is assigned a complex challenge and must work together to work it out within a certain time periodacting and storytellingconcept mappingrepertory gridscritical incident technique, as utilized in job analysis,mentoring·one-on-one Personal Management Interview (PMI) programs, namely purpose-finding and role-negotiation sessions—approximately 60-120 minute each, and preferably twice per month, depending on life cycle rhythm of work. The final installment in this leadership series will address future vistas in leadership.References ATD Research, 2014. Change Agents: The Role of Organizational Learning in Change Management, Association for Talent Development (ATD), Alexandria, VA.  ATD Research, 2014. The Coaching Approach: The Key Tool for Successful Managers, Association for Talent Development (ATD), Alexandria, VA.  Austin, R. and Devin, L., 2003. Artful Making: What Managers Need to Know About How Artists Work, Prentice Hall, Upper Saddle River, NJ.  Cameron, K., Dutton, J., and Quinn, R. (eds.), 2003. Positive Organizational Scholarship: Foundations of a New Discipline, Berrett-Koehler, San Francisco, CA.  Cameron, K. and Spreitzer, G. (eds.), 2012. Oxford Handbook of Positive Organizational Scholarship, Oxford University Press, New York, NY.  CIPD, 2014. HR: Getting Smart About Agile Working, Chartered Institute of Personnel and Development, London.  DMI, 2014. “Design-Driven Companies Outperform S&P by 228% Over Ten Years—The DMI Design Value Index,” Dialog, Design Management Institute (DMI), Boston, MA.  Martin, R., 2009. The Design of Business: Why Design Thinking Is the Next Competitive Advantage, Harvard Business School Press, Boston, MA.  Senge, P., 1990. The Fifth Discipline: The Art and Practice of the Learning Organization, Century Business, Random House, London.  Stanford, 2013. 2013 Executive Coaching Survey, Stanford University/The Miles Group.  Steyrer, J., 1998. “Charisma and the Archetypes of Leadership,” Organization Studies, Vol. 19, No. 5, pp. 807-828.  Tkaczyk, B., 2014. “Critical Clarification on Coaching,” International Coach Federation (ICF).   Tkaczyk, B., 2014. “Positive Inquiry Skills: Crafting Vital Coaching Questions," Association for Talent Development (ATD).  Tkaczyk, B., 2014. “Crafting Continuing Learning and Development: A Positive Design Tool for Leadership Development,” Development and Learning in Organizations, Vol. 28, No. 4, pp. 5-8.  Tkaczyk, B., 2014. “Daily Check-Ins Stimulate Self-Improvement,” TD, Vol. 68 No. 8, pp. 72-73.

Leadership Development in Post-Heroic Times—An L&D Orientation

You’ve got numbers from your training. Now what? If you find that your data are unclear, or don’t seem to guide your next steps, you’re not alone. Let’s take care of that.Preparation CountsFirst, let’s review how to set yourself up for success with metrics. Here are the key steps to prepare for action:Set measurable goals for the course. For example, start from learning objectives and set goals for how much people should learn. (See” Track Success with Metrics.”)Find the features of the course that impact the goals. For example, use A/B testing to measure the effect on the course of including videos, or making videos of a certain type. ( See “Find the Levers with Metrics.”)If you you’ve done these things, you’ll be ready for your next steps. If you didn’t do these things, and you’ve got a pile of numbers to make sense of, you’ve signed up to be a data miner. Now, mining data is an honorable profession, and if you have a background in statistics and a large volume of data (say, thousands of students), you could give it a try.However, with the limited size of datasets that we typically get from training, it’ll be hard to find patterns that are significant. So, you’ll come out ahead if you have a system designed from the start around measurable goals and measurable features.Course Focus vs. Portfolio FocusDesigning a course around measurable learning objectives may already be part of your design process. We are used to thinking about training in this way. And, this is work that can happen in isolation from other courses. For example, the learning objectives for sales training for Widget X probably have no relationship to the learning objectives for engineering training for Widget Y. This compartmentalization makes it relatively simple work to set measurable goals for courses.To act on metrics, however, requires a portfolio view. To find the courses that need work, and to accumulate insights into what levers work best, is a long-term effort that spans many courses.First, to find the courses that need work, we have to know what an underperforming course looks like. While we will have some external criteria for this, such as “X number of people need to get at least 80 percent of the quiz answers correct,” the best way to address this is to look across the portfolio and find the lowest courses. In other words, establish a baseline, then find the low outlier that should be prioritized for improvement.Next, your experiments to find levers of quality (for example, through A/B testing) will be spread out across many courses. You won’t test every feature of every course, though. In fact, you might test only one or two features in each course to make the job manageable and data easier to interpret. As you accumulate a “bag of tricks” through your testing, you can apply things you've learned from one course to the design and troubleshooting of other courses.Put Best Practices to WorkCongratulations—you’ve set yourself up with some well-designed goals and levers. Now, your next steps are relatively easy.Redesign an existing course: If your data show that a course is underperforming, and it can be adjusted by one or more of your levers, consider redesigning and republishing. This assumes that course will continued to be delivered long enough to justify the investment, which you can ensure by checking your data early in the life of the course. In fact, it’s best to plan for iteration with any course: publish, collect data, adjust, and publish again.Approach future courses differently: Keep track of the best practices that you uncover through testing, and apply them in the planning stage for new courses. More importantly, once you’ve established best practices, you’re going to include them in all of your new courses. What if one of these new courses still underperforms? At that stage, you can have a discussion about the level of investment in the course. Because even with knowledge about best practices, you’ll naturally make tradeoffs between what works best and what you can afford.If the course is a priority for the business and it’s not working well, it may be time to add more high-end features, such as virtual instructors. And thanks to your work with metrics, you were able to point out that it wasn’t working, and you know what to suggest to improve it.Share Best PracticesAs your final action, share your findings with others. When you find what works best for your training, there’s a good chance it’ll be a benefit to other designers. Of course, mileage for a particular design choice may vary. Again, what you find works for engineers in your industry may not apply to another industry or to sales, for instance. So, also keep track of the demographics of your audience and include that information with your report.In the long run, you could help lay the groundwork to make training a leading profession in the data-driven workplace of the 21st century.

Search results for 'Alan B. Knox'

Categories

Refine By

Search results for 'Alan B. Knox'

Categories

Refine By

Topic