ATD Footer Logo

Bernetta Reese is the web and project manager for the U.S. Department of Agriculture. She has more than 10 years of public service. Reese also serves as vice president of public relations for Speaking Publicly at Immigration and Customs Enforcement Toastmasters. Views expressed do not reflect those of her employer. Contact her at bernettareese@gmail.com or @BernettaReese.

Bernetta Reese

Spring 2014 - The Public Manager

Embracing the Call to Service

New ATD research finds that using experiential learning (learning by doing) to develop leaders is more likely to improve learning effectiveness and success in the marketplace.

New Research Shows Investing in Experiential Learning for Leaders Pays Off

Sally is the chief learning officer for a large well-known bank. She was recently brought in by her CEO to “professionalize” the Learning and Performance function within the company. Sally inherited a very extensive learning measurement system that rigorously cataloged the results of each training course using Likert-Like questions and numerical averaging.As Sally sifts through the data, she notices something curious. Instructors whose smile sheet ratings were consistently low were still teaching, even after several years of poor results. Sally, of course, is thrilled! She’s been mandated to professionalize the learning function and already she’s found a problem she can fix. She makes a plan. She’ll bring her team together to improve—or fire—the low-performing trainers.Brilliant, right? Maybe! It depends on whether Sally’s assumptions are accurate:Instructors who get the best smile sheet ratings will help their learners achieve the best learning results and the best on-the-job performance. Instructors who get the worst smile sheet ratings will produce learners who achieve the worst learning results and the worst on-the-job performance.Interestingly, Sally’s predecessors never thought to correlate their smile sheet ratings with learning results or on-the-job performance. Sally, like most L&D professionals, doesn’t think of doing this either. She just assumes that smile sheet results are related to learning outcomes. Indeed, as an industry, we tend to assume that smile sheets are telling us something important about the successes and failures of our training efforts. Are we right? Is Sally right? And does it really matter, anyway?In 2013, ATD and i4cp teamed up to examine how organizations measure the effectiveness of the learning programs. The research, reported in The Value of Learning: Gauging the Business Impact of Organizational Learning Programs, found that Level 1 smile sheets were by far the most common method of measuring our learning solutions. One implication from these findings is that we, as workplace learning professionals, get most of our feedback from smile sheet data.Consequently, because feedback enables improvement, the validity of smile sheet data is critical. In other words, if we’re not getting good data from our smile sheets, we’re not going to be able to create maximally effective learning.Research on Smile SheetsSo, what does the research say? Are smile sheets effective in giving good feedback? The short answer is No. Two meta-analyses in the scientific literature covering more than 150 research studies found that smile sheets are virtually uncorrelated with learning results (see Further Reading below). Specifically, both meta-analyses found that smile sheets were correlated with learning results with a correlation of r = .09.Statisticians tell us that correlations below r = .30 should be considered weak correlations. Therefore, a correlation of r = .09 is practically no correlation at all! It would be like correlating smartphone usage with hair color.Bottom line: Sally’s assumptions were faulty. Our assumptions, as an industry, have been faulty for decades. We’re all caught in the delusion that smile sheets tell us something important, when in fact they may regularly mislead us.A Smarter Smile SheetWow! What should we do then? Should we throw out our smile sheets? Should we collect data to be polite and then ignore the data? I don’t think so. After thinking about the smile sheet problem for a decade or so, I’ve come to the conclusion that if L&D professionals redesign smile sheets from the ground up—based on the science of learning and guidelines from the measurement field—we can build effective smile sheets.In Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, (more info available at www.SmileSheets.com), I describe how to build a much more effective smile sheet. There are two critical aspects.First, questions should be based on the science of learning. Because we’re asking that our smile sheets tell us about learning, it makes sense to base our smile sheet designs on learning science. Second, we have to do a better job of designing our smile sheet questions. Most of us use Likert-like scales or numerical items, both of which are fundamentally inappropriate for smile sheet use. We also have to help our learners make good smile sheet decisions. According to research from Brown, Roediger, and McDaniel (2014), as well as Kirschner and van Merriënboer (2013), learners don’t always know their own learning. Because of this, asking them questions about their learning may give us faulty information—unless we design our questions to support their smile-sheet decision making. Too many of our questions ask learners to answer questions they are not prepared to answer.Here are some of the major areas that smile sheet questions should target:ability of learners to successfully apply what they learned to their work learners’ full comprehension of what was presented to them learners’ capability to remember what they’ve learned level of motivation that learners will bring to implement what they’ve learned post-training resource that will guide and enable learners to persevere in putting their learning into practice.Most smile sheets do a poor job of targeting these issues, which is unfortunate because it is through these types of research-inspired factors that learning effectiveness can be discerned.Bottom LineMost of our smile sheets are ineffective and provide poor feedback. Fortunately, we can build better ones. Of course, smile sheets will never give us all the information we need. It’s critical that we also measure learning by focusing on learner comprehension, decision making, and memory, as well as measure application by examining the factors and results produced in the way of on-the-job performance.While smile sheets are never enough, most organizations will continue to use them, so we ought to make them as effective as possible. Scientific research makes it clear that our current smile sheet efforts are ineffective. Fortunately, the science of learning and wisdom from the measurement field give us hope for better smile sheets.Like what you read? Join us at the ATD 2016 International Conference & Exposition in Denver, CO, May 22-25 to hear Will Thalheimer speak.Further ReadingAlliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341-358.Brown, P. C., Roediger, H. L. III, & McDaniel, M. A. (2014). Make it stick: The science of successful learning. Cambridge, MA: Belknap Press of Harvard University Press.Kirschner, P. A., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169-183.Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied Psychology, 93, 280-295.

Are Smile Sheets Supported by Research?

Elisabeth E. Bennett, PhD, is an assistant teaching professor with Northeastern University’s doctor of education program, where she teaches organizational leadership, adult learning, communication, and research courses. Her scholarship has focused on organizational culture and virtual human resource development, and increasingly informal learning. She spent five years in scholarly practice as director of education and research development at the Western Campus of Tufts University School of Medicine. A longtime member of AHRD, she served on the board of directors, and is presently on the editorial board of three journals.

Elisabeth Bennett

This is the second post in a three-part series examining leadership.  (See post 1 and post 3.) In the present business climate, the leaders need to become champions of the “learning organization” as designers, teachers, and stewards (Senge 1990), rather than position themselves as the traditional charismatic hero (Steyrer 1998).  With that in mind, leading others requires constant learning and development (L&D) and designing of knowledge-creative workplaces that foster growth. Agile Skills Required Enter “design thinking.” Research reveals that design-driven companies, such as Apple, Coca-Cola, Ford, IBM, Intuit, Nike, P&G, Starbucks, Starwood, Target, Walt Disney, or Whirlpool, have outperformed America’s S&P 500 large companies by an extraordinary 228 percent over 10 years (DMI 2014). Likewise, being smart about agile working, agile workforce, and agile environments and cultures should be on every leader’s list of priorities. In fact, a 2014 CIPD survey of HR leaders and employees found that improving “leadership and management capability” (64 percent of respondents) and “learning and development” (54 percent of respondents) are top tactics to deliver organizational agility. Meanwhile, respondents in the CIPD research ranked “reward management” (15 percent of respondents) and “increasing workforce diversity” (17 percent of respondents) much lower. In the 2014 report, Change Agents: The Role of Organizational Learning in Change Management, ATD Research concurred that organizations and their leaders should be more adaptive to change. Indeed, the research study found that 61 percent of organizations experience three or more changes per year, but they are not prepared to manage that change. Only 17 percent of respondents rate their enterprises as highly effective at managing change, and a mere 30 percent of organizations have change management teams. The good news is that there is clear evidence that an adaptive, generative, and knowledge-productive learning and development (L&D) approach to leadership development can provide the infrastructure for numerous progressive methods designed to produce adaptive leaders. Coaching and self-managed learning are clear examples of this approach. Why Coaching Works Through my own research into leadership development, I have found that coaching is about building and sustaining high-quality links with others. Extremely good coaches energize, ask, listen, invite, and equip others to develop themselves and achieve their personal and professional best. How does this work? The coach energizes the relationship, while the coachee empowers it. As a result, the link re-energizes the coachee, so that he or she is better able to discern, ideate, transform, and deliver (Tkaczyk 2014). More importantly, numerous coaching skills can be applied routinely with clients, peers, or direct reports. They include setting the stage, bringing forth positive leadership presence, tracking information, identifying issues and key factors, playing back exchanges and checking-in, giving and receiving constructive feedback. In addition, one of the most important tools when working with others is the skill of positive inquiry (Tkaczyk 2014). Meanwhile, Stanford’s 2013 Executive Coaching Survey revealed that nearly 100 percent of CEOs enjoy the process of coaching and receiving leadership advice from professional coaches and outside advisors. Although CEOs are receptive to making changes based on these interventions, the Stanford study found that a majority (66 percent) do not receive leadership advice or coaching. To be sure, the benefits of coaching for developing leadership skills are tangible. ATD Research’s report, The Coaching Approach: A Key Tool for Successful Managers, concluded that, in reality, coaching improves communication (69 percent), raises engagement (65 percent), improves skills-to-performance transfer (63 percent), and stimulates productivity (61 percent). Becoming Self-Managed Learning Leaders As leaders grow, they need to become “L&D crafters” (Tkaczyk 2014). To maximize their leadership presence and learning agility, rooted in design thinking (Austin and Devin 2003, Martin 2009) and positive organizational scholarship (Cameron at el_._ 2003, Cameron and Spreitzer 2012), such L&D crafters may routinely use a “continuing developmental portfolio” comprised of two components: Continuing Executive Development Check-In: A disciplined L&D system, self-managed by an individual who constantly “checks in” with themselves by asking reflective questions regarding any critical and opportunistic incidents that happened in the workplace or outside, daily or in the past week.Design: A natural and drilled L&D mechanism that is self-guided by the leader, in which they constantly “design” their own L&D by asking developmental inquiries regarding what it is that they still want to learn and develop, and how exactly they will execute their development plan. International research conducted by the Learning Entrepreneurs project from September 2011 through September 2013 found that such self-managed learning produces personal gains: upgrades your skills and helps monitor your learning in an evidence-based way (91 percent) and boosts your credibility as a reflective and creative leader (90 percent). It also offers organizational benefits: unlocks HR potential (80 percent) and builds “knowledge-creative” workplaces (73 percent). Interestingly, participants in this global research project represented diverse geographies, including Africa, Europe, Latin America, Middle East, and North America. Still, more scientific research is needed, especially as Asia/Pacific leaders were not represented in the study (Tkaczyk 2014). For more on “self-coaching” as a self-managed leadership development method, see my August 2014 TD article, “Daily Check-Ins Stimulate Self-Improvement.” Finally, other approaches for developing leaders for future success can include:cognitive or relational job re-designjob enlargementsposition transfersjob rotationssabbaticalsexternships—professional development through a full-time temporary position at another organizationself-assessment inventories360-degree feedbackpsychological profiling and strengths-findingleaderless group discussions in which a team of multiple leaders is assigned a complex challenge and must work together to work it out within a certain time periodacting and storytellingconcept mappingrepertory gridscritical incident technique, as utilized in job analysis,mentoring·one-on-one Personal Management Interview (PMI) programs, namely purpose-finding and role-negotiation sessions—approximately 60-120 minute each, and preferably twice per month, depending on life cycle rhythm of work. The final installment in this leadership series will address future vistas in leadership.References ATD Research, 2014. Change Agents: The Role of Organizational Learning in Change Management, Association for Talent Development (ATD), Alexandria, VA.  ATD Research, 2014. The Coaching Approach: The Key Tool for Successful Managers, Association for Talent Development (ATD), Alexandria, VA.  Austin, R. and Devin, L., 2003. Artful Making: What Managers Need to Know About How Artists Work, Prentice Hall, Upper Saddle River, NJ.  Cameron, K., Dutton, J., and Quinn, R. (eds.), 2003. Positive Organizational Scholarship: Foundations of a New Discipline, Berrett-Koehler, San Francisco, CA.  Cameron, K. and Spreitzer, G. (eds.), 2012. Oxford Handbook of Positive Organizational Scholarship, Oxford University Press, New York, NY.  CIPD, 2014. HR: Getting Smart About Agile Working, Chartered Institute of Personnel and Development, London.  DMI, 2014. “Design-Driven Companies Outperform S&P by 228% Over Ten Years—The DMI Design Value Index,” Dialog, Design Management Institute (DMI), Boston, MA.  Martin, R., 2009. The Design of Business: Why Design Thinking Is the Next Competitive Advantage, Harvard Business School Press, Boston, MA.  Senge, P., 1990. The Fifth Discipline: The Art and Practice of the Learning Organization, Century Business, Random House, London.  Stanford, 2013. 2013 Executive Coaching Survey, Stanford University/The Miles Group.  Steyrer, J., 1998. “Charisma and the Archetypes of Leadership,” Organization Studies, Vol. 19, No. 5, pp. 807-828.  Tkaczyk, B., 2014. “Critical Clarification on Coaching,” International Coach Federation (ICF).   Tkaczyk, B., 2014. “Positive Inquiry Skills: Crafting Vital Coaching Questions," Association for Talent Development (ATD).  Tkaczyk, B., 2014. “Crafting Continuing Learning and Development: A Positive Design Tool for Leadership Development,” Development and Learning in Organizations, Vol. 28, No. 4, pp. 5-8.  Tkaczyk, B., 2014. “Daily Check-Ins Stimulate Self-Improvement,” TD, Vol. 68 No. 8, pp. 72-73.

Leadership Development in Post-Heroic Times—An L&D Orientation

As a researcher and practitioner, Dr. William A. Gentry examines what leaders can do to be successful and avoid derailment. As a keynote speaker for conferences and events, he informs and inspires leaders, particularly entry- and first-level supervisors and managers who are leading for the first time to be their best. During his career he has amassed more than 70 academic presentations and has published more than 40 academic articles. In addition, his research has been featured in more than 50 internet and newspaper outlets including Forbes.com, the Wall Street Journal, CNN.com, Harvard Business Review, Chief Learning Officer, ChiefExecutive.Net, and TrainingIndustry.Com. Most recently, William has used his passion for research and developing first-time managers to write the highly-acclaimed book Be the Boss Everyone Wants to Work For: A Guide for New Leaders (Berrett-Koehler publishers, 2016). Currently, William is the Executive Director of Brave Leaders Inc., a Brené Brown company that provides courage-building programs for leaders at every level of an organization by bringing Brené’s latest research on leadership development and culture change to teams, leaders, entrepreneurs, change makers, and culture shifters.

Search results for 'Bernetta Reese'

Categories

Refine By

Search results for 'Bernetta Reese'

Categories

Refine By

Topic