Distinctive Learner Survey Questions Elicit Actionable Results

Q: In our field, surveys are often used to get the learner’s feelings about the training such as its usefulness or the learners confidence in applying what they’ve learned. Can you tell us more about your work in learner surveys?

A: My work has always been based on my time spent looking at the scientific research on learning and memory, and then translating that into practical recommendations. I’ve been doing this since 1998 through my organization Work Learning Research.

And as I was doing that work, helping people build more effective learning interventions, as always, dipping into the research, I realized that some of what we do in learning evaluation just doesn’t make any sense. For example, we know that people forget and yet we measure people’s remembering or their ability to respond to our questions right at the end of learning.

And then I came across some research. It was a meta-analysis (research synthesis) really that showed that learner surveys, traditional smile sheets are correlated with learning results at .09. Anything below .30 is considered a weak correlation. There is virtually no correlation at all between smile sheets and learning. Another meta-analysis came 10 years later and found the exact same weak correlation number.

My first instinct was we should get rid of smile sheets as no good. But I realized we’ve been doing this for decades, it’s a tradition, and it’s also respectful to ask our learners for their views. So, then I simply asked the question—can we make them better? I wrote a book on this published in 2016, and the second edition came out last year 2022. So, my answer is yes, we can make them better.

I spent time thinking about what was wrong with the current ones and found three key problems.

1. Numeric Scales are Fuzzy

Well, one thing that’s wrong is we use Likert scales and numeric scales, and those are sort of fuzzy for the respondents, they are making subjective evaluations so they’re struggling between loads of questions all asking strongly agree, or agree, or disagree. That creates a lot of issues where bias can jump in.

2.0 Bias Built-in

When writing Likert scale questions for learners to react to—that statement has a positivity or negativity associated with it, so the bias is built in.

3. Results Unclear

We get an average rating, like “My course is a 4.1”. On examining a lot of smile sheets, you see most of the numbers are between 3.8 and 4.5. There’s no real differentiation that we could use to evaluate the course or make it better. That’s practically useless. People tell me they are paralyzed by their data—not able to interpret it or take any action from it. Only rare anomalies would pop to the surface—like if something happened during the course that would violate an HR policy. That’s easy enough to fix—but what about the rest of the learning data?

Distinctive Questions

When I first published the book, people prompted me to come up with a specific name for this type of question. After some consideration, I decided to call these Distinctive Questions. The idea is that there’s distinctiveness between the answer choices and because of that distinctiveness and granularity, there’s three advantages.

1. Learners are more motivated to answer thoughtfully. Unlike smile sheets that learners can complete quickly, and often do so thoughtlessly, these questions prompt deeper thinking.
2. Learners are supported in answering. With these distinctive choices, you’re really supporting the learners in decision making about the questions. They can wrap their heads around the question and their own thinking. They can easily answer:
   • Do I understand this?
   • Can I use this?
   • Am I confused?
   This results in clearer insights than a sequence of agree or disagree.
3. Leaders can get actionable data. Instead of saying My course is a 4.1, you can see that 20% of people said they are confused, and they don’t know what to distill to know what to do. Then 40% said they’re ready to go, except they need more experience. This is much clearer about what the learners need.

Some people expressed concerns about whether learners would respond well to this new style of question. My thinking was that it’s likely the learners were not great fans of the Likert questions, so we’ve got nothing to lose in trying.

Two global organizations, one for-profit and one not-for-profit worked with me to validate this method. We examined 20 different courses of all types from soft skills to technical training, compliance, and so on. We also asked the question—whether the learners liked the new questions better or equal to the old questions. Eighty percent of the learners liked the new questions better, and 90 percent like them better or equal to the old questions. Recently, another organization that has adopted my method asked their learners the same question and found that 100% of their people liked the new question better than the old type.

Q: Let’s talk about AI.

A: My initial response to the surge of interest in AI, especially since ChatGPT became widely known in November 2022, was one of skepticism—a typical stance for a researcher. People often look to me for insight, asking for my take on new technologies like AI. I advise caution, knowing that with any novel technology, there’s a tendency to become overly excited and initially misuse it.

Many adept learning and development professionals have since demonstrated practical and effective ways to incorporate AI into their workflows. This prompted me to deepen my understanding. One impactful resource was Mustafa Suleyman’s book, “The Coming Wave,” which not only explores AI but also synthetic biology. Suleyman, a co-founder of DeepMind and another AI venture, along with a co-author, provides a thoughtful historical and technological perspective. His book convinced me of AI’s permanence and its classification as a general-purpose technology—like the internet or the automobile—that will inevitably progress and cannot be stopped.

Suleyman discusses the dual-edged nature of such technologies: they bring both benefits and challenges. He emphasizes the importance of containment strategies to mitigate the potential dangers of AI. Based on his insights, I believe AI is here to stay and will be beneficial. It has inspired me to add a chapter on generative AI in my new book, which is aimed at CEOs with guidance for L&D professionals on fostering effective collaboration.

A critical aspect of AI utilization is trustworthiness. Organizations must be diligent to ensure their AI systems are reliable and free from biases, inaccuracies, and making stuff up–known as AI hallucinations. As L&D professionals, we are well-positioned to educate others about AI, helping them to navigate its advantages while avoiding pitfalls.

Some may speculate that AI will replace many jobs, but I urge caution. While AI might handle front-line tasks, suggesting a reduction in front-line staff, we must consider the long-term implications. If we diminish the front-line workforce, we may inadvertently deplete the pool of talent who could advance from those ranks into more responsible positions in the organization.

AI’s advancement is inevitable, and those of us in the L&D field have a significant role to play. We can support our organizations in leveraging AI responsibly, ensuring a balance between technological efficiency and the value of human expertise.

Q: In learning, what is AI currently good at?

A: In the realm of learning, AI is currently showing promise in several areas based on feedback from those who have been integrating it into their practices.

One reported strength of AI is in creating assessment materials, such as test questions, particularly scenario-based questions. It’s believed to be useful for brainstorming, which can be a critical part of the learning and development (L&D) process, where there is often a need to generate marketing text or content creation.

From my own experimentation, I tasked an AI with developing a leadership development curriculum for frontline supervisors. The AI provided a range of relevant topics. However, as a researcher, I wouldn’t rely solely on AI-generated suggestions without verifying their quality. The appropriate approach would be to first consult research on leadership, then use AI-generated content as a supplementary resource. A dialogue with the AI, or a Large Language Model (LLM), could refine the suggestions, discerning which are evidence-based versus those derived from popular but potentially unscientific sources.

Another application I’m considering is using AI to analyze data from my workshops. I use a performance-focused learner survey, and I’m curious if AI could effectively evaluate the open-ended responses, which is traditionally a challenging task.

However, it’s important to remember that our current fascination with generative AI should not be about what AI can do in isolation. Instead, we should focus on our specific problems and opportunities and evaluate if and how AI could be a suitable tool for addressing them.

Regarding AI safety and ethics, transparency is a key concern. Stakeholders are interested in understanding why an AI system makes certain recommendations, which is not inherently possible with deep neural networks as they currently operate. They analyze vast datasets and recognize patterns without being able to explain their reasoning. This is an area that researchers are actively working on, as increasing transparency is crucial as we advance with AI technology.

So, AI’s potential in learning is significant, but it must be harnessed with a clear understanding of its capabilities, limitations, and the ethical implications of its use.

Q: There’s a push in some sectors to get people who were working from home during the pandemic to return to the office. There are ongoing arguments about whether people are more productive remotely or together in an office. Based on your history of building team leaders, what is your perspective? What have you seen that would argue one way or the other?

A: It’s complicated. Some employers are having a knee jerk reaction and want employees back in the office for the sake of control and domination. They’re not really considering the health of their employees, the pollution created by driving to work, and other factors. While some people, like me, have been effectively working from home for decades, for many others they are pioneering their way through this new world of remote and hybrid work. I’m probably biased on this issue. I’ve never liked working in an office. Whether I was working independently or as part of a team—I always work remotely.

Overall, we’re still learning how to interact remotely. An important way that we learn from each other and build relationships is being able to swap stories. Rather than focusing on keeping employees busy every minute of the day, it’s valuable to craft time for connecting. It’s not as easy to do remotely as it is in an office, but it’s also not that hard to do with a remote or distributed team.

There are benefits to being able to reach out to a peer to hear the real life experience that is not found in the training or documentation available. These kinds of interactions can be intentionally planned so the benefit is still there—but not at such a severe cost to the workers.

Here’s an example of how it could be done.

Have a meeting each week where people are randomly broken up into groups of three. Each person shares a story from this week, something they learned, or something they are struggling with or want others’ perspectives on. You can retain the richness of this knowledge sharing without the expense to your employees in time and money wasted sitting in a car or train and commuting.

There are also a lot of online tools arising, collaboration and whiteboarding tools for example, which can make this easier. Together, we’re figuring it out.  

Q: What else?

A: Well, I just think in the learning and development space, we should have some optimism. We’re making progress. There’s new learning technologies. There’s new tools, but also the learning sciences have really solidified and now we’re talking performance sciences. And we’re moving forward.

You know, one of the issues in our field is that we get new people all the time that are moving into our field that don’t have background. You know, it’s good, I guess, in some ways for new blood, but we do need to figure out a way to educate them. We build a common body of knowledge and bring them in so they can start off in the right direction.