Evaluating the Effectiveness of AI in Learning

Q: What measurements or metrics do you use to evaluate the effectiveness of AI-driven learning programs in organizations?

A: To evaluate AI-driven learning programs like those involving ChatGPT, you could use simple measures for each of four areas.

Quadrant 1: Significantly Different.

The projects are different from anything we have done before. For example, learners do mentoring and self-study assisted by ChatGPT, enabling learners and workers to access beyond prescribed content to real-work data like spare parts status checks while learning to fix a cylinder engine block. This brings learning not just closer to work but in the process of doing work.

Quadrant 2: Value Back.

Catapult projects, those that drive your learning practice forward significantly, support exponential growth of new value-add. A good illustration is training people to ask prompt questions for deeper queries and conversations with ChatGPT. Learning to ask questions is a sustaining value-add for learners and workers. This can potentially reduce our dependencies and propensity for mass memorization. If workers can find answers faster with the right prompt questions in ChatGPT, the need to recall all content may eventually fade away. We will only need to memorize content that truly matters.

Quadrant 3: Scalability.

Catapult projects consider how we can make learning and performance solutions scalable for as many people as possible. Does the software and platform integration expand capacities for learning and performance? For example, Salesforce has acquired Airkit.ai to help their partners build lightweight and easy-to-develop and deploy Apps for customer-driven activities. In L&D, we see the early Generative AI apps that help learners, workers, and trainers rapidly develop content sharing, and limit this to course developers.

Quadrant 4: Organizational and Individual Wellness.

Technological breakthroughs like ChatGPT AI do not flourish in a vacuum or are isolated from the people they are designed to serve. “It is the people who decide if the technology works or not.” As we design tools, we need to keep an eye on making changes that align with espoused values and vision, cultural and community preferences, and concerns. Consideration for ethics, projection of rights, bias, and accuracy of content are inherent discussions that must take place. Central to this is people’s well-being and organizational agility. Don Norman, the leader in “humanity-centered design,” implores us to design technology centered around people.

For each of these quadrants, both qualitative and quantitative data should be collected. Qualitative data might include learner feedback, focus group discussions, and anecdotal evidence of the program’s impact. Quantitative data could encompass usage statistics, completion rates, assessment scores, and other measurable outcomes.

The actual tools and methods for gathering these metrics might include learning management systems (LMS) analytics, surveys, interviews, observational studies, performance metrics, and data analytics platforms. It’s also critical to align these metrics with the organization’s strategic goals and learning objectives to ensure that the AI-driven learning programs deliver the desired outcomes.

Q: How does the user experience need to be adapted for AI-enhanced corporate training to be effective?

A: To ensure that AI-enhanced corporate training is truly effective, the user experience must focus on solving real-world problems rather than simple, theoretical “toy problems.” The key to engaging in training is to immerse learners in real-world scenarios that they are likely to encounter in their actual work. This approach not only tests their ability to apply what they’ve learned but also enhances their problem-solving skills in a practical context.

When integrating tools like ChatGPT into training programs, it’s important to move beyond using it merely as a question-answering machine. If learners are just memorizing prompts to get quick answers, they’re not fully engaging with the content or developing critical thinking skills. This reliance can diminish the true educational potential of ChatGPT and leave learners unprepared to deal with complex issues such as AI-generated errors, misinformation, and inherent biases.

Instead, learners should be encouraged to critically evaluate ChatGPT’s responses, challenge inaccuracies, and use AI to assist in identifying and mitigating biases, especially those that could impact the learning environment. By using ChatGPT as a tool for deeper inquiry and critical analysis, workers can enhance their learning experience and prepare themselves to navigate the complexities of their roles effectively.

Q: How can training solutions balance standardization vs personalization in learning at scale?

A: AI training solutions can achieve a balance between standardization and personalization by focusing on context-specific learning that targets workplace problems.

Standardization in training ensures consistency and compliance, but it might cater to something other than individual learning needs. Chat AI, while not ideal for standardized learning, excels in areas like exploration, problem-solving, and innovation. It allows learners to discover solutions through interaction, which can lead to a deeper understanding of complex subjects.

To find the right balance, AI training should be designed around the specific problems employees face in their roles. This approach leverages the strengths of AI for personalization—adapting to the learner’s style and pace, while also ensuring that the core learning objectives remain aligned with the company’s standards. By solving real work-related problems through AI, employees can apply what they learn directly to their tasks, making learning both relevant and immediately applicable.

Q: What ethical considerations come into play when rolling out widescale AI for corporate learning? How can risks be mitigated?

A: The question of whether ChatGPT is morally good comes down to several critical ethical issues such as the following.

Copyright concerns.

Instructional designers must ensure ChatGPT doesn’t produce content that infringes on copyright, using only what’s permissible under fair use.

Bias reduction.

Human designers must continually refine ChatGPT to avoid biases. Training with inclusive examples and balanced prompts is essential for representing diverse perspectives.

Accuracy checks.

Human oversight is necessary to ensure ChatGPT’s training materials are accurate and relevant, with validation steps to correct errors and a focus on objective topics.

Privacy protection.

Regular audits of training datasets and vigilant monitoring are required to protect privacy, with private models offering enhanced security.

Data security.

When using ChatGPT for business training, it’s crucial to avoid using confidential data in training and to develop models with public domain information to prevent data leaks.

Whether ChatGPT is considered morally acceptable or otherwise, depends on how well these issues are handled by its creators to make it a tool that respects users, is used responsibly, and follows ethical guidelines as AI technology keeps advancing.

Q: What excites you most about the future possibilities of using AI like chatbots for corporate learning and development? What innovations do you expect to see?

A: AI and chatbots hold great promise for corporate learning, mainly because they can improve how we teach and learn in the workplace.

Traditional learning methods often spend too much time on basic concepts before moving to practical skills. ChatGPT changes this by quickly getting learners to the point where they can apply their knowledge. This not only saves time but also helps learners become productive more rapidly.

With AI, companies can feed massive amounts of documents into a system like ChatGPT, which can then create a customized learning experience. This customization can be done securely, without needing an internet connection, ensuring privacy and relevance to the company’s needs.

These advancements suggest a future where corporate training is not only quicker but also tailored to the specific needs of each business. AI and chatbots are set to make workplace education more targeted and effective.