By Patti Shank, PhD .
Designer, analyst, author, and facilitator who writes about evidence-based workplace learning.
Q: Recently, you published a book about multiple choice questions. Tell us a bit about why you felt it was necessary for a book and what’s new or different about multiple choice questions?
A: People often think multiple choice questions are awful, yet they use them anyway, often poorly, leading to multiple problems, including bad data. My goal was to delve into the research and share insights on how to use these questions effectively. My journey began with teaching classes on this topic to a wide range of educators, including those in higher education, K-12, and workplace learning. What I discovered contradicted my initial beliefs. For instance, it’s possible to write competent, decision-based multiple choice questions if you truly understand the work involved. The problem is, people struggle with crafting even basic multiple choice questions well. My aim is to supply guidance on writing better questions. I’ve helped many clients improve their question-writing skills to the point they no longer need my assistance. I’m hoping it will inspire people to view multiple choice questions differently and recognize how they can be well-used in summative and formative evaluations.
Q: How do you perceive AI’s role in editorial assistance and content creation?
A: AI is a good editorial helper, but it has its quirks, like favoring certain words and telling you what it thinks you want to hear. While it’s helpful for tasks like generating summaries or assisting with copy editing, it’s not advanced enough to replace human work in these areas. The key is to use AI as a tool to supplement, not replace, human effort and expertise. It’s effective in streamlining some processes, but its limitations, such as generating inaccurate content, mean it still requires human oversight and intervention. AI’s current state reflects the imperfections in the data it’s trained on, making it a magnifier of human errors rather than a standalone solution.
Q: As someone who has run Learning Peaks for 27 years, what advice do you have for entrepreneurs and solo practitioners, especially regarding business development?
A: Business development can be challenging. My advice, which worked for me due to my love for writing, might not suit everyone. If you can’t display past work, consider creating your examples that showcase your skills and methods. Writing for LinkedIn and your website is crucial. Ensure that everything in your portfolio is of high quality. Sometimes, clients push you to make compromises, but it’s important to display work that show research-based instructional and other design principles you believe in.
Q: How has writing and publishing influenced your career? Do you also engage in public speaking?
A: Writing and publishing have been instrumental in my career and helped me get noticed. Early writing and editing by others helped me become very comfortable with the process and improved my skills (Thanks, Amy Sitze!). Although I always considered myself a writer, I didn’t get good at it (in my mind) until doing a lot of it. I do a lot of public speaking as well, but to be honest, I’m more comfortable writing, as I have time to edit myself.
Q: How important is finding the right clients, and how do you approach this?
A: Finding the right clients is crucial. In the beginning, most people take any work they can get, but as you gain experience, it’s important to be more intentional. Your website should reflect the work you excel at and what’s important to you. Conversations with potential clients should include discussions about what they’re looking for and if you’re a good fit for what they need. For example, some clients prefer someone who follows instructions without pushing back, while others might value a more challenging consultant. Knowing what kind of work you excel at and referring clients to others when necessary is essential. For me, I work best with projects that are mission-critical because clients tend to want more evidence-based direction.
Q: Have you encountered clients who are interested in the whole lifecycle of learning experiences, focusing on improvement and long-term value, rather than treating them as disposable?
A: Most clients have had a very limited scope in mind. For example, helping them improve specific training materials, developing a specific online course, or improving their assessment questions. I’ve had some long-term projects, including working with a healthcare organization on performance improvement projects and a government client on improving all aspects of their learning design and delivery. With this last client, we’ve evolved from fixing problems with materials to working on long-term, critical projects that require thorough assessment. These involve ongoing collaboration with high-level training personnel who genuinely care about the end result. It’s refreshing and exciting to be involved in a project like this, where the emphasis is on quality and impact rather than just content.
Q: What is the role of AI tools like ChatGPT in writing multiple choice questions?
A: AI tools like ChatGPT can indeed write multiple choice questions, but they can’t begin to match the quality that a human expert would produce. While AI can assist in the process, it’s crucial to understand that AI tools should not be solely relied upon to craft these questions. Their output needs to be closely monitored and edited.
The effectiveness of AI in writing multiple choice questions also depends on the domain. For instance, in rapidly evolving fields like technology, AI might not even be up-to-date with the latest developments. In my experience, AI might take you halfway or two-thirds of the way in creating questions, but you still need to put in significant effort to refine and tailor them to your specific content and needs. They tend to produce more general and stereotypical content.
I’m currently exploring how AI can aid in this process and plan to include these findings in an article or book. I’m also looking into recent research from 2023 and 2024 on the use of AI in writing multiple choice questions to compare with my own experiences.
Q: What is the role of research in building a network?
A: Research plays a significant role in expanding and diversifying your professional network. By investigating topics of interest, you can identify key researchers and professionals in those fields. This process not only broadens your understanding but also connects you with experts worldwide, offering diverse perspectives.
For instance, when working on multiple choice questions, I discovered Thomas Haladyna as an important researcher in this area. This led me to explore his work and subsequently the work of those he referenced, expanding my network and understanding. Also, reaching out to researchers directly for specific inquiries has been extremely helpful.
This approach enriches your knowledge base and connects you with individuals who share similar interests or expertise, often leading to meaningful professional relationships and collaborations. Additionally, this method allows you to stay abreast of the latest trends and developments in your field of interest, which can be invaluable for continuous learning and professional growth.
Q: If you could go back in time and give your younger self career advice, what would you say?
A: The most beneficial aspect of my career has been the deep dives into specific areas of interest each year. This practice of choosing one thing to focus on intensively helped me immensely. It led to reading research and eventually pursuing a PhD, which, while I don’t recommend it for everyone, was valuable for my deep dives.
A few of my deep dives along the way:- Performance analysis
- Multimedia
- Instructional authoring using different tools
- How we understand writing
- How practice and feedback impacts learning
- Multiple-choice questions
- AI
Another key piece of advice is to invest in your learning. For instance, I attended a three-day workshop and pursued additional consulting to learn performance consulting, which I paid for myself. This decision to focus on professional development, even at my own expense, has been extremely critical to my learning and performance.
It’s also important to adapt to circumstances, as I did when family issues required me to work from home, a practice that was not as accepted as it is now. This adaptability and commitment to continuous learning and deep diving into subjects have been pivotal in my career growth and effectiveness.