By H Andrew Wilsey, PharmD
As a recent pharmacy school graduate, I have a secret to share. If you, as a professor, repeat exam questions, a more than small percentage of students have access to them and may be simply memorizing old questions instead of actually learning the class material.
You may say in response, “But wait, this question has a great point biserial! It’s difficult to write a question of this quality.”
To which I would reply, “This statistic is meaningless when a student is memorizing your exam question rather than critically thinking about the content.”
Perhaps your comeback is, “Well, to ensure their integrity, I’ll restrain how students can review their exams. Less question exposure, less chances of their dissemination.”
And I would argue therein may lie the problem: sacrificing teaching opportunities by restricting exam reviews in favor of securing quality exam questions.
Repeating Questions
I understand – composing reliable and quality questions is no easy feat. Designing questions with appropriate distractors can take hours, and for some types of content, there are only so many ways to structure questions that ascertain crucial, but finite concepts. Furthermore, if we do try to implement new questions, we gamble with the ability to measure the attainment of our learning objectives should the question be of poor quality. Unfortunately, even among healthcare professionals, academic dishonesty still occurs at shockingly high rates.1 Repeating questions from year to year weakens the incentive for future students to apply themselves if they have (unethical) access and, consequently, leaves gaps in students’ clinical knowledge. Nonetheless, if the response to these challenges is to simply restrict exam exposure and recycle questions, we are doing students a disservice.
Review, Not Repeat
The exam review process is critical for student growth and development, even if that means accepting an increased exposure of exam questions. Reviewing an examination in a group setting can be a powerful way to help strengthen meta-cognitive abilities.2 This process forces students to identify the problems in their thought process and, as the instructor, offer strategies to remedy them. Facilitated exam reviews also may address the common student mistake of obsessing over the minutiae of a question, instead teaching them to comprehend the larger application of the concepts.
Exam reviews represent a venue for formative feedback, a teaching opportunity guided by an expert in the area, creating a dialogue that benefits the learner and instructor. As instructors of future healthcare professionals, we are charged with instilling a practice of lifelong learning and self-reflection. Placing barricades along this process for the sake of protecting exam content is not only futile, but also may be harmful. For example, the common practice of offering examination review solely during one-on-one office hours places another obstacle between a student and formative feedback, favors the highly motivated student, is an inefficient use of a professor’s limited time, and eliminates the synthesis and integration of skills typical of group discussion.3 That leaves most students with just a grade, which essentially provides no effective feedback to constructively inform students’ learning.4
Recommendations
I believe that the teaching and learning opportunities that arise from hosting exam reviews are not only invaluable for students and instructors, but also that exam content should be refreshed annually. Restricting exam reviews curbs the learning of the current students, while offering exam reviews without altering exam content thwarts the educational experience of future students. Even if exams are reviewed in a controlled environment, academic dishonesty can find a way to worm itself in.
To optimize the examination and review processes for all learners, consider these recommendations:
- Schedule an hour each week to serve as a review hour for all courses, as to prevent reviews from cutting into instructional time.
- Review general exam topics, emphasizing concepts missed rather than directing students’ attention to extraneous exam details.
- Test 2 to 3 new questions with each exam that are not graded to provide a low-stakes opportunity to identify areas of improvement for first-time questions (similar to the NAPLEX approach). This would be performed with the goal of developing a pool of questions to provide variety between years.
- Use a variety of exam questions, including short essay.
- Review and avoid common question writing flaws when incorporating new content.5
- Evaluate variables within questions/cases that may be altered to present learners with a new question.
- Utilize electronic exam formats, which can facilitate an immediate review process in a more secure manner.
- Revisit whether there are alternative methods/activities other than exams to ensure class understanding, especially if struggling to rework a basic, but fundamental exam question.
Do you have any “secrets” you’d like to share with me on how to protect exam question integrity while using exam reviews as a form of teaching?
Acknowledgements:
I’d like to thank Jeff Cain for fostering and guiding this scholarly opportunity.
References:
- Austin Z. et al. The fault lies not in our students but in ourselves: academic dishonesty and moral development in health care professions education- results of a pilot study in Canadian pharmacy Teach Higher Educ 2005 (10) 147 60
- Medina M. et al. Strategies for Improving Learner Metacognition in Health Professional Education. Am J Pharm Educ. 2017 81:4
- Stephen D. Brookfield, Discussion as a Way of Teaching (2nd Edition, 2005) John Wiley and Sons. 21-22.
- Schinske J, Tanner K. Teaching More by Grading Less (or Differently). CBE Life Sciences Education. 2014; 13 (2):159-166.
- Dell K. et al. How to guide for writing multiple choice questions for the pharmacy instructor. Currents. 2017;9 (1): 137-144
H Andrew Wilsey is a PGY1 Pharmacy Practice Resident at the University of Kentucky Medical Center. Educational scholarship interests include gamification of classroom learning and developing student metacognitive abilities on clinical rotations. In his free time, H Andrew enjoys spending time outdoors and trying new restaurants around town with his co-residents.
Pulses is a scholarly blog supported by Currents in Pharmacy Teaching and Learning
Pulses 
Very important topic! Reviewing and reflecting on our successes AND failures is how people learn. I’d add one recommendation I picked up at an AACP annual meeting a couple years ago: peer review of exam questions. Gather a collection of colleagues to exchange and critique exam questions to make sure they meet all the various criteria of a good exam question. Exchange 5 to 10 per year, and after a few years you can have a sizeable bank of validated questions, so every year you can have a different exam, thereby eliminating the risk that students will simply memorize every question. (Apologies to the people who presented this info — I no longer have the citation handy.)
LikeLike
Dr. Wilsey, thank you for starting this dialogue. There is a body of information that discusses reusing test items. The take away, as I understand it, is that even though students have knowledge of some of the test items, the impact on overall performance is not as meaningful as we tend to think. Though the percent of “wrong-to-right” (the answer would have been wrong, but since the learner knew of the question they answered it right) answers was increased, the percent of “right-to-wrong” was also increased. It seems like a tradeoff of questions that learners were not prepared to answer. The concern of learners recording and sharing exam items is real. It’s a shame, but it’s real. Faculty invest time writing and revising exam items to ensure they are fair to everyone who encounter them. It’s concerning that our academic culture, in turn, shies away from assessment as learning because it may jeopardize assessment of learning. Your piece calls to our attention the opportunity to return to assessment as learning.
LikeLike