Navigating the world of program evaluation: Don’t let it scare you!

By Bob Bechtol, MS

“That’s one small step for man, one giant leap for mankind.” These famous words from American astronaut Neil Armstrong came at a time in history when space exploration was the subject of science fiction novels. However, the Moon Landing mission blazed a trail for future exploration of space. Similarly, program evaluation is a world that has been largely unexplored in pharmacy education. How could the world of program evaluation advance pharmacy education? Evaluation focuses on making judgments about students (e.g., knowledge, attitudes and skills) in the context of questions about specific program goals (e.g., student performance, retention and graduation), or curricular elements (e.g., content or structure).¹ Program evaluation is any effort to use assessment evidence to improve institutional, departmental, divisional or agency effectiveness.1 The evaluation process may involve various components (i.e. programmatic outcomes evaluation, process evaluation, and investigating the unexpected outcomes). As pharmacy educators, we need to explore the world of program evaluation more thoroughly and look for ways to apply its methods. Specifically, we can learn from program evaluation models and professional resources. We might incorporate evaluation science in our examination of the entire Doctor of Pharmacy program or an aspect of it. For example, at the University of Minnesota, we are currently applying program evaluation theories, models and approaches to the evaluation of our leadership program within the Doctor of Pharmacy curriculum. As we have planned the evaluation of this sequence, which spans all four years of our curriculum, we have identified approaches from the evaluation sciences that have helped our evaluation work to be more robust, credible and useful in program improvement. The following are tools to help implement program evaluation.

Use High Potential Evaluation Models & Frameworks
It may be tempting to jump in with a series of surveys or focus groups. However, it’s important to stop and consider the overall evaluation approach. The Centers for Disease Control and Prevention (CDC) Program Evaluation Framework² is a model that follows standards for effective evaluation and uses 6 connected steps to tailor an evaluation of a program (see Figure 1). The steps describe what evaluators do. The standards define elements of quality that must be considered for an evaluation to be optimally effective. The framework has been used to help guide specific evaluation projects and develop guidelines for evaluation. Even though it was originally developed for public health programs, it is applicable to educational programs as well.

Figure 1. CDC Program Evaluation Framework2

In addition to better understanding whether an outcome was achieved, as described by the CDC model, it’s important to know how an outcome is achieved (i.e., the process). When outcomes do not turn out as expected, understanding the process may provide reasons and explanations. Process evaluation can provide feedback to make continuous quality improvements to the content and delivery of the program, in order to reach desired outcomes. However, over the past few decades, outcomes-based models have dominated program evaluation practice compared to models that also evaluate the processes that lead to outcomes. Haji, Morin, and Parker provide a model to guide a more balanced examination of health professions educational programs.³ In addition to examining process and outcomes, Haji, et al. also recommend that we seek to establish what else is happening when programs are implemented (i.e., unexpected processes and outcomes). Positive or negative outcomes may result from the program that were not originally expected or planned. At the University of Minnesota, we are currently applying aspects from the Haji et al. model to better understand which activities within the leadership program have the strongest impact on the desired outcomes. Along with models and frameworks, it’s important to access program evaluation resources to gain knowledge about the field and evaluation standards.

Access Program Evaluation Resources
If program evaluation is new to you, here are a few resources that can help you to get started:

• The American Evaluation Association (AEA) provides mechanisms to build community with other practitioners and scholars of evaluation and provides professional development opportunities in advancing evaluation knowledge and skills.
• The AEA 365 blog is a venue to “read and share hot tips, cool tricks, rad resources, and lessons learned” for and from practicing evaluators and evaluation scholars worldwide.
• The 2011 Joint Committee on Standards for Educational Evaluation developed standards to assess the quality of evaluative activities.⁴ They provide guidance on evaluation options and include standards related to: utility, feasibility, propriety, accuracy, and evaluation accountability.
• If program evaluation is an area of interest, consider signing up for the e-table of contents at the American Journal of Evaluation (AJE) and New Directions for Evaluation.

The world of program evaluation has much to teach us. “How will you explore this new world and apply it to your programs?” Stay tuned for more on program evaluation!

Acknowledgements
Thank you to Dr. Kristin Janke for her guidance and mentorship in preparation of this article.

References

1. Anderson HM, Anaya G, Bird E, Moore DL. A review of educational assessment. Am J Pharm Educ. 2005;69(1):Article 12. doi:10.5688/aj690112.

2. Milstein B, Wetterhall S. A framework featuring steps and standards for program evaluation. Health Promot Pract. 2000;1(3): 221-228. doi:10.1177/152483990000100304.

3. Haji F, Morin MP, Parker K. Rethinking programme evaluation in health professions education: Beyond “did it work?” Med Educ. 2013;47(4):342-351. doi:10.1111/medu.12091.

4. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers FA. The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users. 3rd ed. Thousand Oaks, CA: Sage Publications, Inc.; 2011. 

Robert Bechtol received his Masters of Science in Pharmaceutical Sciences at The University of Toledo College of Pharmacy and Pharmaceutical Sciences. He is currently a Ph.D. Candidate in Social and Administrative Pharmacy at the University of Minnesota College of Pharmacy and an affiliate member of the Wulling Center for Innovation and Scholarship in Pharmacy Education. His education-related scholarship interests include: program evaluation, the scholarship of teaching and learning, leadership development, and faculty development. In his free time, he enjoys going on walks with his wife, playing basketball and softball, and playing board and card games.

Pulses is a scholarly blog supported by Currents in Pharmacy Teaching and Learning