By: Kali M. VanLangen, PharmD, BCPS; Lisa A. Salvati, PharmD, BCACP
With the coronavirus (COVID-19) sweeping across the globe, many educators have embraced new delivery methods. We had previously hesitated to conduct virtual skills assessments due to the potential for technology-related issues, academic dishonesty, and losing face-to-face interaction. Despite these concerns, the pandemic forced a transition with only six days of preparation.
Developing the Virtual Teaching Objective Structured Clinical Examination (vTOSCE)
The TOSCE’s purpose was to assess pharmacy students’ verbal and written communication skills related to the Pharmacists’ Patient Care Process and provide feedback prior to the high stakes OSCE. The TOSCE involved 142 students, 19 faculty, and eight standardized patients, in five stations, three of which would be graded. In this unprecedented situation, we relied on familiar technology and past experiences with virtual interprofessional education events to design the vTOSCE.1,2
The two ungraded stations, originally planned to be timed and monitored by a P4 student, were modified to be self-monitored, and all patient case materials were posted in the learning management system. Two graded stations involving students interacting with standardized patients while a faculty grader observed, were modified to remove the standardized patient and be conducted synchronously via Blackboard Collaborate©. To accommodate potential technological issues, an additional minute was added to each station. All grading forms were deployed using ExamSoft©.
Sixteen Blackboard Collaborate© sessions were created with one faculty assigned to each session. Additionally, one faculty was assigned to grade SOAP notes; with another two faculty assisting with technology-related issues and monitoring the virtual sessions. Students were emailed links to the two separate Blackboard Collaborate© sessions they would need to enter, along with phone numbers in the event of technical problems. They were instructed to enter no more than two minutes prior to their start time to prevent one student overhearing another student’s feedback. If a student was unable to connect within the first two minutes of the scheduled interaction, they were rescheduled to the end.
A total of 141 students completed the vTOSCE. Most faculty only made one or two phone calls to students who were unable to connect. One faculty was unable to connect to Blackboard Collaborate© and needed to call all of their assigned students (n=17). Four students were unable to connect within the first two minutes and were rescheduled for later in the day.
Overall, results are similar between the Spring 2019 (S19) and Spring 2020 (S20) student cohorts (Table 1). Of note, different cases are used each year, but the grading rubric is the same.
Table 1: Average Score and Standard Deviation (SD) on each vTOSCE Station
|Patient Consultation |
|S19 TOSCE |
Standardized Patients (SP)
|8.06 (SD: 2.20)||8.27 (SD: 1.92)|
Virtual with Faculty as SP
|8.86 (SD: 1.33)||8.57 (SD: 1.58)|
Faculty and Student Feedback
Faculty provided positive feedback overall. The majority of faculty involved are located at least an hour from the TOSCE site and appreciated not traveling while still interacting with and providing feedback to students. The primary piece of constructive feedback was to provide additional time between students to allow for grading and connectivity issues.
Although a student evaluation of the event was not conducted all students with connectivity issues were appreciative of faculty responding to messages quickly. One student reached out and thanked the coordinator for the opportunity to still practice these skills and get valuable feedback from faculty to continue to improve.
Future of vTOSCE Assessments
As our healthcare environment continues to evolve, providing learners with telemedicine experience will be valuable.3 Literature on “teleOSCEs” in the medical education can provide a starting point for pharmacy educators.4,5 These reports indicate that teleOSCEs are a feasible and cost-effective way to assess future physicians on telemedicine skills that is generally accepted by students and able to accomplish the assessment objectives. The positive feedback and student performance results suggest vTOSCEs may become the new standard. However, these methods likely would not be useful for physical assessment and other hands-on skills.
Questions for the Academy
- What research is still needed to optimize the use of virtual skills assessments in pharmacy education? For example, what is the effect on faculty-student relationships?
- If virtual skills assessments are effective, under what circumstances should faculty still be present for face-to-face skills assessments?
- Surely we are not alone in this endeavor! What have you encountered while adapting your in-person skills assessments to virtual modalities?
- Quesnelle KM, Bright DR, Salvati LA. Interprofessional education through a telehealth team based learning exercise focused on pharmacogenomics. Curr Pharm Teach Learn. 2018 Aug;10(8):1062-1069.
- McCutcheon LRM, Alzghari SK, Lee YR, Long WG, Marquez R. Interprofessional education and distance education: A review and appraisal of the current literature. Curr Pharm Teach Learn. 2017 Jul;9(4):729-736.
- DeJong C, Lucey CR, Dudley RA. Incorporating a new technology while doing no harm, virtually. JAMA 2015;314(22): 2351– 2.
- Palmer RT, Biagioli FE, Mujcic J, Schneider BN, Spires L, Dodson LG. The feasibility and acceptability of administering a telemedicine objective structured clinical exam as a solution for providing equivalent education to remote and rural learners. Rural Remote Health. 2015 Oct-Dec;15(4):3399. Epub 2015 Dec 3.
- Cantone RE, Palmer R, Dodson LG, Biagioli FE. Insomnia Telemedicine OSCE (TeleOSCE): A Simulated Standardized Patient Video-Visit Case for Clerkship Students. MedEdPORTAL. 2019 Dec 27;15:10867.
Kali M. VanLangen is a Professor of Pharmacy Practice at Ferris State University College of Pharmacy. Her research interests include APPE readiness, Entrustrable Professional Activities, and the use of electronic health records in the laboratory setting. In her free time, Kali enjoys spending time outdoors with her husband and two daughters.
Lisa A. Salvati is an Associate Professor of Pharmacy Practice and the Interprofessional Education (IPE) Coordinator at Ferris State University College of Pharmacy. Her research interests include topics related to IPE especially assessment of IPE. Outside of the workplace, Lisa likes cooking, exercising, and spending time with her family including her husband and two daughters.
Pulses is a scholarly blog supported by Currents in Pharmacy Teaching and Learning