Our study on virtual patients came out in Perspectives on Medical Education last month, and the finding I keep returning to is not in the title. It is a small embarrassment buried in a survey table. Of the five things we asked twenty-one medical students to rate, the one they liked least was the one the rest of the evidence points at. I would like to say we predicted this. We did not.
What we built
The design was not exotic. A single 110-minute session, slotted into an existing oncology elective rather than bolted on as extra work. An oncologist worked through a head-and-neck cancer case aloud, reasoning where students could hear it. Students then worked individually through a virtual patient case — half got one patient, half got another, differing in age, comorbidities, presentation. Then we paired them across the two groups and handed them a structured set of “what-if” questions. What if the patient were an eighty-year-old woman? What if she had a history of diabetes? Finally, two paper cases: another head-and-neck case, to see whether anything carried across a short distance, and a lung case, to see whether anything carried across a long one.
Then we asked them what they thought of it, scored their answers against a rubric built with two surgical oncologists, and interviewed eight of them.
The table nobody expects
Students rated the overall experience 4.48 out of 5. Role modelling, 4.44. Practice with varied cases, 4.35. Their own sense that they could now transfer what they knew, 4.33.
The “what-if” discussions: 3.76.
Every component in the room scored in the high fours except the one where we made them argue with a peer about a patient who did not exist. That component also had the widest spread of any scale we measured. Students did not mildly dislike it. They disagreed with each other about it, sharply.
Now the interviews. Asked what helped them understand why rather than what, students went to the what-if discussions and stayed there. One described realising his treatment choice was wrong for his partner’s patient, and only then seeing what the choice had actually depended on. Another said the prompts forced him to ask why a test would be necessary at all if the tumour were in the lung instead of the neck. Nobody said this about watching the expert, which they had rated far higher and which they described, accurately, as clear.
And the scores went where that account predicts. Overall performance on the near case — another head-and-neck patient — did not significantly improve. Performance on the lung case did. Within the items, the gains sat on staging reasoning and on first impressions of a patient; they did not sit on choosing tests or picking a treatment. The intervention did not make students better at oncology in general over a hundred and ten minutes, which would have been an extraordinary claim. It made them better at the part where you have to say why, in a room they had not been in before.
What I think the low score means
The obvious reading is that students rate comfort. Watching an expert reason is pleasant; the pieces click. Doing a case is satisfying. Being handed a question designed to make your earlier answer look contingent is not pleasant at all, and rating it four out of five would require a kind of pedagogical sportsmanship that nobody owes us.
I believe that reading, but I do not think it is the whole story, because it does not explain the spread. A uniformly uncomfortable activity produces a low mean with everyone clustered near it. What we got was disagreement — and our own limitations section says why. The discussions were student-led, and their quality almost certainly varied from pair to pair. Some pairs got a real argument. Some got twenty minutes of two people agreeing politely. Both groups filled in the same survey.
So the low mean is probably not measuring the activity at all. It is measuring the variance in how the activity was run, which is a different thing and a much more actionable one. “Students rated this 3.76” invites you to cut it. “Students rated this 3.76 with the widest spread in the study, and the interviews say it is where the understanding came from” invites you to fix the facilitation and the discussion guide. Same number. Opposite decision.
The part that should worry curriculum committees
Educational programmes are steered, in practice, by satisfaction data, because satisfaction data arrives quarterly and in a format a committee can read. Transfer data arrives in three years, if someone funds it, and comes with a p-value that requires explanation. Given a scoreboard where one component reads 4.44 and another reads 3.76, the institutional gradient runs one direction only.
I want to be careful here, because I am the first author and it would be convenient for me to conclude that my favourite component is secretly the important one. The honest position is narrower. The design was integrated — modelling, then varied practice, then discussion, then the transfer tasks — and we cannot isolate what any single piece contributed. Twenty-one students is a small number. A hundred and ten minutes is a short time. The interviews reflect eight people who volunteered. None of this licenses “what-if discussions cause far transfer.”
What it does license is a warning about the instrument. Course evaluations measure whether a component felt good. They are frequently read as measuring whether it worked. On our data those two quantities pointed in opposite directions, and if we had only run the survey — which is what most teaching evaluation consists of — we would have concluded that the discussions were the weak link and quietly trimmed them next year.
The cheapest thing an educational institution can do is not run a better study. It is to stop treating a satisfaction score as a verdict. When something rates low, it is worth two more questions before the axe comes out: what is the spread, and what do the people who rated it low actually say happened. Occasionally the answer is that the thing was badly run. Occasionally the answer is that it was the only part of the afternoon that asked anything of them.
The paper is open access, appendices and all — the what-if questions and the discussion guide are in there, so anyone who wants to run this can take the materials and improve on our facilitation, which would not be difficult.
Li, Z., Asoodar, M., Keulers, T., Joosten-ten Brinke, D., & Dolmans, D. (2026). Ensuring Knowledge Transfer: The Critical Role of “What-If” Discussions in Virtual Patient Simulation. Perspectives on Medical Education, 15(1), 633–643. DOI: 10.5334/pme.2363