So you want to revert to a lottery?

In an era in which the BMJ listed evidence-based medicine as one of the greatest medical advances in the last 170 years, we were startled to read Dr Day’s editorial, “So you want to be a doctor?”[1] Dr Day raises many issues that are beyond the scope of a brief response. Most surprising, however, is how strongly beliefs about student selection were expressed without apparent consideration of the published evidence on the topic.

The simple fact that many outstanding applicants are excluded from medical school annually is a function of the 6:1 applicant-to-position ratio at UBC, not the selection process. The suggestion that the situation would improve if only admissions committees would revert to the process of sitting candidates down for lengthy conversations with interviewers is simply inaccurate. 

Despite common intuitions, substantial evidence suggests that personal interviews do not yield trustworthy information, even with physician interviewers.[2] One interview measures how candidates perform in that interview, not how they will perform in the next interview, let alone in medical school or practice.[3] 

Overcoming the problems of such a disguised lottery process requires sampling a candidate’s performance repeatedly. This multiple biopsy approach has been shown to reduce the effect of chance and improve predictive capacity,[4] leading to its promotion by a global panel of assessment experts.[5] 

Context is important in judging the value of particular questions, but Dr Day can rest assured that continuous quality improvement efforts are undertaken with routine removal of individual questions that are found wanting. The insinuation that the hundreds of physicians involved in writing scenarios, interviewing, and reviewing files do so with less than the principled, fair, and professional intent Dr Day advocates is not acceptable. They use all of those things plus the power of empirical testing. Anything less would be unethical.
—Kevin W. Eva, PhD
Senior Scientist, Centre for Health Education Scholarship
Associate Professor and Director of Education Research and Scholarship, Department of Medicine
University of British Columbia
—Joseph H. Finkler, MD
Associate Dean, Admissions, MD Undergraduate Program
University of British Columbia

Read more: Dr Day responds.

References

1. Day B. So you want to be a doctor? BCMJ 2011;53:389.
2. Salvatori P. Reliability and validity of admissions tools used to select students for the health professions. Adv Health Sci Educ Theory Pract 2001;6:159-175.
3. Axelson RD, Kreiter CD. Rater and occasion impacts on the reliability of pre-admission assessments. Med Educ 2009;43:1198-1202.
4. Eva KW, Reiter HI, Trinh K, et al. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ 2009;43:767-775.
5. Prideaux D, Roberts C, Eva K, et al. Assessment for selection for the health care professions and specialty training: Consensus statement and recommendations for the Ottawa 2010 Conference. Med Teach 2011;33:215-223.