Meaningless exams?

In the first month of UBC medical school, students are taught that, in the vast majority of cases, listening to what a patient has to say (i.e., taking a good history) yields the diagnosis. Ordering expensive laboratory tests or special investigations are only warranted when they will provide additional information aiding in the diagnosis or management of disease. However, the UBC Faculty of Medicine obviously does not practice what it preaches.

In August 1997, the faculty introduced its new problem-based learning (PBL) curriculum that effectively replaced the older didactic-based curriculum. Under this new system, students in their preclinical years have a limited number of instructional hours in lectures and laboratories. In the 3 years since its inception, it has been suggested by some that the new curriculum is less effective at training students and that during their clinical years, students arrive less prepared compared to those trained in the old curriculum.

In early 2000, the Faculty decided that students at the end of their second year should write the Comprehensive Examination of the Basic Sciences. This examination is prepared by the group that writes the United States Medical Licensing Examination, Step 1. As an external examination, it was touted as an effective way of comparing students from UBC to other students across North America. However, to make valid comparisons it would be necessary to know which other schools take this examination, what type of curriculum these schools employ, and how they allow their students to prepare.

Upon further questioning, Dr Wes Schreiber (associate dean of Undergraduate Medical Education) was unable to name any other US or Canadian schools participating in this examination (because there are none), and therefore could not explain how a valid comparison could be made. The reality of the situation is that the faculty needs to assess its new curriculum and in its view, assessment equals examination.

The faculty could easily ascertain the strengths and weaknesses of the new curriculum simply by asking students for their input and listening to what they have to say. It seems rather pointless for the supposedly cash-strapped faculty to order an expensive examination that will yield results that are meaningless. Perhaps the faculty needs to go back to medical school to learn to take a history first...

—Some concerned medical students

UBC responds

The authors of this letter to the editor question the value of writing the Comprehensive Examination in the Basic Sciences at the end of the second year of medical school. In responding, I will provide some background on our decision to use this exam, its benefits to both students and faculty, and its cost.

The Comprehensive Examination in the Basic Sciences is a 180-question written examination that tests knowledge, concepts, and reasoning skills across all of the sciences basic to medical practice. The questions come from the same pool used in constructing the United States Medical Licensing Examination, Step 1. More than 100 content experts from across the United States and Canada sit on the committees that prepare these questions, and the performance of each question is measured and tracked by the National Board of Medical Examiners. The test is therefore of very high quality and allows a student to compare his or her performance to a nationally derived peer group.

The addition of the Comprehensive Examination in the Basic Sciences to our in-house evaluation system was considered attractive for several reasons: (a) it tests a wide variety of facts and concepts considered important by experts across North America, (b) the questions are well written and the exam is carefully constructed, (c) it provides a standardized, externally developed measure of how students are performing, and (d) it encourages cumulative learning, since material from all of the basic sciences is examined.

The inclusion of the Comprehensive Examination in the Basic Sciences in the student evaluation system was debated by the Faculty of Medicine’s Student Evaluation Committee. Several members of the committee analyzed a series of 150 sample questions for their appropriateness and relevance. The committee also asked for feedback from all basic science department heads, as well as a group of third-year medical students who had already completed the first two years of the program. The overall conclusion was that the examination tested appropriate material, had a reasonable degree of difficulty, and was relevant to medical practice. Based largely on this feedback, the Student Evaluation Committee voted to include the Comprehensive Examination in the Basic Sciences as part of the evaluation system at the end of the second year.

The authors of the preceding letter have criticized this exam because (they claim) no other U.S. or Canadian medical schools give the exam to their students. In fact, the National Board of Medical Examiners has informed our office that 52 medical schools in the United States and Canada use the Comprehensive Examination in the Basic Sciences (they did not tell us which schools use it).

The scores of UBC students will be compared with results from 12,250 other medical students across North America. Individual scores, which are also reported as percentiles in comparison to the entire pool of test takers, will provide students with feedback on how they perform relative to this larger peer group. The faculty will obtain an external check on how well UBC students are learning the basic sciences, information that is simply not available from locally constructed examinations.

The cost of the Comprehensive Examination in the Basic Sciences is US $35 per student. There is no equivalent Canadian examination, and the only other externally developed exam available to us is the full United States Medical Licensing Examination, Step 1, which costs more than ten times as much. At that price, the Comprehensive Examination in the Basic Sciences is a bargain.

Finally, a few words about assessment of the new curriculum are in order. Over the past 31/2 years, the faculty has solicited and received student input about all aspects of the program. We have listened carefully and introduced a number of changes in response to those comments. Students also have a voice in governance of the curriculum through membership on committees and participation in faculty meetings. However, it is ultimately the responsibility of the faculty to ensure that UBC medical graduates have met standards that are recognized locally and nationally. To do this effectively, we must use both internal and external measures of student performance. The Comprehensive Examination in the Basic Sciences will help us to evaluate students, and the program as a whole, in a manner that is objective and fair.

—William E. Schreiber, MD
Associate Dean,
MD Undergraduate Program, UBC