Evaluating the Development of Professional Skills:
A Vocabulary and Method for the Descriptive Evaluation of Students in Clinical Clerkships

(Submiitted for publication)

word count: 2828

Louis N. Pangaro, M.D.

Vice-chairman for Educational Programs

Department of Medicine

Uniformed Services University of the Health Sciences

Bethesda, Maryland 20814-4799

phone: 202 - 782 - 4923

fax: 202 - 782 - 7363

Abstract

Credible, consistent evaluation of medical students in their patient care activities can be strengthened . Articulation and application of a clear, cogent and concise vocabulary of expectations is an useful strategy for evaluating the professional growth of medical trainees, especially students. This paper presents a framework and taxonomy for describing the transitions of a medical trainee from Reporter to Interpreter to Manager to Educator. The second part of the paper describes characteristics of a process of descriptive, clinical evaluation that models professionalism even as it evaluates it in students. The paper describes how one third-year Internal Medicine clerkship uses the framework in a formal process to promote credible descriptive evaluation.

Introduction

Research in the global, descriptive, clinical evaluation of medical students and residents has lagged behind advances in discrete areas of grading, such as extended matching multiple choice questions for knowledge and the use of standardized patients for the assessment of interviewing and physical examination skills. In the clinical years of medical school students work intensively under the supervision of faculty and house staff who are invited to contribute to the evaluation of the students they supervise. Yet, there has been a tendency to regard these "clinical" evaluations of trainees as "subjective". Although considerable work has been done to develop instruments which can more precisely quantify the performance of students under test conditions, much more needs to be done to refine the process of clinical evaluation, especially for values and behaviors. What a student is able to do for the short term of a final examination is not the same as what the student will habitually do for sustained periods of time, and often test results are not available for immediate feedback to students. Hence the need and benefit of giving more rigor and credibility to the evaluations of those who work with a student on a day-to-day basis. Finally, quantified final examinations, using standardized patients and multiple choice tests, cannot replace faculty responsibility for deciding on the fitness of a student to enter our profession. Articulation and application of a clear, cogent and concise vocabulary of expectations is an essential strategy for capturing and cultivating the unique, direct observations of the teachers - who may vary widely in experience and interest - about their students. The first part of this paper presents a framework and terminology for describing the professional growth of medical trainees, especially students. The second part of the paper describes criteria for the process of clinical evaluation, and a specific example of how the framework can be used in a formal evaluation process to promote honest standards. This combination has been part of the faculty development process in our institution, and has been used for the evaluation of more than two thousand students at the Uniformed Services University of the Health Sciences over the last decade, with sufficient reliability and validity for "high stakes"^, decisions.

I: A Vocabulary for Evaluating the Core of Professionalism

The R.I.M.E. Evaluation Framework.

We describe student performance goals using the following progression: Reporter, Interpreter, Manager, Educator (R.I.M.E.). The framework emphasizes a developmental approach, and distinguishes between basic and advanced levels of performance for both ward and clinic rotations. Each step represents a synthesis, a final, "common pathway" of skills, knowledge and attitude.

"Reporter": the student can accurately gather and clearly communicate the clinical facts on his/her own patients. Mastery in this step requires the basic skill to do a history and physical examination and the basic knowledge to know what to look for. It emphasizes day-to-day reliability, for instance, being on time, or follow-up of a patient’s test results. Implicit in the step is the ability to recognize normal from abnormal and the confidence to identify and label a new problem. This step requires a sense of responsibility, and achieving consistency in "bedside" skills in dealing directly with patients. These skills are often introduced to students in their preclinical years, but now they must be mastered as a "passing" criterion.

"Interpreter": Making a transition from "reporter" to "interpreter" is an essential step in the growth of a third year student, and often the most difficult. At a basic level, the student must prioritize among problems identified in their time with the patient. The next step is to offer a differential diagnosis. Because a public forum can be intimidating to beginners, and third year students cannot be expected to have the "right answer" all the time, we define success as offering at least three reasonable diagnostic possibilities for new problems. Follow-up of tests provides another opportunity to "interpret" the data (especially in the clinic setting). This step requires a higher level of knowledge, more skill in selecting the clinical findings which support possible diagnoses and in applying test results to specific patients. The student has to make the transition, emotionally, from "bystander" to see himself/herself as an active participant in patient care.

"Manager": This step takes even more knowledge, more confidence and more judgment in deciding when action needs to be taken, and to propose and select among options for patients. Once again we can’t require students to be "right" with each suggestion, so we ask them to include at least three options in their diagnostic and therapeutic plan. A key element is to tailor the plan to the particular patient’s circumstances and preferences.

This evaluation framework has been used for ten years in the internal medicine clerkship of the Uniformed Services University of the Health Sciences F. Edward Hebert School of Medicine. It has been useful in both ward and clinic rotations. It has been a key part of a strategy for achieving consistency in descriptive evaluation across all our clerkship sites, across evaluators, month-to-month and year-to-year. On the premise that explaining goals to learners and teachers is an essential step in defining the curriculum and supporting consistent evaluation, we sought a framework that is economical yet credible with a wide range of evaluators.

The R.I.M.E. system is fundamentally synthetic, not analytic. Rather than separating evaluation of knowledge from skill or behaviors, it integrates student achievement. It provides a basic structure that teachers can readily use for putting student problems in perspective. For instance, if a third year student is a consistently reliable "reporter", but not yet interpreting, we can address whether the student lacks the knowledge, the confidence or the opportunity to offer an assessment of patient findings. In a sense, the R.I.M.E. framework is a step towards a "taxonomy" of "descriptive" evaluation which often suffers the label of "subjective" when compared with "quantitative" final examination methods which are considered more "objective".

Goal setting should specify the expected level of proficiency for each level of training, and explicitly state the passing criteria. The "Curriculum growth matrix" (Table 2) links the two explicitly. Students and faculty should clearly understand and agree upon the required, core performance goals for the level of training. A "passing" grade should not be predicated on mastery of a skill that has just been introduced. For example, most schools introduce interviewing in the first year, but the expected performance standard ("passing grade") does not include proficiency in the first year. However, professional standards might well require demonstration of competence as a reporter before leaving the core clerkships, and to show independence as an "interpreter" prior to internship. Within the third year clerkship setting itself, the matrix provides a system that we have been able to apply to both ward and clinic, and thus to unify goals and grading criteria ward and clinic experiences. Simply, a student must be a consistent "reporter" to achieve an unqualified grade of "Pass" (with no restrictions on their fourth year schedule). Consistency as an "interpreter" is required for a "High Pass" grade and as a "manager" for "Honors". If the faculty member has limited opportunity to observe the student in the care of patients (for example, is a tutor for a small group), then the criterion of "educator" may be more appropriate for "Honors". Each school or department can enter on the matrix when core abilities are introduced, practiced to be achieved with proficiency in its own curriculum, and evaluation criteria are automatically defined. We feel that demonstration of higher skills(e.g. educator) does not merit a higher grade unless earlier stages have been met with consistency.

The R.I.M.E. progression provides a brief, cogent taxonomy of goals that is easily grasped and used by teachers at all levels, from interns supervising students to faculty describing the progress of residents and even fellows. Its simplicity is part of its virtue, yet it provides more precision than a "gestalt", and a more sustained picture than the snapshot provided by tests..

II: An Evaluation System for Modeling Professionalism

The R.I.M.E. evaluation framework provides a vocabulary of professional development that can be used for evaluation and feedback. Its simplicity of use in clinical evaluation avoids the uncertainty about goals for trainees that can be a barrier to successful evaluation. Faculty should not evade the responsibility for grading, and academic leaders must invest the necessary resources in order to make the process of descriptive evaluation credible, in other words to avoid inconsistency and the charge of subjectivity. In the second part of this essay, we discuss the nature and kinds of evaluation, and describe essential characteristics of formative and summative evaluation. One method of achieving consistency is described.

In his influential 1983 essay on feedback, Ende distinguishes between feedback and evaluation. Feedback presents students with factual observations intended to help them improve, and is therefore, "formative". Evaluation conveys a final grade at the end of an educational experience; it is normative, judgmental and "summative". I believe that assigning "value" to student performance is implicit in feedback as well as in evaluation, and that desirable characteristics of formative and summative evaluation can be elaborated. The methods chosen for evaluation - for instance whether they are honest, fair and consistent - are a demonstration to students of what our own values are.

Formative evaluation: the purpose of formative evaluation is to provide the observations and conclusions that are the basis for immediate feedback, that is, to improve student performance. Ende has described the characteristics of effective feedback in detail, and several are applicable to formative evaluation: it should be well-timed, expected as part of the teaching process and based on first-hand observations. Additionally, we should add that evaluation of professionals is a "high-stakes" situation; it should be "accurate", that is, observations about student performance should be structured according to institutional goals rather than the personal beliefs of the observer. Next, the evaluation framework should be used with enough consistency from one group of teachers to another that the student can rely on the process. Third, it should allow time for improvement (if needed or if desirable) prior to summative evaluation. Fourth, it should have an action plan that is both concrete and expressed in the same evaluative framework as the observations.

Summative evaluation: Ultimately, the purpose of summative evaluation is to fulfill a societal responsibility - to guarantee the competence of graduates. As such it a reaffirmation of our professional values. Additionally, summative evaluation provides students and deans with information for on-going modification of curricular goals, for career guidance and for use in letters of recommendation. This final use may lead to a lack of candor from the faculty, not wishing to "harm" students in our temporary care. Yet the value and credibility of these "grades" also reflect the professional values of our community, and deserve the most through attention.

Summative evaluation must be accurate in the same sense as formative evaluation: it must reflect institutional or departmental goals that have been clearly understood by all parties. Second, summative evaluation should be broad-based and should not depend on a single teacher or testing method. It must incorporate the observations of multiple evaluators who have directly observed the student. Third, the process and the evaluative framework must have sufficient consistency from month to month, site to site and evaluator to evaluator. (Grades should not depend on where a student happened to do the rotation.) Final evaluation should employ the same framework as midway feedback, so that a student can rely on the value of the information in helping improve performance and the final grade. Fourth, the process should include documentation of key items: observations about the student, about the evaluative framework and of the process for feedback to the student, and about the student’s subsequent performance. For students who have not met all competence standards, or for whom there uncertainty about a grade, the procedures for departmental review should also be specified. In the written final grade additional details of quantified final examinations should also be specified. Use of more than one technique - drawn from a supplementary triad of multiple choice exams, standardized observation of interviewing and examination skills using simulated patients, and free response examinations of analytic ability and problem solving - is also recommended. The use of quantified examinations in the evaluation process has been described in more detail elsewhere.

Our strategy for consistent evaluation is based on the R.I.M.E. framework for describing student progress described above. It is easily understood and remembered by students, house staff and faculty. We orient teachers to the framework as they begin a rotation with students, and we distribute evaluation forms which use detailed "descriptors" to guide their ratings. However, we do not rely solely on these techniques. For many years we have employed a unique method to manage the evaluation process: the formal evaluation session.

Formal evaluation sessions

At each rotation site the clerkship director sits down with a student's group of teachers (including house staff for ward rotations) to discuss a student’s progress. Each gives a brief summary of a student’s strengths and areas for improvement, then the director asks specific questions. For example, if an intern recommends a grade of Honors because a student is a great "team player", always staying late to help with the "scut" work, the director may ask if the student is a reliable "reporter" for day-to-day findings and making the transition to "interpreter". Fifteen minutes are scheduled per student. Since development of teaching skills is accomplished at the same time as the evaluation process, the student is not present. We run these sessions at least monthly during our three month clerkship. The total time invested is about one hour per student per clerkship, divided among the directors for two consecutive clerkship sites.

Our system of formal evaluation sessions permits trained clerkship coordinators to guide evaluators in applying departmental goals consistently. With an additional investment of fifteen minutes per student, individual feedback is given the next day (and response to feedback becomes a subject of future evaluation). The sessions have doubled our "sensitivity" in detecting students’ knowledge deficiency compared with written evaluations. Even more importantly, the interaction develops house staff and faculty as observers of student performance, giving them feedback on their evaluations. Finally, the system allows for an action plan to be formulated during the session - e.g. adjusting one student's patient load, or coaching another in case presentation. The key question asked of each evaluator is "What’s the next step for this student?" For most students, areas of weakness are only a function of their status as beginners, and are seen as judgments on character. For the very few students whose work is below standards even after feedback, their problems are honestly identified and detailed. The process increases the both the quantity and "quality" of comments about students’ demeanor (i.e., comments are more likely to indicate what a student can do to improve). Helping individual students to meet the professional standards for their level of training is an explicit goal of the evaluation session. We feel it is both rigorous and humanistic.

This evaluation system places "values" at the core of the process. Expectations are explicitly communicated and applied at each step. While some may be troubled by asking faculty to "label" or "diagnose" a student’s level of performance, the process need not be adversarial any more than working with our patients. The use of the R.I.M.E. framework and evaluation sessions are only two options for achieving fairness and consistency. But consistency and honesty about standards are themselves part of professionalism. What we evaluate and how we do it, define and teach what we think of ourselves as a profession. Evaluation is the core of professionalism.

2. Pangaro LN, Jamieson T, Hemmer P, Gibson KF, DeGoes JJ, Descriptive Clinical Evaluation Can Achieve Reliability Comparable to Standardized Tests, Association for Medical Education in Europe, 1997, Vienna, Austria.

3. Skeff KM, Stratos GA, Berman J, Bergen MR, Improving Clinical Teaching. Evaluation of a National Dissemination Program, Arch Intern Med (1992) 152: 1156-61.

4. Bloom BS, Taxonomy of Educational Objectives, Handbook 1, Cognitive Domain, Longman, New York 1956.

5. Pangaro LN, Gibson K, Russell W, Lucas C, Marple R, A Prospective, Randomized Trial of a Six-week Ambulatory Internal Medicine Clerkship, Acad Med 70(1995): 537-541.

6. Ende J, Feedback in Clinical Medical Education, J Amer Med Assoc, 250(1983): 777-781.

7. Hunt DD, MacClaren CF, Scott CS, Chu J, Leiden LI, Characteristics of Dean’s Letters in 1981 and 1992, Acad Med 68(12) (1993): 905-911.

8. Parenti C, A Process for Identifying Marginal Performers Among Students in a Clerkship, Acad Med 68(7)(1993): 575-7.

9. Elnicki DM, Ainsworth MA, Magarian GL, Pangaro LN, Evaluation Techniques for the Internal Medicine Clerkship: A CDIM Commentary, Amer J Med: 97(1994): I-VI.

10. Noel GL, A System for Evaluating and Counseling Marginal Students, J Med Educ 62 (1987):353-355.

11. Hemmer P, Pangaro L, The Effectiveness of Formal Evaluation Sessions during Clinical Clerkships in Better Identifying Students with Marginal Funds of Knowledge, Acad. Med (1997): 72: 641 - 643.

12. Pangaro LN, Hemmer P, Gibson KF, Holmboe E, Formal Evaluation Sessions Enhance the Evaluation of Professional Demeanor, 8^th International Ottawa Conference on Medical Education and Assessment, Philadelphia, July 1998.

13. Stobo JD, Blank LL, ABIM’s Project Professionalism: Staying Ahead of the Wave, Amer J Med (1994): 97, I-III.

I = introduced in the curriculum

P = practiced, repetition

Table 1

Reporter: consistent in interpersonal skills, reliably obtains and communicates clinical findings

Interpreter: prioritizes and analyzes patient problems

Manager: consistently proposes reasonable options incorporating patient preferences

Table 3

• based on direct observation of teachers
• accurately reflects institutional or departmental goals
• reliably occurs at key points in rotation
• develops a plan for improvement
• anticipates same framework as final evaluation

Table 4

Characteristics of the USUHS Formal Evaluation Session