This page last updated 04/03/02

Bibliography for Descriptive Evaluation using the RIME Framework and Formal Evaluation Sessions

          1. Pangaro LN. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad. Med.1999; 74:1203 –7.

-        [This article addresses innovative approaches to making the ongoing in-training evaluation (ITEv) of trainees during their clinical experiences more reliable and valid. The innovations include the development of a standard vocabulary for describing the progress of trainees from "reporter" to "interpreter" to "manager" and "educator" (RIME), the use of formal evaluation sessions, and closer consideration of the unit of clinical evaluation (the case, the rotation, or the year, including Turnbull’s method of “work sampling”.). Also includes initial results of studies assessing the reliability and validity of descriptive methods, as well as the use of quantified methods to complement descriptive methods.]

2.     Pangaro L. Investing in descriptive evaluation: a vision for the future of assessment. Med Teach.2000; 22:478 -81.

-        [emphasis on synthetic evaluation; competence defined as “ consistently giving to each situation that a professional might face all that properly belongs to that situation, and no more”.]

3.     Noel G. A system for evaluating and counseling marginal students during clinical clerkships. J Med Educ [Acad Med 1987] 62: 353-355.1987.

-        [original description of formal evaluation sessions as method of clinical evaluation.]

RELIABILITY

4.     Roop S, Pangaro L, Measuring the Impact of Clinical Teaching on Student Performance during a Third Year Medicine Clerkship, Amer J Med. 110(3), 205-209, 2001.

-        [study demonstrating impact of teaching [through student ratings] on student growth across 12 weeks of clerkship; reliability of teachers’ evaluations of students > 0.8, sufficient for high stakes decisions]

5.     Pangaro L, Jamieson T, Hemmer P, Gibson KF, DeGoes JJ. Descriptive evaluation of clinical performance can achieve  reliability comparable to standardized tests . Proceedings of the Association for Medical Education in Europe, Vienna, Austria, 1997.

-        [different data set from ref#4, Roop; ratings (typically 8 per student) were analyzed for reliability  467students; inter-class alpha =.85.]

VALIDITY

6.     Battistone MJ, Pendeleton B, Milne C, Battistone ML, Sande M, Hemmer PA, Shomaker TS, Global Descriptive Evaluations Are More Responsive than Global Numeric Ratings in Detecting Students' Progress during the Inpatient Portion of an Internal Medicine Clerkship , Acad Med 2001 76: S105-S107.

-        [The R-I-M-E descriptors demonstrated greater changes in mean ratings over subsequent evaluation sessions than did the global numeric method. The changes in R-I-M-E ratings were statistically significant (p <.05 for both residents' and attending physicians' evaluations) across all three evaluation sessions, while numeric ratings did not consistently change until the third session. In contrast to the numeric system, descriptive evaluations were distributed more normally and had greater range; this finding was observed at each evaluation session. The frequency analysis also demonstrated a rapid "ceiling effect" for the numeric ratings, beginning at the first evaluation session and persisting over the subsequent evaluations.]

7.     Hemmer PA, Pangaro L. The effectiveness of formal evaluation sessions during clinical clerkships in better identifying students with marginal funds of knowledge. Acad Med.1997; 72:641 -3.

-        [For the checklist descriptors, ratings of "marginal" identified three of these 16 students (a sensitivity of 19%). For the written comments on the evaluation form, ratings of "marginal" identified four of the 16 (a sensitivity of 25%). For the formal evaluation session, however, ratings of "marginal" identified seven of the 16 (a sensitivity of 44%). Although the ability of housestaff and faculty to identify students with weak funds of knowledge may be less than ideal, it may be improved by the routine use of a formal evaluation session.]

8.     Hemmer PA, Hawkins R, Jackson JL, Pangaro L. Assessing how well three evaluation methods detect deficiencies in  medical students' professionalism in two settings of an internal medicine clerkship. Acad Med. 2000;75:167

-        [From 1994 to 1997, 18 students at The Uniformed Services University of the Health Sciences failed to satisfactorily complete their core 12-week third-year internal medicine clerkship due to deficiencies in professionalism. Three evaluation methods had been used to assess all students' professionalism during the two rotations of their clerkship: standard checklists, written comments, and comments from formal evaluation sessions. In the clerkship studied, deficiencies in professionalism of such magnitude as to require remediation were more likely to be identified in the inpatient than in the ambulatory care setting. Of the three evaluation methods studied, the face-to-face, formal evaluation sessions significantly improved the detection of unprofessional behavior in both clerkship settings.]

9.     Lavin B, Pangaro L. Internship ratings as a validity outcome measure for an evaluation system to identify inadequate clerkship performance. Acad Med . 1998;73:998 -1002.

-        [Responses to questionnaires from internship program directors were available for 75 of 97 remediators (78%) and 268 of 313 non-remediators (86%). The remediators were 12.9 times more likely to have low internship performance scores and 9.4 times more likely to receive unfavorable comments than were the non-remediators. However, the majority of the remediators (80%) received only favorable comments. The medicine clerkship grade was more sensitive than the non-medicine grade-point average in predicting problems during internship (75% vs 8%).]

FEASIBILITY

10.  Battistone MJ, Milne C, Sande MA, Pangaro LN, Hemmer PA, Shomaker TS. The  feasibility and acceptability of implementing formal evaluation sessions using descriptive vocabulary to assess student performance on a clinical clerkship. Teach Learning Med 14(1): 5-10, 2002.

-        [Attendance reproduces USU experience and was high for residents (79%) and faculty (72%). Mean survey responses from residents and faculty rated the descriptive system "more valid" than the previous method. Time requirement for eight to ten students at each teaching site, for evaluation and feedback sessions was one-half day per week of the clerkship director, every three weeks.]

11.  Hemmer PA, Pangaro L. Can a descriptive evaluation system detect student growth during a clerkship? Using descriptive evaluation to detect student growth. Proceedings from Annual 2000 Meeting of the Clerkship Directors of Internal Medicine, Teach Learn Med , 13:199-205, 2001.

-        For the 1993-96 classes at USU, 343 third year medical students did 12 weeks of inpatient medicine.  We compared the mean final ratings given in the second six weeks to the first six weeks (t-test).   Differences due to pre-clinical GPA, USMLE step one, and clerkship pretest scores were tested by ANOVA. There was no difference in students’ academic characteristics.  Instructors gave a higher percentage of available points in the second six weeks compared to the first six weeks. Overall growth rates did not differ by academic quarter. Accepting the construct that students do improve, these findings validate our criterion-based evaluation in which higher achievement represents progress beyond “reporting”. For grading, weighing student performance in the second six weeks more heavily than the first appears justified.

12.   Pangaro LN, Gibson K, Russell W,  Lucas C,  Marple R,  A Prospective, Randomized Trial of a Six- week Ambulatory Internal Medicine Rotation,  Academic  Medicine,  70 : 537- 541, 1995.

-        [descriptive evaluation used in programmatic evaluation, as one outcome for prospective randomized trial if new ambulatory rotation.]

FACULTY DEVELOPMENT

13.  Hemmer PA, Pangaro L. Using formal evaluation sessions for case-based faculty development during clinical clerkships. Acad Med. 2000;75:1216 -21.

-        [the third-year internal medicine clerkship at the Uniformed Services University uniquely incorporates faculty development into the process of evaluation and generating feedback for students. Formal evaluation sessions are held monthly at all clerkship sites throughout the 12-week clerkship and are moderated by either the internal medicine clerkship director or the on-site clerkship directors. Although designed to provide an opportunity for faculty to evaluate student performance and prepare formative feedback, the sessions also function as formal, planned, and longitudinal forums of "real-time," "case-based" faculty development that address professional, instructional, and leadership development. The evaluation sessions are used as a means to model and teach the key concepts of the Stanford Faculty Development Program. ]

14.  Pangaro L, Expectations of and for the medicine clerkship director. Am J Med 1998 Nov;105(5):363-5

-        [Running formal evaluation sessions requires time and clerical support for the clerkship director. This position paper’s recommendations include evaluation tasks and resources for the clerkship director. On average, clerkship directors report spending 28% of their time on clerkship administration . The Clerkship Directors in Internal Medicine (CDIM) recommend that 25% should be considered a minimum estimate of time for the "essential" administrative aspects of running a clerkship, not including time spent teaching students in seminars and lectures or in patient care activities.]

USUHS Home Page                         Dept. of Medicine Home Page   

The Dept. of Medicine Web Site is maintained by Sol Levy.   Please send me your suggestions and comments.