This page last updated 06/14/99

Implementation and Programmatic Evaluation of Standardized,
Formal Evaluation System with Synthetic Descriptors (Reporter-Interpreter-Manager-Educator)

 

Question: is the system developed and validated at one clerkship in one school of sufficient power to be applicable across schools and clerkships.

 

Project: The GEA Project on Reliable and Valid Assessment is recruiting up to TEN medical schools to join an inter-school project. The purpose of the program is to establish the generalizability of this method for evaluation of students on clinical clerkships.

 

Eligibility:

All AAMC listed medical schools who can commit to (1) having at least three core clerkships (which need not be the same for each school) implement the system (described in more detail in this website), (2) collecting the pre/post implementation data (described below); (3) in-school review of masked student evaluations to determine "credibility" of evaluation with faculty, students and deans.

 

Characteristics of Credible Clinical Evaluation of Students

Formative Evaluation

 

Summative Evaluation

 

The project will provide training materials for the clerkship director to learn the system.

 

 

OUTCOME ISSUES AND MEASURES - credibility issues

 

Formative Evaluation

Issue                                                                              Evaluation method /data needed

1. Does it occur? documents of mid-way evaluation*
number of forms,
number of teachers,
teachers spoken with,
roles (intern, resident, etc) attending evaluation session
2. Does it reflect departmental goals panel review of masked evaluations
specific language
structured interviews with students’ teachers
3. Is feedback generated?
    is an action plan generated?		 Documentation that mid-way feedback occurred    questionnaires of students during clerkship
4. Does feedback anticipate final evaluation? masked panel review of evaluations*

 

Summative Evaluation

1. does it reflect departmental goals? Collection of clerkship final grades
masked review of student final grades*
2. is it broad based? review of student folders for number of teacher
3. consistent across sites, teachers, rotations inter-site grade distributions
masked review of student final grades by site*
4. is there documentation of core goals masked review of student final grades*

*compared to a control group of student evaluations from two years prior to implementation of new program

OUTCOME ISSUES AND MEASURES - Reliability and Validity; Feasibility

Reliability

1. inter-case correlation and generalizability coefficients of student evaluations across teachers for a given rotation and clerkship

2. inter-clerkship correlation and generalizability coefficients for student final grades

 

Face Validity

  1. survey of students before/after implementation (for acceptance )
  2. survey of faculty
  3. survey of deans
  4. survey of program directors reading deans’ letters

 

Content Validity

  1. systematic review of masked written evaluations from clerkships before/after for documentation concerning core goals
  2. survey of evaluation "users" - students, deans on helpfulness of evaluations

 

Predictive Validity

  1. survey of PGY1 directors, with ratings
  2. structured interviews with PGY1 directors for stratified sample of students
  3. fourth year sub-internship performance (if available)
  4. end-of-clerkship or end-of-clerkship year OSCE /practical examination

 

Construct Validity

sample evaluation of house officers using RIME vocabulary

sample evaluation of house officers with formal evaluation system

final examination scores (corrected for USMLE1) before/after system

 

 

Feasibility

  1. time and effort of clerkship directors; of clerical support
  2. time of teachers
  3. attendance rates