R3P Meeting
“Trade Show” Consultant: Eric Holmboe
Training Faculty to Evaluate Competence Effectively
Background
Although there is a need to develop new methods of assessment, medical education has access to a wide range of effective evaluation tools and methods now. Most of the major limitations of current tools and methods relate to how the evaluator uses the tool, not the tool or method itself. Work in the performance appraisal field over the last 25 years has emphasized the importance of evaluator (rater) training. Yet, medical education has been slow to adopt evaluator training methods for faculty. I will provide information about assessment methods and how to prepare faculty to use them most effectively. I will also provide information on evaluator training methods.
Evaluation Systems
The following pages provide information on various methods to train faculty in assessment skills. All of this should occur in the context of an evaluation system, defined as a group of people who work together on a regular basis to provide evaluation and feedback to a population of trainees, from students to fellows. This definition is adapted from a high functioning clinical microsystem. In an effective evaluation system, evaluators share common educational aims and outcomes, linked processes and information about trainee performance, and produce a student ready to enter residency, or a resident or fellow truly competent to enter independent medical practice at the end of training. This latter attribute requires faculty be competent evaluators. The evaluation system has a structure (faculty, curriculum and resources) to carry out processes (teaching, clinical experiences, evaluation and feedback) to produce an outcome (appropriate competence at the end of training. This system must provide both summative and formative evaluation for trainees, and at a minimum summative evaluation for the profession and public.
A systems approach provides the framework for effective evaluation. The underlying assumption of the systems approach is that all components must be identified and integrated efficiently and effectively into the evaluation process. Attention must be directed toward the purpose of the evaluation, the competencies to be assessed, the evaluators, individuals to be evaluated, the settings of the evaluations, the quality of the learning environment, the timing of the evaluations, the methods used to collect and summarize the information about each resident, and the management of information by the program.
BOX: Elements of an Effective Evaluation System· Clear purpose of the evaluations
· Clear definitions of the competencies to be evaluated
· Appropriate training and preparation of the evaluators
· High quality learning environment
· Timeliness of evaluations
· Reliable and valid processes to collect, summarize, and disseminate evaluation information
· Transparency
· Efficient management of information
This approach to evaluation optimizes the methods of collecting, acting on, and storing information about trainees. A systematic process strengthens the position of the educational leader to provide high quality feedback and to respond to appeals by candidates over adverse judgments. In addition, this strategy permits the eventual aggregation of data and feedback to clerkship and program directors about problems that trainees manifest, guides effective remediation for the problems identified, and provides the time frames in which improvements should occur.
Finally, a more structured evaluation system helps clerkship and program directors meet their responsibilities to the public and profession by encouraging recognition of early warning signals, facilitation of professional growth among trainees, clear decisions on the annual status of trainees, explicit documentation of problems and remedial attempts, and development of a final summary of the training and evaluation process.
Educational leadership
Leadership in evaluation is essential. The leader of an effective evaluation system should possess several characteristics. First, the leader must be willing to do whatever they ask others to do. Research from the quality improvement world supports that “leading by doing” helps to promote change and give credibility to the process. Second, leaders in an evaluation system must be knowledgeable about evaluation and feedback methodologies. One of our main goals with this book is to provide practical information to educational leaders and faculty that can be used to implement an effective evaluation program. Third, the leader needs to interact collaboratively not only with other faculty and trainees but also with nurses, administrators and other staff responsible for comprehensive evaluations of the trainees’ competence. Educating all of these groups helps to promote a better assessment environment. Fourth, the leader needs to take negative evaluations seriously. Failure to do so can have substantial untoward consequences for the evaluator who had the courage to bring forth a negative evaluation and may squelch further negative evaluations from the individual, as well as others in the program.
However, no single individual can manage all aspects of a successful evaluation system. Evaluation committees, often referred to as clinical competency committees, can be an effective and efficient mechanism to detect deficiencies early, provide real-time faculty development, and promote positive changes in the evaluation culture. The collective wisdom and decision-making capacity of committees provides helpful support of educational leaders and faculty members, at times if only to provide encouragement to stand by one’s convictions. In part, faculty can be insulated from making final decisions regarding trainee progress by referring ultimate decision making authority to medical education, student promotion or resident competency committees.
All committee members must be fully committed and not view this activity as simply another onerous task. The committee should not be a “rubber stamp” but should collect and review meaningful evaluation information to help all trainees’ progress and improve. A negative or disinterested climate on these committees, or breaches of trainee confidentiality regarding information shared at these meetings, can have a pernicious effect on the entire evaluation program.
Faculty development
Many individuals with different qualifications and diverse roles participate in the evaluation of residents: program directors, members of the evaluation committee, attending physicians who are full-time or voluntary clinical faculty, chief residents, fellows, senior residents, peers, nurses, medical students and potentially others in the education or healthcare environment. Evaluators should have sincere interest in their evaluative role; proper knowledge, skills, and attitudes for fulfilling the role effectively; and the time and opportunity to evaluate. Accurate evaluation and feedback is at its essence a “willing and able” phenomenon. The ability to identify deficits must be accompanied by the willingness to report them. Program leaders must be willing to make difficult decisions and support faculty members in providing negative assessment.
An evaluation instrument or method is only as good as the individual using it. Evaluator training is absolutely essential in any effective evaluation system. Methods and tools do not magically produce an evaluation – they are simply provide a means to guide, collect and document the judgment of the evaluator about the trainee, whether it be a standardized patient (using a checklist) to a peer (completing a survey). We find it somewhat ironic that a standardized patient undergoes hours of training to provide a reliable and valid rating for a systematized patient encounter while faculty receive no or little training for judging complex trainee interactions with unpredictable real patient encounters. Faculty development and support is essential to maintaining the health of the program. Faculty need to be trained to accurately appraise trainee performance and supported by program leaders when accurate assessment leads to negative learner feedback or decisions. The techniques for helping individuals evaluate more effectively have been described throughout the book and we strongly encourage you to systematically incorporate faculty training in evaluation. We provide additional detail below.
Trainee Engagement
The evaluation system should be transparent to all of those involved in the assessment process, including the trainees. Transparency leads to an impression of fairness. An explicit link with educational objectives and clear understanding of the assessment goals and methods create an impression of fairness that is essential to acceptance of the results and incorporation of feedback. Students, residents and fellows should clearly appreciate the dual nature of the assessment process as it relates both to the provision of feedback to encourage their continuous development and how it supports judgments about their progress and readiness to advance in their training. Trainees should understand that clerkship and program directors are accountable to the public and to the profession to make important decisions about their attainment of professional standards and preparedness to deliver safe and effective care. The purpose of evaluation, and its relationship to the educational objectives and their educational experiences, should be explained at entry to the program, and reiterated on a regularly scheduled basis.
As key stakeholders or “customers” of the educational process, a sound argument can be made for including trainees in decision making about the assessment process and in providing feedback on its effectiveness. Indeed most residents feel that their participation in evaluation of the educational program and its faculty are important. While published data suggest that residents could stand to learn more about the evaluation methods used in their programs, they are able to identify areas where assessment approaches to specific skills are inadequate. Residents are aware that accurate evaluation of their clinical competence requires a multi-modal approach and that different assessment methods capture different domains; they recognize that commonly used tools may not be ideal for measuring PBLI and SBP and are able to see the potential value of methods (360o assessment for example) for which they have no personal experience.
Similarly, students are able to understand the need for multiple assessment tools in measuring their competence and are able to identify the relative value of individual methods for measuring specific aspects of competence. Furthermore, students are able to recognize when assessment practices are driving learning approaches that may be maladaptive and not ideally suited to the particular educational content or conducive to durable acquisition of vocationally-relevant knowledge. The abilities of trainees to understand and meaningfully participate in the evaluation process should not be underestimated and opportunities to include them should not be overlooked, particularly since ability to engage in peer assessment constitutes an important educational and professional achievement.
Table 1: Methods to Train Faculty in the General Competencies
Training Method / Rationale and Description / ExamplePerformance Dimension Training
(PDT) / Rationale: Ratings can be improved if evaluators understand the dimensions and/or elements of a competency.
Description: Familiarize faculty with appropriate performance dimensions or standards to be used in evaluation by reviewing the dimensions of a specific competency. Faculty work in small groups to improve their understanding of these definitions with review of actual resident performance or clinical evaluation vignettes. / Faculty discuss the elements of what constitutes an “effective utilization of resources” by a resident discharging a patient who needs physical therapy and follow-up (systems-based practice).
Frame-of-Reference Training
(FoRT) / Rationale: Get evaluators to shared meaning of performance dimensions by teaching them how to use the evaluation tool. Specifically:
· What are dimensions of performance? (PDT)
· Does everyone agree on the dimensions?
· What is the meaning of different ratings? (e.g. low versus high)
Description: After completing PDT, faculty define what would constitute “satisfactory” performance, followed by the opportunity to practice evaluating residents performing at various levels of competence using the evaluation instrument of choice. The group then discusses reasons for the differences between faculty ratings. / Faculty are given several vignettes along with examples of the medical record, regarding a discharge performed by this resident. For each vignette, the faculty rate the level of performance (unsatisfactory, marginal, satisfactory, and superior). The vignettes provide examples of different levels of competence in systems-based practice. After each rating, the group discusses the ratings given. This exercise helps to “calibrate” faculty to be able to discriminate between different levels of competence.
Rater Error Training
(RET) / Rationale: By teaching evaluators about halo and distributional errors they will be more likely to recognize these errors in their own ratings.
Description: Faculty discuss the common errors (such as halo effect or compensation fallacy) in ratings. Each error is described and defined / Examples of each error are provided for discussion and review. Actual examples from the program could be used.
Behavioral Observation Training
(BOT) / Rationale: Improve observational accuracy and recall of behavioral events
Description: Provide tools to assist in guiding the observation and documentation of specific events related to competence. / Many tools can serve as observational diaries, such as the miniCEX. Even simple aides as a 3X5 card can be used to record events and observations
Direct Observation of Competence Training
(DOC Training) / Rationale: Improve the quality (validity and reliability) of the direct observation of clinical skills (patient-trainee interactions).
Description: A systematic process that starts with a PDT exercise (define dimensions of a clinical skill such as counseling), followed by a frame of reference exercise to differentiate levels of performance. The focus is on observable behaviors. After the PDT and FoRT, the participants practice observation skills with videotapes and if available, live standardized residents and patients. / Faculty develop the criteria for an effective counseling session, followed by defining the minimum criteria for a satisfactory performance. Faculty then apply this learning by observing and evaluating trainee-patient counseling sessions scripted at different levels of performance. Faculty work in small groups to discuss differences in ratings and the reasons for those differences.
Table 2: Methods for Assessing the ABMS/ACGME General Competencies Across the Continuum of Education and Practice.
Undergraduate
Medical Education / Graduate Medical Education / Clinical
Practice
Medical Knowledge / Local Examinations
NBME Subject Examinations
USMLE Step 1 and 2CK / Local Examinations
USMLE Step 3
Specialty In-training Examinations
Board Certifying Examinations / Re-certification Examinations
Examinations linked to CME/CPD
Interpersonal and Communication Skills / Direct Observation
SP Examinations
Global Ratings / Direct Observation
SP Examinations
Global Ratings / 360o Ratings using Multi-Source Feedback (MSF) or Individual Ratings by Peers, Patients or Supervisors
Professionalism / Direct Observation
SP Examinations
Mult-Source Feedback (MSF) / Direct Observation
SP Examinations
Multi-Source Feedback (MSF) / Multi-Source Feedback (MSF)
Licensure / Credentialing Actions
Maintenance of Certification / ABMS
Board Actions
Patient Care / Direct Observation
SP Examinations
Global Ratings
Medical Record Audit / Chart
Stimulated Recall / Direct Observation
SP Examinations
Global Ratings
Medical Record Audit / Chart
Stimulated Recall / Medical Record Audit / Chart
Stimulated Recall
Measurement of Healthcare
Processes and Outcomes
Licensure / Credentialing Actions
Practice-Based Learning and Improvement / QA / PI Projects with others
EBM Exercises
Chart Stimulated Recall
Portfolios
Personal learning projects / QA / PI Projects
EBM Exercises
Medical Record Audits / Chart Stimulated Recall
Portfolios / QA / PI Activities
Measurement of Healthcare
Processes and Outcomes
Portfolios
Systems-Based Practice / SP Examinations
Medical Record Audit / Chart
Stimulated Recall / Multi-Source Feedback (MSF)
Medical Record Audit / Chart
Stimulated Recall / Multi-Source Feedback (MSF)
Measurement of Healthcare
Processes and Outcomes (focus on
cost-effectiveness & resource
utilization)