Mentor Evaluationprofessional Masters Programs in Software Engineering

Mentor EvaluationProfessional Masters Programs in Software Engineering

David Garlan

(Last revised 11/15/2016)

1. Background

A central component of the professional masters in software engineering (MSE) degree programs[1]at Carnegie Mellon University is a major project course. Project courses typically span multiple semesters, are carried out in teams, and account for a substantial portion of a student’s curriculum – typically up to a third of a program’s overall course units.

Critical to the effectiveness of such project-based courses are faculty mentors. Mentors act as coaches helping students mature in their practices, guiding their exploration of new techniques, asking the right questions at the right time, and working with both teams and individual students to make sure they take full responsibility for their decisions and actions.

As such, the role of a project mentor is very different from a typical classroom lecturer. Faculty-student Interactions are primarily through meetings and formal student presentations. Mentors may also work with students by reviewing the various artifacts that they produce during the project lifetime, including project documents (statement of work, architecture design, management plan, etc.), presentations (end-of-semester reviews, etc.), and final reflection reports.

For teaching track faculty in the MSE programs, mentoring often occupies a substantial portion of a faculty member’s time – often as much as 50% of their overall student-related activities. Hence it is particularly important for reappointment and promotion (R&P) cases to evaluate these activities (not to mention benefits to the faculty through feedback on performance).

However, evaluating faculty mentors presents a number of challenges. First, unlike traditional lecture-based classes there are typically few curricular materials that are developed or presented directly to students. Rather, faculty work with students to help them “learn by doing.” Second, a project typically spans multiple semesters. Thus, unlike traditional courses where it is natural to evaluate the faculty and course at the end of the semester in which it is taught, there are many more logical points when a mentor might be evaluated. Third the role of the mentor may vary considerably over the course of a project. At the beginning of a project, mentors may be focusing on team-building and start-up activities; in the middle,mentors may be guiding students through implementation phases of the project; at the end, mentors may be helping students to consolidate their experience through a reflection paper. Fourth, there is often high variance in the make-up of project teams depending on maturity, cultural background, prior knowledge, and general attitude of the project members. This can lead to a large variation in the roles that mentors play: for some teams mentors may have to exercise much more oversight and guidance than for other teams. Fifth, unlike faculty course evaluations (FCEs) for typical lecture classes, the population size of students who interact with the mentor is small – typically 5students each year. Hence it is important to balance student feedback with other information.

Because of the different nature of the educational role of mentors, and to address these challenges, it is reasonable to expect that mentor evaluation would require evaluation instruments that are specialized for thetask of measuring mentor performance. In this document we outline the current mentor evaluation process and the instruments that we use in carrying it out.

The plan for mentor evaluation described in this document evolved out of a series of mentor meetings and the recommendations that resulted from those meetings. Thanks especially to Dave Root, Tony Lattanze and Cliff Huff for their substantive contributions.

2. Qualities of a Good Mentor

Before outlining a specific approach to mentor evaluation, it is important first to be clear what the attributes of a “good” mentor are.

Technical – knowledge of core course materials, current technology, best practices, management techniques. Helps teams determine realistic scope for project; knows when to refer team to expertise (on campus, at the SEI, or in industry); provides insightful, actionable technical guidance; points out opportunities for students to leverage materials from their courses; helps teams identify risk areas and possible mitigation schemes.
Human Interaction – leadership, ability to facilitate communication, manage conflict, foster team cohesiveness. Uses team and individual students meetings to foster mutual respect; identifies mechanisms for conflict resolution; helps students bridge cultural differences and attitudes; is a resource that students seek out for help and guidance.
Professionalism – timeliness, frequency of meetings, other involvement in the professional programs. Meets with students regularly, both as a team and individually; attends important program meetings such as Black Friday, student progress reviews,all-mentor meetings, and end-of-semester presentations.

3. Considerations

The choice of an approach to mentor evaluation boils down to a set of decisions that address the following aspects of information collection: why, what, who, when, and how. To provide a rationale for our current mentor evaluation process, we consider each of these in turn.

3.1 Why

The most obvious reason to collect information about mentors is for the purpose of evaluation. Specifically, all regular teaching track faculty are evaluated at regular intervals, and evidence of teaching performance is critical to making R&P decisions.

But collecting information can also to help faculty improve their performance through constructive feedback. Unfortunately, evaluation and feedback are sometimes at odds. For example, information collected at the end of a “course” (or even later) is often most useful for evaluation, while information collected early may be more useful for incremental improvement.Perhaps more importantly, feedback may entail the disclosure of its source (even when done anonymously, since the number of responders is small), compromising candid responses needed for evaluation. Another related issue is that many mentors for MSE projects have adjunct appointments, and are not subject to R&P processes. Should they also be included in the evaluation process?

In this plan, we are primarily interested in evaluating performance for the purpose of teaching track faculty R&P. If the collected data can also function to improve the quality of mentoring, that is viewed as a secondary benefit.It is also worth mentioning that the information to be collected would be expected to complement, not replace, existing forms of faculty evaluation, such as letters from peers, FCEs for lecture-oriented classes, etc.

3.2What

Given the qualities of a good mentor, listed above, the information collected should provide feedback in each of the three categories: technical, human interaction, and professionalism.

3.3 Who

There are three obvious categories of people who might in principle provide information on the mentor’s performance.

The first category is students. Within this category are current students and former students. Requesting information from current students has the advantages that (a) information is easier to collect because they are within our environment, and (b) the information is likely to be timely, since students can reflect on recent experience with the mentor. However, there are also advantages to requesting information from former students: in many cases, the impact of a mentor may not be fully appreciated until a student has had the chance to put into practice some of the project’s lessons in an industrial setting following graduation. We collect information from both current and former students.

The second category is faculty. Within this category there are several important subgroups. First, there are “senior mentors.”Within the MSE programs we have designated a small set of seasoned mentors, who,due to their professional experience and demonstrated ability to effectively mentor MSE students,offer guidance to the entire team of mentors for improving overall mentoring practices.Senior mentors are ideal for providing an evaluation of mentor effectiveness based on a multi-year perspective and many years of experience. Second, there are “co-mentors.” Some projects (notably MSE projects) have two or more mentors who work together in coaching the students, jointly attending team meetings, advising individual students, etc. Given the close working relationship, these co-mentors have detailed information regarding the mentor’s effectiveness within the context of the specific project. Third, there are other mentors and faculty, who may observe the impact of the mentor through their interactions with students in other courses, through discussions at Black Friday, through advising, or through attendance at regular team presentations (such as end-of-semester presentations).We collect information from the first two subgroups– senior mentors (also called “uber-mentors”) and co-mentors (if any). The third subgroup is already covered by peer letters within the normal R&P process.

The third category of people isproject clients. Clients typically come from outside the university and have interests in the products produced by the team on the project. We do not collect information from this projectclients, as their visibility into mentoring is (at best) indirect. Further, it was felt that their evaluation would be overly biased towards the quality of the team’s output, rather than the learning that went on to produce that output.

3.4 When

As noted earlier, given the long timespan of projects (typically spanning multiple semesters), it is not obvious when information should be collected. While frequent reviews have the benefits of timely results, possibly leading to incremental improvement in mentor effectiveness, one must also consider administrative overhead and possible effects of diminishing quality from frequent requests for information. A further consideration is the fact that students may feel reluctant to provide candid evaluations midway through the project, for fear of faculty recrimination (justified or not).

We collect evaluations once per project from the current students. Currently we believe that the best time for collecting information from current students istoward the end of their project, when they have achieved the majority of their project goals. We also collect evaluations from former students one year after they have graduated. Finally, we collect evaluations from peer faculty (uber-mentors and co-mentors) on an annual basis.

3.5How

While the mechanisms for collecting information are not particularly important, the issue of confidentiality is. Existing R&P processes use different policies regarding confidentiality that depend on the form of review. Student evaluations, such as FCEs, are largely public. However, other kinds of evaluations (such as peer evaluations in R&P-related letters, or letters from students) are not.

Given that the primary purpose of this evaluation is for R&P, all evaluations will remain confidential.

4. Evaluation Summary

To summarize, the evaluation process for evaluating regular faculty mentors for R&P will consist of four parts.

Co-mentor reviews: Annually we will request written evaluation of the faculty member from any co-mentors with whom he/she has worked during that year.The request for evaluation will be specifically targeted to information related to mentoring quality. These reviews will be kept as a confidential part of the R&P dossier for the faculty member.A copy of the letter requesting this information is attached at the end of this document.
Senior mentor reviews: Similar to co-mentor reviews, a letter will berequested from at least one senior mentor for every R&P cycle. A copy of the letter requesting this information is attached at the end of this document.
Current student reviews: We will collect mentor evaluations from current students on the faculty’s projects towards the end of the project, using a questionnaire designed for this purpose. These evaluations will be kept confidential. A copy of the questionnaire is attached at the end of this document.
Former student reviews: We will collect mentor evaluations from former students approximately one year after a student has graduated. A copy of the letter is attached at the end of this document.

The following table summarizes the various instruments used:

Who / How / When / Confidential?
Faculty: co-mentors / Letter / Annually / Yes
Faculty: senior mentors / Letter / Once per R&P period / Yes
Students: current / Questionnaire / Once per project / Yes
Students: former / Letter / End of first year after graduation / Yes

6. Evaluation Instruments

Attachments: co-mentor evaluation request letter, uber-mentor request letter, former student letter, current student questionnaire.

SAMPLE LETTER SENT TO CO-MENTORS (Annual)

Dear <Co-mentor Name>:

A central component of the Professional Masters Programs in Software Engineering within the Institute for Software Research at Carnegie Mellon University is project-based courses, such as the MSE Studio and MSIT Practicum. Key to the effectiveness of such courses are faculty mentors. Mentors act as coaches helping students mature in their practices, guiding their exploration of new techniques, asking the right questions at the right time, and working with both teams and individual students to make sure they take full responsibility for their decisions and actions.

As part of the Reappointment and Promotion processes for Teaching Track Faculty, annually we request an assessment of each faculty member from the co-mentors who worked with that faculty member on a project course. I would be grateful if you could provide a letter with a candid evaluation of <Name>, who currently holds a five-year appointment at the rank of Associate Teaching Professor. Your letter is confidential, and willbe protected as such to the extent consistent with and permitted by law.

When writing your letter, I would ask that you consider the following general criteria of mentoring quality:

Technical – knowledge of core course materials, current technology, best practices, management techniques.
Human Interaction – leadership, ability to facilitate communication, manage conflict, foster team cohesiveness.
Professionalism – timeliness, frequency of meetings, other involvement in the professional programs.

We would appreciate receiving your letter by <date>. You may submit a pdf of your letter via email or by hard copy to the address below:

Victoria Poprocky

R&P Administrator

Carnegie Mellon University

Institute for Software Research

Pittsburgh, PA 15213

Thank you in advance for assisting us in this important endeavor.

Sincerely,

William Scherlis

SAMPLE LETTER SENT TO UBER-MENTORS (Once every R&P period)

Dear Uber-mentor Name>:

As part of the Reappointment and Promotion processes for Teaching Track Faculty, we request an assessment of each faculty member from one or more uber-mentors. I would be grateful if you could provide a letter with a candid evaluation of <Name>, who being considered for reappointment at the rank of Associate Teaching Professor. Your letter is confidential, and willbe protected as such to the extent consistent with and permitted by law.

When writing your letter, I would ask that you consider the following general criteria of mentoring quality:

Technical – knowledge of core course materials, current technology, best practices, management techniques.
Human Interaction – leadership, ability to facilitate communication, manage conflict, foster team cohesiveness.
Professionalism – timeliness, frequency of meetings, other involvement in the professional programs.

Also, please indicate the ways in which you interacted with the candidate (e.g., all-mentor meetings, EOSP presentations, Black Friday, etc.)

We would appreciate receiving your letter by <date>. You may submit a pdf of your letter via email or by hard copy to the address below:

Victoria Poprocky

R&P Administrator

Carnegie Mellon University

Institute for Software Research

Pittsburgh, PA 15213

Thank you in advance for assisting us in this important endeavor.

Sincerely,

William Scherlis

SAMPLE LETTER SENT TO FORMER STUDENTS (One year after graduation)

Dear Student Name>:

A central component of the Professional Masters Programs in Software Engineering within the Institute for Software Research at Carnegie Mellon University is project-based courses, such as the MSE Studio and MSIT Practicum. Key to the effectiveness of such courses are faculty mentors. Mentors act as coaches helping students mature in their practices, guiding their exploration of new techniques, asking the right questions at the right time, and working with both teams and individual students to help assure a valuable learning experience through the project.

As part of the Reappointment and Promotion processes for Teaching Track Faculty, annually we request an assessment of each faculty member from former students in project courses mentored by that faculty member. I would be grateful if you could provide a letter with a candid evaluation of <Name>, who was a mentor for your project when you were a student in our Masters Programs. Your letter is confidential, and willbe protected as such to the extent consistent with and permitted by law.

When writing your letter, I would ask that you consider the following general criteria of mentoring quality:

Technical – knowledge of core course materials, current technology, best practices, management techniques.
Human Interaction – leadership, ability to facilitate communication, manage conflict, foster team cohesiveness.
Professionalism – timeliness, frequency of meetings, other involvement in the professional programs.

Also, I would be particularly interested in hearing your thoughts about the ways in which the project and the mentor’s guidance have affected you professionally and otherwise, from the perspective of one or more years experience after completing the degree.