The Impact of Performance Assessment and Electronic Portfolios on Teacher Learning and Education
Raymond L. Pecheone
Stanford University
Matthew Pigg
University of California Office of the President
Ruth R. Chung
Stanford University
Randall J. Souviney
University of California, San Diego
28 January 2005
Submitted for publication in March/April 2005 Volume of The Clearing House
The Impact of Performance Assessment and Electronic Portfolios on
Teacher Learning and Teacher Education
A number of developments in the assessment of teachers and teaching in recent years provides opportunity to examine the potential effects of performance-based assessments on the professional learning of teachers in the short- and long-run. In Connecticut and California performance-based assessments have been implemented as part of the initial licensure process. Teachers nation-wide have participated in the National Board’s assessments, and the use of performance assessments in teacher education programs and pilot testing of the Praxis III (ETS) and INTASC assessments in other localities also have yielded insights into the impact of assessment on teacher education programs and state licensing systems. In the end, of course, all assessment of teachers must rest on examining their effectiveness in raising student achievement, and recent work gives us reason to believe teacher assessment has the potential to not just measure, but also promote, effectiveness.
In 1998, California elected to require teacher preparation programs to use performance assessment, along with other measures (e.g., coursework, observations), in making credentialing decisions The Commission for Teacher Credentialing contracted with Education Testing Service (ETS) to develop a standardized performance-based assessment instrument. Teacher education institutions were given the option of developing an alternative to the ETS model if the alternative met both the California teaching standards as well as rigorous reliability and validity standards. A total of 16 higher education institutions formed the PACT (Performance Assessment for California Teachers) consortium to collaborate on the development of a content-specific, standards-based alternative to the State assessment system. A key motivation for the PACT consortium is to develop a set of rigorous, transparent, subject-specific, standards-based certification assessment instruments that are consistent with the curricular and professional commitments of the member institutions and that meet State criteria for quality assessment.
The PACT consortium is thus attempting to contribute to professionalizing the teacher preparation process (see Mayer, this volume) by developing an assessment instrument within the teacher education community as a viable alternative to an externally created ETS model, which indeed meets state assessment criteria but may not reflect institutional commitments. Having piloted and scored more than 1500 assessments in 2002-04, PACT’s effort to create aset of subject-specific capstone instruments is a well-advanced model and has served as the basis for establishing the reliability and validity of the PACT Teaching Event (TE).
The design of the PACT Teaching Event (TE) is based on a Planning, Instruction, Assessment, and Reflection (PIAR) model in which candidates use knowledge of students’ skills and abilities as well as knowledge of content and how best to teach it in planning, implementing, and assessing instruction. Candidates complete several entries organized around a brief unit of instruction or learning segment (about a week of instruction.). These entries include:
- A description of their teaching context,
- An planning overview and rationale for a 3-5 lesson learning segment with a central focus,
- One or two videotapes of instructionfrom these lessons accompanied by commentary describing the instruction that took place each day and in these excerpts,
- An assessment plan and analysis of samples of student work from one learning segment,
- Written reflections on instruction and student learning.
In 2003-04, to complement the subject specific Teaching Events (TEs), member institutions began developing or adapting Embedded Signature Assessments (ESAs) that are currently in use within most PACT institutions. ESAs will be instrumental in providing formative feedback to the teaching candidate and faculty supervisors as well as to better inform the credentialing decision.
This paper explores the learning opportunities that standards-based portfolio assessment offers to promote teacher and faculty learning in teacher education. We will discuss the impact on learning from two perspectives: (1) from that of teaching candidates and faculty using a paper and pencil approach for portfolio development, mentoring and scoring and (2) from that of teaching candidates and faculty who used an electronic platform (UCOPIS) for portfolio development, mentoring and scoring. The first section reviews the empirical literature studying the impact of performance assessments on teacher learning and elucidates methodological difficulties in measuring teacher learning. The second section points to opportunities and challenges that electronic applications may provide for measuring changes in teacher knowledge and skills and for enhancing teacher learning.
Review of the Teacher Performance Assessment Literature Stanford Teacher Assessment Project (1987-1990)
In the late 1980s the Stanford Teacher Assessment Project piloted the performance-based assessments that served as the basis for NBPTS assessments. This work provided evidence that portfolio development, structured exercises that simulate teaching tasks, and structured interviews that probe teachers’ thinking, and the like, can measure differences between the pedagogical thinking, analysis, and skills of accomplished teachers and others (see Stanford TAP, 1987-1990; also Athanases, 1994; Collins, 1991 and 1993; Gellman, 1993; King, 1991a and 1991b; Stigler, Gallimore & Hiebert, 2000; Vavrus & Collins, 1991; Wilson & Wineburg, 1993). Significantly, a few of the studies went beyond measurement and examined the impact of the assessments on teachers’ professional learning.
In King’s (1991b) study of 15 teachers who had participated in BioTAP, teachers reported a year later that the assessments had influenced their practice of teaching for the better, that many of the assessment activities had challenged them, and that they had become more reflective during portfolio development and simulations. Athanases’ 1994 study examined self-report data from 18 teachers who had participated in a prototype portfolio assessment for literacy instruction and a case study of one third-grade teacher’s portfolio to analyze how the portfolio construction process influenced teaching and thinking. Of the 18 teacher participants, 16 identified some influence of the portfolio field test on their teaching; the most frequently mentioned change was the way teachers assessed student progress. Based on data from the case study teacher’s portfolio entries and interviews with her, Athanases found that the portfolio assessment stimulated changes in the teacher’s pedagogical practice, in the ways she assessed students’ writing, and in her initial and ongoing assessment of the whole class (even a year after the portfolio field test). The assessment also heightened the teacher’s awareness of her teaching practices and the reasons behind them.
Because the TAP instruments were not created to identify minimal teaching competency, because the participants had teaching experience, application of the Project’s findings for understanding the impact of performance assessments on novice teacher learning is somewhat limited. Also, the studies were exploratory in nature with a primary goal being to test the feasibility of performance assessments to differentiate teaching expertise rather than to determine long-term impact on teachers’ professional growth. Although the effects of these assessments on teacher learning and teaching practice in the long run were not studied in detail, the findings are important because they support the argument that teacher expertise can be differentiated while improving that expertise at least in the short term.
Impact of National Board for Professional Teaching Standards Assessments
Studies of teacher participants in the National Board certification process report that individual teachers can learn a great deal from developing a portfolio and completing the assessment center exercises (e.g., Haynes, 1995; Rotberg, Futrell, & Lieberman, 1998; Tracz, Sienty, & Mata, 1994; Tracz et al., 1995). Tracz and her colleagues (1995) detected changes in teacher’s perceptions of their teaching effectiveness following involvement in a field test of the National Board assessment. In this study 48 fully certified teachers rated their own teaching skills on a 37-item survey before beginning work on their portfolios and again after completion of the assessment.[1] Results of dependent t-tests of the teachers’ mean ratings before and after showed significant differences in teachers’ perceptions of their teaching skills. In all six categories of the survey (Organizing and managing the classroom, Planning and designing instruction, Delivering instruction to all students, Subject-matter knowledge, Diagnosing and evaluating student learning, Participating as members of a learning community), teachers’ mean ratings moved from 7 to 9 on a 10-point scale. Qualitative data from personal journals kept during the assessment process as well as data from individual interviews after portfolio completion corroborate and explicate the quantitative findings. Teachers overwhelmingly felt positively about the portfolio process because it stimulated them to examine their teaching critically and to rethink the decisions they made on a daily basis. In another study by Tracz, Sienty, & Mata (1994), teachers stated that videotaping their own teaching and analyzing student work made them more aware of how to organize teaching and learning tasks, how to assess student learning, and how to revise their practice when necessary.
In another study of National Board participants Rotberg, Futrell, and Lieberman (1998) conducted telephone interviews with 28 out of 38 teachers in 1997 to determine their views about the contribution of the assessment to their teaching skills and the consistency between National Board standards and current teaching practices. Most of the teachers interviewed reported that preparing for National Board certification was a powerful professional development experience, although a few of the teachers did not perceive changes in their teaching as a result of going through the process. Other findings suggest that support from colleagues and administrators were an important factor in successful completion of the assessment.
Sato (2000) interviewed 17 teachers who had recently participated in the National Board certification process and found that they reported learning within a variety of domains, including disciplinary content knowledge, teaching repertoire, students, learning about oneself, and thinking metacognitively. With specific reference to the portfolio construction process, teachers reported that they were prompted to try new teaching strategies such as the use of small group work, assessment data collection strategies, new curricular materials, and variation in writing assignments. The portfolio assessment also prompted teachers to pay greater attention to the learning needs of individuals in their classrooms, to think more deeply about their beliefs and values about teaching, to gain new metacognitive understandings of their practice, and to better articulate how and why they make instructional choices. Interestingly, though, no two teachers learned the same things from the process. Within each of the domains, teachers reported learning different things. In her interviews with teachers, Sato also investigated aspects of the assessment and environments that supported teachers’ opportunities to learn. She found that teachers used the National Board teaching standards as a guiding framework for evaluating their teaching, as benchmarks for professional growth, and as a tool for analyzing instructional choices. Because the assessment was grounded in teachers’ classroom experience, teachers were prompted to look closely at their practice through the examination of student work and classroom videos. And finally, teachers found the collegial interactions they had with other candidates at their schools or at monthly support sessions to be helpful by enabling them to have professional discussions and to get feedback on their work.
One common criticism of the NBPTS studies is that they report data on self-selected samples of teachers – all volunteered to participate in the National Board assessment and in the study. Although this does not invalidate the studies, positive feelings about the portfolio process may be partly explained by the characteristics of the sample teachers, who may be more predisposed to seek out opportunities for professional development and to improve their teaching practice. In addition, because most of the teachers participating in the study were experienced teachers (candidates for National Board certification must have at least three years of full-time teaching experience and most have more years of experience than that), questions about the generalizability of these studies’ findings to new teachers (and preservice teachers) who have not had as much classroom experience are raised.
Another methodological limitation, one that characterizes much of the literature on the impact of performance assessments on teacher learning, is that most of these studies rely on self-reports or interviews of participating teachers, the data often being the products of the performance assessments themselves (such as portfolio reflections). The reliability of artifacts from the portfolio assessment itself as evidence of teacher learning is questionable because teachers know they will be evaluated on the basis of these artifacts for their “reflectivity.” Other means of collecting data that allow us to track changes in teacher knowledge and skills would strengthen the evidence on teacher learning.
Impact of Performance Assessments on Preservice Teachers
A number of studies have examined the use of teacher performance assessments in teacher education programs (e.g., Anderson & DeMeulle, 1998; Darling-Hammond & Macdonald, 1999; Darling-Hammond & Snyder, 2000; Davis & Honan, 1998; Lyons, 1996, 1998a,b, 1999; Shulman, 1992; Stone, 1998; Whitford, Ruscoe, & Fickel, 2000). Because teacher education programs have been developing and experimenting with the use of portfolios and other forms of performance-based assessment (e.g., cases, exhibitions) since the 1980s, they have provided opportunities to study the effects of these innovations on the learning of preservice teachers.[2]
For example, Alverno College in Milwaukee has been using performance assessments in its teacher education program over the last twenty years. The performance assessments or “exhibitions” (which include essays, letters, positions papers, case study analyses, production of videos and curriculum materials, and simulated events such as parent-teacher conferences) are embedded in coursework. In a study of Alverno’s program and its graduates (Rickards & Diez, 1992, cited in Zeichner, 2000) survey data gathered during student teaching and in first-year follow-up studies showed that cooperating teachers, supervisors, and preservice teachers themselves rated preservice teachers “strong” or “satisfactory” on 96 percent of the performance criteria. In the follow-up study, principals rated Alverno graduates as “strong” across the five professional education abilities underlying Alverno’s program (conceptualization, diagnosis, coordination, communication, and integrative interaction). In Zeichner’s survey study (2000), which included 96 graduates of Alverno College over four years and 29 employers of the graduates, Milwaukee principals rated Alverno graduates highly for their reflective and learner-centered teaching.
These studies suggest alternative methodologies for measuring changes in preservice teachers’ learning and professional practice following participation in a performance assessment. Principal ratings of teacher performance seem to offer a more objective source of data that can be used to corroborate the self-reports of teachers. However, principal ratings have also been criticized as being unreliable and inaccurate (Peterson, 1995), lacking in variability with most principals rating all of their teachers as performing competently (Dwyer & Stufflebeam, 1996), and perfunctorily carried out with little or no effect on teaching practice (Kauchak, Peterson, & Driscoll, 1985). The ratings of teachers’ performance based on classroom observations by cooperating teachers and college supervisors may also be a valuable strategy to assess preservice teachers’ learning and professional growth, but cannot be used to judge teachers’ learning and performance beyond the one or two semesters of student teaching.
Studies on the use of portfolio assessment in teacher education programs in Maine (e.g., Lyons, 1999) and California (e.g., Snyder, Lippincott, & Bower, 1998) have used the reflections of teacher candidates and evidence from portfolios to make inferences about what they have learned through the process of developing their portfolios. An early study conducted by Richert (1987) with teacher candidates at the Stanford Teacher Education Program looked at the way two tools of reflection, portfolios and peer review, affected the nature of preservice teachers’ reflectivity. She found that portfolios tended to facilitate teachers’ reflections about the content of their lessons while peer review tended to facilitate teachers’ reflections about pedagogical strategies. Although the portfolio was not used as an assessment in this case, Richert’s research showed the potential use of portfolios as learning tools in preservice programs and the kind of reflectivity portfolios might foster.
The secondary teaching program at the University of Southern Maine (USM) has been using portfolio assessment since 1994, requiring candidates to select evidence to serve as the basis for a final judgment about certification made by a panel of school-based and university faculty following a portfolio interview in which the candidate presents and defends her work. Based on the reflections of teacher candidates and evidence collected in their portfolios, Lyons (1999) found that through the portfolio development process, intern teachers developed habits of mind that helped them to define good practice, to reflect on their own teaching and learning, and to support the reflection of their students. In a three-year longitudinal study of ten preservice teachers at USM’s Extended Teacher Education Program (ETEP), including undergraduates preparing to enter the program, preservice teachers in the post-baccalaureate program taking part in a year-long, intensive internship, and graduates of the program in their first or second years of teaching, Lyons (1998a) examined the meaning students gave to the portfolio assessment process through case studies involving analysis of their portfolios and open-ended interviews. Cross-sectional and longitudinal analyses of the data yielded several emergent themes: 1) initial efforts at reflection are simple and become elaborated over time; 2) critical conversations interrogating portfolio entries provide a scaffold that fosters teachers’ consciousness of their own knowledge of practice; 3) significant aspects of teaching practice, such as teaching philosophy, are identified and emerge; and 4) the sometimes painful process of public, collaborative inquiry results in teachers learning about themselves and the values they hold for teaching and learning. Although this study relies on teachers’ portfolio entries and interviews, its longitudinal design strengthens the value of teachers’ self-reports because it is possible to track changes in the ways teachers’ describe their views of teaching practice and teaching philosophy.