Electronic Writing Portfolio Readings Report

AY 2015

Number of Readers: 17

Number of Portfolios: 208

Number of Papers: 616

Methodology

In Fall 2005, volunteers were solicited from the faculty and staff. Readers were required to be full-time at Eastern, to have completed a master’s degree or higher, and to teach at least one undergraduate course each year. Twenty-three readers were trained at a 1 ½-day workshop by Dr. Robert O’Brien-Hokanson, Associate Professor of English and Coordinator of the Communication Ability Department at Alverno College. Readers came from all four colleges, the School of Continuing Education, the Center for Academic Support and Achievement, and Booth Library. A second training was held in October 2008, a third in October 2010, and a fourth in January 2014; Dr. Karla Sanders delivered these trainings based on the work completed by Dr. O’Brien-Hokanson.

The following 2015 readers were trained at one of these sessions; they represent all four colleges:

Leann Akins, English, CAH, 2005

Jay Bickford, Early Childhood, Elementary, and Middle Level Education, CEPS, 2010

Diane Burns, Geology/Geography, COS, 2014

Jonelle DePetro, Philosophy, CAH, 2005

Crystal Duncan-Lane, Family & Consumer Sciences, LCBAS, 2014

Bill Feltt, English, CAH, 2008

Kathryn Fenton, Music, CAH, 2014

Rose Gong, Secondary Education & Foundations, CEPS, 2005

Angie Jacobs, Communication Studies, CAH, 2008

Nicole Mulvey, Communication Disorders & Sciences, COS, 2014

Mary Caroline Simpson, Art, CAH, 2014

Jeff Snell, Management, LCBAS, 2005

Tim Taylor, English, CAH, 2014

Gordon Tucker, Biological Sciences, COS, 2005

Traci Worby, Kinesiology & Sports Studies, CEPS, 2010

Marjorie Worthington, English, CAH, 2014

Diana Wyatt, School of Technology, LCBAS, 2008

Readers were urged to evaluate writing patterns across the portfolios rather than focusing on each individual document submitted to the portfolios. The reading guide asks readers to provide an assessment of writing ability for completed portfolios across seven aspects of writing: focus/purpose, organization, development, audience, style, mechanics, and use of sources. Readers were also asked to assess each portfolio overall. They complete a reading guide for each portfolio.

A sample of 10% of the completed portfolios are read each year; readers were given a month to read their portfolios, were given access to a secure web site containing the documents and the students’ account of the writing assignment, and were assigned to read 12 or 13 portfolios that were chosen at random from the completed portfolios. After reading their set of portfolios, readers were asked to complete a reader’s observation sheet and to attend one of three focus groups held at the beginning of March to discuss student writing as displayed in the portfolios.

The qualitative data that follow represent the discussions at the focus groups as well as information taken from the readers’ guides and observation sheets; all information has been collated and summarized by the Executive Director of the Center for Academic Support and Assessment. A draft of this report was disseminated via email for readers’ comments. Changes to the report were made based on faculty feedback. The percentages given for each assessment of the various areas of writing are taken from a compilation of scores given by the readers for each portfolio, not each document.

The following data are divided into the categories assessed by the readers. Each section gives the readers’ impressions of the portfolios as a whole and the final section offers potential uses for the data in terms of improving the curriculum/pedagogy. Where appropriate, readers’ written and verbal comments have been quoted to support the general conclusions that have been drawn. Percentages refer to the percentage of portfolios that were rated in the categories described.

Portfolios Overall

Strong Portfolios: 26%

Adequate Portfolios: 56%

Weak Portfolios: 17%

The increase in weak portfolios seen in the reading evaluations in AY14 was not the beginning of a new trend; this year’s portfolios had only 17% of portfolios rated as weak overall. Strong portfolios were about a quarter of those read while over half were deemed “adequate” by the faculty readers. Readers indicated that students’ writing communicated information but did not do so in a pleasing or elegant manner. Little evidence of revision and polishing was displayed in the samples read. “Most students are not putting the effort in to make the writing more interesting or engaging. It is just an assignment and that’s it.”

The weakest areas remain those aspects of writing related to critical thinking and developing an argument. Appropriate citation of sources remains an issue as well as asking students to think critically and analytically. One reader summed up these issues, which were described by many readers:

I found a couple of portfolios quite troubling because there were papers that showed research but did not properly paraphrase and/or cite sources—unintentional plagiarism. One trend that I noticed is that students appear to be assigned informative research papers, which make them work with sources. Often though, the papers were mostly data dumping (providing quotations and paraphrases) and very little of students’ writing. I think learning new information is important (part of what college is about); however, many of the papers I read didn’t seem to ask students to move to the next levels in terms of critical thinking—detailed analysis, sound critique, evaluation, and argument.

Organization and focus were primarily adequate with some portfolios displaying strong skills. While papers are only required to be 750 words in order to be submitted to the EWP, some readers found that shorter papers did not allow for development. “I had only 2 papers that were 22 and 25 pages long, and they were the best papers. They developed everything, had headings, and they stayed on focus. Then, my 3-page papers weren’t that good.” Some readers speculated that longer papers might have encouraged students to spend more time on revision, so they would naturally be stronger papers because of the time and care that students would need to put into them. Some readers that no papers in their set of portfolios were longer than 5-6 pages.

The sections that follow discuss the strengths and weaknesses readers found in portfolios by writing trait followed by a section on ways to improve writing through pedagogy or curriculum changes.

Focus/Purpose

Strong (Consistently strong sense of focus/purpose throughout): 30%

Adequate (Clear focus/purpose in most or all submissions): 46%

Weak (Some evidence of ability to focus on a purpose): 21%

Poor (Very little or no evidence of focus): 2%

The percentage of portfolios rated as “strong” in focus grew by 6% while the percentage deemed “adequate” grew by 1% and those rated “weak” dropped by 7%. The number of portfolios in the “poor” category is the most consistent. Readers indicated that staying on task (having a focus) was one of the stronger areas whereas creating a sense of purpose in one’s writing and working toward that purpose was more difficult and less apparent than simply staying with their topic.

Readers indicated that many portfolios were missing writing conventions that help guide the reader and maintain focus on a topic or argument or create a sense of purpose. These missing conventions include thesis statements, topics sentences, and transitions combined with weak aspects such as introductions and conclusions. “They had a general focus. But, if you wanted something specific like a thesis statement that they were trying to develop and really come to a point, most of mine weren’t getting to that level of focus.”

Several readers commented that students seemed to show the process of finding their purpose in the introduction rather than starting strong with a clear reason for writing to a particular audience. Assignments with specific genres with particular conventions were strongest in this category. “I had a number of proposals and business analysis and anthropology papers that were really good. They tended to have a really strong focus and purpose because of the genre they had to work within. The weaker ones tended to be, you get what you ask for, and a number of papers were just data dumping.” Many readers indicated that focus was fine (identification of the topic), but there were more significant problems with purpose (why I am writing this paper).

Organization

Strong (Consistent use of structure in ways that enhance presentation of ideas/information): 22%

Adequate (Logical organization and/or clearly identifiable structure): 51%

Weak (Inconsistent sense of structure and/or lapses in organization): 26%

Poor (Very little or no sense of structure or organization): 1%

Little change was seen this year in the number of portfolios rated in each category related to organization. The percentage deemed “strong” rose by 3% while those evaluated at “adequate” increased by 2% and the “weak” portfolios dropped by 4%. As with the other categories, the number of portfolios rated “poor” remains very consistent.

As indicated by the percentages given in the chart above, readers felt that organization was a skill students had mastered at the “adequate” mark. Papers could be followed; students rarely went off on tangents, but very few provided well-developed, thoughtful introductions and conclusions.

In addition, transitions that could have guided the reader in terms of purpose and enabled a cogent organization were largely absent. “There were clear beginnings, middles, and ends, and I think they were able to somehow introduce their idea even though it wasn’t extraordinarily strong, and they were able to conclude that idea. Transitions were a little weaker sometimes; they were repetitive and sometimes they didn’t exist at all. It was just, ‘oh, here is another idea, and here is my second idea, and here’s my third idea.’ So, organization was very basic at best.” Several readers remarked on the lack of transitions and other markers to move pieces along or make comparisons and contrasts apparent. Most samples had simple organizational patterns, and many adopted an organization from an article or book being reported on. “The ones that did well [with organization] used subheadings to help with organization. Others did well in the beginning and end and sort of got lost in the middle. I had meandering.” Readers indicated that paragraphs did not build on each other. “So many of them you could cut the paragraphs out and re-arrange it and it wouldn’t have made a difference in their essay.” Students also included repetition of ideas and did not seem to revise for clarity and ease of following for the reader.

Development

Strong (Ideas consistently developed in depth and supported with rich and relevant details): 17%

Adequate (Ideas developed in depth with appropriate supporting evidence/details): 42%

Weak (Some development of ideas and use of supporting evidence/details): 31%

Poor (Very little or no development of ideas or use of supporting evidence): 5%

Development remains one of the weakest traits in student writing as displayed by the portfolios although the percentage rated as strong did increase by 6% this year. Portfolios rated “adequate” in development dropped 7% from AY14 and 12% from AY13. Portfolios deemed weak rose to 38% showing a 7% increase from the previous year, while those rated “poor” portfolios remained the same.

With 42% of portfolios given poor or weak ratings, this writing skill is the most problematic as displayed in the papers submitted to the EWP. “They wrote with clarity but did not write with complexity. I think it had to do with tasks. Most of them were summarizing a book or writing about their feelings. There really wasn’t much depth,” one reader noted. Readers indicated that students either did this very well or had issues, and more seemed to struggle with development than not. Students did not provide integrated evidence, the readers found. Some of the strongest examples were those in which student followed the scientific method, which forces analysis.

Since many of the assignments asked students for their own opinions or reflections, few asked for an in-depth analysis or development of a side of an argument. One reader explained, most papers “were personal opinions. ‘I feel this way, I don’t feel this way, but I like this.’ Or, ‘Let me tell you my life story,’ and they weren’t really developing an idea. Many portfolios didn’t even use sources and the ones that did use sources, it was abstract. ‘This source said this, and this source said that.’ But, there was never a bringing of these things together. They weren’t developing ideas beyond just stating facts.”

Readers indicated that students tended to throw in sources and not explain who the quote was from or the sources relevance for the topic at hand. They did not guide the reader nor try to use evidence to prove a point and use of details was scarce. “There were a lot of quotes and paraphrasing but not a lot of their writing. With the weaker papers there was a lot of data and just informational research papers and not much of the writer there at all. Like a patchwork quilt of other peoples’ stuff.” Several readers concurred with this characterization. Another called it, “the inability to analyze the data and information.” Many concurred.

Some readers found students using data that did not support their points or ran contrary to their argument. Readers felt that few papers ran the gamut of Bloom’s taxonomy, and it was rare to encounter students attempting evaluation and not many had attempted analysis.

Audience

Strong (Sophisticated sense of audience—e.g., distinctive voice and/or appropriate tone): 23%

Adequate (Some awareness of and/or attempt to communicate with audience): 58%

Weak (Little or no awareness of audience): 16%

Poor (No sense of writing for an audience): 3%

With each reading, faculty have indicated that it is often difficult to discern any reader other than the instructor for the course unless the genre of the paper demanded that an audience be identified, such as in a letter or report for a company. “If students are going to have to write in their professions and in their civic lives, course could have assignments that mimic rhetorical situations in people’s possible professions and other situations,” suggests one reader.

Some writers, however, showed a good sense of audience by defining terminology while other writers seemed to be writing only for themselves. Readers speculated that an increase in reflective writing could have lead some writers to be more focused on themselves than on an external audience.