Saint Louis University-Madrid Campus

Arts & Sciences Programs

Learning Outcomes Assessment

May 2014

1

Contents

Overview

Communication

English

Overview

ENGL-190/2 Writing Core

Literature Courses

English as a Second Language

Overview

ENGL-111 M01 Advanced Integrated Skills

ESL 112 M01 Advanced Written Expression

Fine and Performing Arts

Humanities Division

Overview

History

Philosophy

Political Science: International Relations

Psychology, Sociology and Anthropology

Spanish

B.A. in Spanish

M.A. in Spanish

Overview

Anne McCabe, Ph.D., Associate Dean, Arts & Sciences Programs

The Arts & Sciences Division of the Madrid Campus was created as an academic unit in Fall, 2012. It consists of the following academic sub-units:

  • Communication
  • English
  • ESL & Modern Languages
  • Fine and Performing Arts
  • Humanities: History, Philosophy, Theology and Classical Languages
  • Political Science
  • Psychology, Sociology and Anthropology
  • Spanish

This assessment report includes recent assessment activity from these academic units. In November of 2013, the Madrid Campus Assessment Committee sent out the following as an assessment task for Spring 2014:

Phase III • Data Collection via Artifacts and Data analysis

Tasks
  1. Design a strategy to collect data via artifacts. Other strategies are possible!
  2. Sample strategy 1:
  3. Pre-test & Post-test strategy:
  4. Pre-test (i.e. Pre-beginning formal instruction):
  5. Use an artifact at the beginning of the semester to identify students prior knowledge and prior development of a potential outcome.
  6. E.g. Video tape a public speech during the first week.
  7. E.g. Have students do a Math problem on the first day of class.
  8. E.g. Have students analyze a fragment of a film on the first day of class.
  9. Post-test 1: (This post-test measurement is optiona)
  10. Use an artifact in the middle of the semester to measure development from the pre-test measurement.
  11. E.g. Embed an question/problem/exercise in the midterm exam.
  12. E.g. Video tape another public speech.
  13. E.g. Have students do a similar Math problem to the first one on the first day (but adapted to the level of the course at that point in the semester)
  14. Post-test 2:
  15. Use an artifact at the end of the semester to measure development and establish comparisons between Pre-test, Post-test 1 and Post-test 1.
  16. E.g. Embed an question/problem/exercise in the midterm exam.
  17. E.g. Video tape another public speech.
  18. E.g. Have students do a similar Math problem to the first one on the first day (but adapted to the level of the course at that point in the semester)
  19. Sample strategy 2:
  20. Senior Residency Exit Interview or Focus group.
  21. Data Analysis.
  22. Individual faculty collect data.
  23. Individual faculty do a preliminary analysis of the data collected.
  24. Faculty meet as a group and assess data to see the achievement of program-level outcomes.
  25. Data collected is measured against rubric with expectations about program-level outcomes achievement.

This report provides results, where available, by Arts & Sciences programs from the Madrid Campus from this task. Artifacts have been collected, and are available upon request.

Future actions include:

  • Providing more connections between individual course assessment and program assessment (as has been done in the Spanish Department).
  • Closing the feedback loop based on reflections/analyses of results.
  • Working on core assessment for requirements spanning more than one Department (Social Science core).

Communication

Department Information
Department / Communication
Academic Programs
Degrees / Bachelor of Arts
Contact Person
Name / Brian M. Goss, Ph.D.
Title / Program Director
e-mail address /
Phone / X236
Assessment of Student Learning Outcomes
Outcomes: / To preface this report, assessment is a new practice that was not in place during any of our undergraduate or graduate training, thus we are very much figuring it out as we enact it, with inevitable if jarring starts and stops.
The following department-level conceptualization of outcomes was developed by our delegate on the Assessment Committee, Dr. Daniel Chornet, in 2012. The document was, in turn, distributed in writing to the teaching staff of the department in February 2013. These department-level outcomes are very likely to be streamlined and further abstracted in the proximal future but stand as follows at the moment:
DEPARTMENT OF COMMUNICATION
Department-Level List of Student Learning Outcomes
The Communication Department’s SLOs have been organized according to Bloom’s Taxonomy of Learning Domains.
There are three general terms whose specific meanings are included below to avoid repetition in the articulation of SLOs:
•TEXTS refer to: Speeches, Advertisements, Films, Research Data, Academic Articles/Chapters, Interpersonal and Intercultural Situations,
•VOCABULARY refers to: specific Terms and Concepts of each area of Communication
•AREAS/CONTEXTS refer to: Interpersonal, Intercultural, Organizational, Mass Media, Work Situations, Journalism, Small Groups, Problematic Situations, Gender Constructions. (we can add more here...)
After the successful completion of the B.A. in Communication, the student should be able to:
COGNITIVE DOMAIN
How to assess cognitive skills:
  1. Remembering: Recall or recognize information.
  2. Recognize the different areas or contexts of the discipline.
  3. Recognize communication theories in different areas or contexts.
  1. Understanding: Understand, extrapolate or translate meaning.
  2. Understand the dimensions of language.
  3. Understand the role of communication in the constitution of social reality in different contexts/areas.
  1. Application: Use, apply, or put knowledge into practice in real-life situations.
  2. Apply communication concepts and theories to real-life situations.
  3. Use the specific vocabulary of the discipline.
  4. Apply communication theories to real-life situations.
  5. Use writing organization principles in different areas/contexts
  6. Apply intercultural communication concepts and theories to real-life situations.
  1. Analysis: Interpret structures, organizational principles, internal relationships.
  2. Analyze communication situations in different areas or contexts.
  3. Analyze texts according to specific criteria in their context.
  4. Analyze linguistically and discursively texts according to specific criteria in their context.
  1. Evaluation: Assess effectiveness of concepts in relation to values, outputs, efficacy, viability and other criteria.
  2. Assess communication situations in different areas or contexts according to specific criteria.
  3. Assess texts according to specific criteria.
  4. Assess communication theories in different areas or contexts according to specific criteria.
  1. Creation: Develop new unique structures, systems, models, approaches, ideas, and operations.
  2. Develop speeches and speaking materials according to specific criteria.
  3. Create theoretical knowledge claims using existing theories and research as a foundation.
  4. Write news reports according to specific criteria.
  5. Design advertisements according to specific criteria.
  6. Write research papers according to academic principles.
  7. Generate specific communication knowledge claims in different areas/contexts
AFFECTIVE DOMAIN
How to test affective skills:
  1. Receiving Phenomena: Awareness and willingness to hear; selected attention.
  2. Read, listen to, and attend to different perspectives from an objective point of view.
  1. Respond to Phenomena: Attends and reacts to particular phenomena according to specific criteria
  2. Respond to different perspectives according to specific criteria.
  1. Valuing:Internalization of a set of specified values and their application in real-life situations.
  2. Internalize journalism values of accuracy, fairness, objectivity, thoroughness and meeting deadlines.
  3. Internalize a set of ethical standards in communication research
  4. Internalize ethical and personal values associated with different areas/contexts.
  1. Organizing or Conceptualizing Values: Contrasting values and resolving their copeting features thus creating unique value systems.
  1. Internalizing or Characterizing Values: Having a value system that controls behaviors consistently and predictably.
PSYCHOMOTOR DOMAIN
How to test psychomotor skills:
•Speed needs to be attended to when assessing these skills. The use of equipment or instruments may be necessary for these skills
•To determine the level of the skill, they need to be put into practice and observed by the evaluator.
  1. Imitating: Repeating (trial and error until mastery) an act that has been demonstrated or explained.
  1. Manipulation: Continuous repetition of an act until a degree of automatization has been achieved and a degree of confidence has been gained. The outcome is relatively more complex.
  1. Precision: An act is performed quickly, smoothly, and accurately requiring a minimum of energy. The outcome is complex.
  1. Articulation: An act is performed in a proficient manner that even allows modification of the patterns to fit specific requirements.
  1. Naturalization: An act is performed automatically and the individual begins to experiment creating new acts or ways of modifying them based on understanding.
  2. Speak extemporaneously in public in a proficient manner according to specific criteria.
  3. Use a digital single lens reflex (dslr) camera according to specific criteria.
  4. Use digital imaging software according to specific criteria.
  5. Collect different types of data according to specific criteria.
  6. Communicate in different interpersonal situations/contexts according to specific criteria.
  7. Communicate in different intercultural situations according to specific criteria.

Assessment Method (s) / These have, in practice, varied by instructor. For example, Professor Dale Fuchs employs a method that pivots on examining successive drafts of articles for her journalism course with respect to desired outcomes. Dr. Daniel Chornet planned to do a specific assessment of intercultural sensitivity in CMM 330 Intercultural Communication, but because of the nature of the students (Political Science, other majors who were visiting, and Communication majors) the assessment failed because their levels of competence in the subject were very disparate. Dr. Chornet’s observations during the semester demonstrated how students struggled with some materials and some others were not as much [non majors vs. majors respectively]). These observations suggest that action must be taken on the course design in order to ensure student learning for the non-Communication majors.
Data Source / Examinations, speeches, creative work (e.g., films for course credit, photography) and research projects that implicate either scholarly research or original investigation (e.g., reporting, ethnography).
Assessment Results / Results that I have obtained from Professor Fuchs suggest that the genre demands and techniques implicated in professional training for news writing (e.g., sensory detail, concrete examples, anecdote lead, lead quote and “nut graff”) have been taken up by students when given the chance to revise their work with guidance; i.e., outcomes are assessed via “before” and “after” comparison. In turn, these results appear to affirm the department-level priorities (see above) in the domains of Cognition (2, 3, 4, 6) and Affect (1, 2, 3).
Actions / To be frank, assessment is in an embryonic state in the department at the moment which, at least, does signify its potential for its successful implementation when couched in these metaphorical terms.
In the coming year, the department-level outcomes will be fine-tuned and the roll-out of the process will be better coordinated and explicated.
One advantage that we have in this ongoing process is that we have an open and supportive relationship with out sister department in Missouri that includes our partially shared faculty—even as we are also socializing new faculty on our own campus in addition to our more established instructors with a range of 9 to 14 years of institutional immersion.
The objective that we will keep in our sights is to make the assessment process, for which we did not volunteer, work for us as pedagogues and for our students. Doing so will demand creativity and innovation so that what would otherwise be rote requirement may play out as an opportunity at the intersection of our academic training as in-class educators and investigators.

English

Department Information
Department / English
Academic Programs / Writing Core, English Literature
Degrees / Bachelor of Arts, Master of Arts
Contact Person
Name / Anne McCabe, Ph.D.
Title / Director
e-mail address /
Phone / 91 554 5858 ext. 237

Overview

Assessment information is included below for the Writing Core and the English Literature Core.

In Summer, 2014, an assessment plan for the B.A. and M.A. in English programs will be designed for the 2014-15 academic year.

ENGL-190/2 Writing Core

Note: mentions of artifacts are underlined. These artifacts are available upon request.

Assessment of Student Learning Outcomes
ENGL-190/2 Outcomes: / ENGL-190: Students will (in a 10-12 page research paper and in their final portfolios)
  1. analyze, summarize, synthesize and evaluate research materials in support of a position, while using the most appropriate reliable and reputable secondary sources for academic research
  2. articulate possible counter-arguments to position(s), acknowledge them fairly, and deal with them convincingly and in a balanced way
  3. adapt writing to achieve effect in academic situations and paraphrase and quote others’ ideas accurately and effectively for topic and purpose, using citation practices (MLA, APA, etc.) effectively, always adhering to academic honesty policies

Assessment Method (s) / We have created a rubric (see Appendix A) to assess the final research papers and portfolios of ENGL-190/2 for evidence of these outcomes. The three ENGL-190/2 writing instructors for Spring 2014 discussed the rubric and its application to writing samples, final papers and portfolios.
Data Source / Five student final papers and portfolios, chosen from three different sections of ENGL 190 and from students with differing abilities. When possible, these final papers were compared to an initial writing sample, designed to illicit students’ abilities to:
  1. summarize a text
  2. evaluate the author’s position

Assessment Results / Student results: In her writing sample, student 1 (‘B’ course grade) agrees wholeheartedly with the writer, without analysis of the writer’s argument. Her summaries are not written clearly with a reader in mind. She was deemed “developing” on the rubric for Outcome 2 (see rubric from September 2013 below). Student 1’s final research paper avoids dealing with controversy, thus receiving a “Satisfactory” for Outcome 2. She received “Above average” on Outcomes 1 and 3. In her final reflection, Question 1 shows that she is aware that she needs to continue to work on this aspect of writing.
Student 2’s (‘A’ course grade) initial writing sample shows complete agreement with author, without considering other viewpoints. Her final research paper shows “excellent” ability to juxtapose different viewpoints and reconcile the differences. Her final reflection (p. 3) shows her awareness of her strong abilities in this area. She also received “Excellent” on the other two dimensions of the rubric.
Rubric results: The rubric was shown to have some overlap between outcomes 1 and 2. Also, in reviewing the final research papers and portfolios, we discovered that “above average” papers may articulate differing points of view, but may have problems effectively conceding or refuting to support their arguments, which excellent papers can do. This difference with excellent papers was not clear with the rubric.
Student 3’s (“B+” course grade) portfolio traces her development throughout the course; English is her second language. Her final research paper (graded April 10) shows above average ability across the three rubrics, showing an ability to synthesize but not to highlight counter-arguments. Her final reflection shows that she knows that a counter-argument should be included, yet her final paper shows that she still needs to learn how to make explicit that she is doing so.
Student 4’s (C- course grade) final research paper received a grade of D, as it showed little synthesis of sources (Outcome 1), a lack of rhetorical awareness of audience and argument (Outcome 2) and a lack of organization and understanding of paragraph structure (Outcome 3).
Student 5 was enrolled in ENGL 192. At SLU, only Engineering students take this course; however, this student was visiting, and took it as a Business major. His final research paper received a grade of B+, which shows evidence of excellence in achievement of outcomes 2 and 3 (strong research skills, good organization) but above average on Outcome 1, as he puts forth different arguments to solve the “problem” of cost, but does not really evaluate them.
Results of the discussion about the papers and portfolios: We discussed the difficulty of passing students who are still in the “developing” category, which corresponds to a grade of “D”. However, we concluded that it is inevitable that students who pass the course with a D will go on to have problems with their academic writing. On this note, we found the descriptor “developing” to be misleading, as in essence everyone is “developing” their abilities constantly through lifelong learning. Thus we changed the descriptor words for the rubric categories. We also clarified that a grade of “F” goes to students who make no effort and/or who plagiarize.
Actions /
  • The rubric was revised to remove the overlap and to account for the differing abilities in dealing with different points of view. The descriptors were also changed. (Appendix B, p. )
  • More useful data would be provided by comparing more assessment points throughout the semester, in order to assess more of the program outcomes, which are often interrelated.
  • Next year student work ENGL-192 needs to be assessed.

Appendix A

Rubric: 190/192: September 2013

Outcome 1: Rubric:
35% of grade / EXCELLENT / ABOVE AVERAGE / SATISFACTORY / DEVELOPING
In their final research paper, student / Cites a sufficient number of highly reputable and appropriate sources in supporting the main points made in the paper, synthesizing information from the various sources, even juxtaposing differing (perhaps conflicting) points of view and resolving differences in a way which enhances writer’s purpose / Cites a variety of reputable and appropriate sources in supporting the main points made in the paper, synthesizing information from the various sources / Cites a variety of reputable sources in supporting the main points made in the paper / Cites sources which may not be reputable, varied or appropriate
Outcome 2: Rubric:
35% of grade / EXCELLENT / ABOVE AVERAGE / SATISFACTORY / DEVELOPING
In their final research paper, student / Uses a neutral tone in reporting on different position(s) about the topic; includes a fair critique which demonstrates deep and considered understanding of viewpoint / Uses a neutral tone in reporting on different position(s) about the topic; includes a fair critique which demonstrates some understanding of viewpoint / Uses a neutral tone in reporting on different position(s) about topic, yet which may at times show a biased understanding of viewpoint / Uses evaluative language which demonstrates bias; focuses more on own opinion rather than considering other viewpoints
Outcome 3: Rubric: 30% of grade / EXCELLENT / ABOVE AVERAGE / SATISFACTORY / DEVELOPING
In their final research paper, student / Includes:
  • a thesis statement with a clear purpose and which provides for full scope of paper
  • coherent and unified paragraphs that include topic and transitional sentences
  • complete, effective and meaningful sentences
  • reader-based organization
  • proper layout and format, including citation style
  • relevant title
/ Includes:
  • a thesis statement with a purpose and which provides for scope of paper
  • coherent and unified paragraphs that include topic and transitional sentences complete and meaningful sentences
  • effective organization
  • proper layout and format, including citation style
  • relevant title
/ Includes:
  • a thesis statement with purpose and some indication of scope of paper
  • complete sentences with some errors/lacks which do not impede meaning
  • adequate organization
  • layout and format which may show some inconsistency
  • title
/ Is lacking in several of the areas mentioned

Appendix B: