TITLE: A Case Study to Explore Learning of Assessment during a “Process Based” Faculty Development Workshop

AUTHOR LIST: Robert M. Carson, Scott K. Metlen, Tris ? Utschig, Daniel A. Cordon, Marie Racine, Steven W. Beyerlein, Donald F. Elger

ABSTRACT:

I.Learning Environment

A learning environment is ……

An enriched learning env. Is

One fo the attributes is assessment.

II.Assessment

Assessment is considered a best practice in education, and the practice of assessment has expanded to many contexts beyond the classroom. Some of these contexts include organizations, communities of practice, classroom assessment, course assessment, program assessment, and institutional assessment. However, assessment in the context of historical development of universities, assessment is a relatively new concept, arising in the early 1970s (Heywood, 2000). Consequently, there is confusion in assessment theory and practice due to inconsistency in what people think assessment means.

During the last 15 years, a small group of professors who have worked with the company Pacific Crest, have evolved a theory of assessment—see appendix A. (add guidebook). This theory can be summarized as:

Assessment. Assessment is the process providing guidance to self or others for the purpose of improving future performance. Assessment integrates observations, analysis, measurement, communication, and ing, analyzing, measuring, and providing feedback. This feedback can focus on many things including an assessee’s performance, an asseessee’s work product, and an assessee’s learning. The primary purpose of assessment is to help the assessee improve their performance, their learning, and their work product. Assessment can be performed by a coach (expert assessment), a peer (peer-assessment), or the assessee themselves (self-assessment).

In contrast, the evaluation is a very different process.

Evaluation. The process of judging or comparing against established standards and reporting the results of this judgment. The primary purpose of evaluation is to inform relevant stakeholders, on the degree to which the standards have been met. The evaluator may be an expert, a person in a power position (e.g. manager) or a peer.

While most people confound assessement and evaluation. To confound means to mix up and confuse two distict concepts. For example, students often confound weight and mass and they often confound energy and power. Avoid confounding of assessment & evaluation is fundamental to facilitating learning. The reason are:

  • Assessment is used to help people grow. Evaluation is used for judging to see if standards are met. When a person is evaluated when they need and want assessment, this stifles their growth and damages the trust relationship needed for assessment.
  • Evaluation is built into academic culture, whereas assessment is not. However, evaluation does not produce growth in performance. Thus professors who evaluate most of the time instead of assess miss multiple opportunies to facilitate growth in themselves, their peers and their students.
  • Evaluation impacts the self-worth of the evaluatee, often with devastating effects. For example, when a professor is denied tenure, this often scars them for the rest of their life. If professors received quality assessment prior to tenure evaluation, this problem could be significantly reduced. This same problem occurs to many engineering students.

While most people do not want to be evaluated, evaluation is important and the processes of assessment and evaluation are complementary processes that can work together synergistically so that when people are evaluated, their performance is high and they have clear understanding of what is needed to meeting the evaluation standards. Details of creating aligned assessment and evaluation are presented by (cite guidebook).

III.Professors as Learners

The author team collectively has extensive experience with faculty development. This experience indicates that professors experience many of the same problems and they behave in ways that are very similar to their students. Thus,profs need a enriched learning environment.

To understand learning by professors, researchers have created various theoretical frameworks. Here, we review three frameworks: learning as a process of social participation in a community of practice, learning by an organization, and adult learning.

A.Adult Learning

Cross (1981) presents the Characteristics of Adults as Learners (CAL) model in the context of her analysis of lifelong learning programs. The model attempts to integrate other theoretical frameworks for adult learning such as andragogy ( Knowles ), experiential learning ( Rogers ), and lifespan psychology.

The CAL model consists of two classes of variables: personal characteristics and situational characteristics. Personal characteristics include: aging, life phases, and developmental stages. These three dimensions have different characteristics as far as lifelong learning is concerned. Aging results in the deterioration of certain sensory-motor abilities (e.g., eyesight, hearing, reaction time) while intelligence abilities (e.g., decision-making skills, reasoning, vocabulary) tend to improve. Life phases and developmental stages (e.g., marriage, job changes, retirement) involve a series of plateaus and transitions which may or may not be directly related to age.

Situational characteristics consist of part-time versus full-time learning, and voluntary versus compulsory learning. The administration of learning (i.e., schedules, locations, procedures) is strongly affected by the first variable; the second pertains to the self-directed, problem-centered nature of most adult learning.

Knowles' theory of andragogy is an attempt to develop a theory specifically for adult learning. Knowles emphasizes that adults are self-directed and expect to take responsibility for decisions. Adult learning programs must accommodate this fundamental aspect.

Andragogy makes the following assumptions about the design of learning: (1) Adults need to know why they need to learn something (2) Adults need to learn experientially, (3) Adults approach learning as problem-solving, and (4) Adults learn best when the topic is of immediate value.

In practical terms, andragogy means that instruction for adults needs to focus more on the process and less on the content being taught. Strategies such as case studies, role playing, simulations, and self-evaluation are most useful. Instructors adopt a role of facilitator or resource rather than lecturer or grader.

Rogers distinguished two types of learning: cognitive (meaningless) and experiential (significant). The former corresponds to academic knowledge such as learning vocabulary or multiplication tables and the latter refers to applied knowledge such as learning about engines in order to repair a car. The key to the distinction is that experiential learning addresses the needs and wants of the learner. Rogers lists these qualities of experiential learning: personal involvement, self-initiated, evaluated by learner, and pervasive effects on learner.

To Rogers, experiential learning is equivalent to personal change and growth. Rogers feels that all human beings have a natural propensity to learn; the role of the teacher is to facilitate such learning. This includes: (1) setting a positive climate for learning, (2) clarifying the purposes of the learner(s), (3) organizing and making available learning resources, (4) balancing intellectual and emotional components of learning, and (5) sharing feelings and thoughts with learners but not dominating.

According to Rogers, learning is facilitated when: (1) the student participates completely in the learning process and has control over its nature and direction, (2) it is primarily based upon direct confrontation with practical, social, personal or research problems, and (3) self-evaluation is the principal method of assessing progress or success. Rogers also emphasizes the importance of learning to learn and an openness to change.

B.Learning by Organizations

Handy (1995) proposes that the only way to survive in dynamic internal and external environments is to become a “Learning Organization” (LO). The only set theory of organizations is that continual adaptation is necessary for survival. One way the internal environment of an organization changes is through employee attrition and the employment of new employees. Competitors’ and suppliers’ changing capabilities, shifting customer perception of needs, and advances in process and product technology make the probable state of the external environment of an organization a stochastic variable.

Garvin (2000) states that a learning organization “is an organization skilled at creating, acquiring, interpreting, transferring, and retaining knowledge, and at purposefully modifying its behavior to reflect new knowledge and insights.” pp. 11. Even though survival is dependent upon being a learning organization, few organizations are truly good at learning (Garvin, 2000). A characteristics of a learning organization is where dialogue enables the transferring of individual knowledge into expanded group knowledge that can be recorded as explicit knowledge reflected in policies and procedures (Anderson, 1995). Individual knowledge comes from two sources, training and by doing. Firms who do not encourage employees to venture outside of stated polices and procedures and provide resources such as training to empower employees with the necessary skills and confidence to venture will not reap the benefit of learning by doing (Handy, 1995). Thus, leaders of LO have structure their organization in such a way that learning can happen (Rolls, 1995).

Rolls (1995) states that this structure should be one where employees’ self-esteem should flourish due to relationships with peers and managers that promote dignity and security because everyone in the organization is willing to listen to and recognize each others values and listen to their ideas; all care about each other as people). Through such a structure, the culture becomes safe and a person can respectfully say what they think about products and processes because of their system understanding, and cultured humility and compassion (Kofman & Senge, 1995). In such a culture, change will be driven from the bottom and be seen as an opportunity instead of a top down driven challenge. Because employees see change as an opportunity, it will happen (Kanter, Rosabeth, & Moss, 1995).

To establish such a culture where learning can happen and subsequently implemented, leaders have to make a commitment to make learning a sustainable competitive advantage. The learning has to be supported with continually allocated resources to continually nurture learning. Resource guardians typically allocate according to a firm’s accepted vision. Thus, the vision has to clearly state that the firm will continually be a LO and all within the firm have to embrace that vision. To achieve acceptance of the vision, leaders have to model learning, ‘do as I say, not as I do’ does not work any better in organizations than it does in families. To learn if learning is taking place and how to increase learning in the future, assessment of individual learning and the new knowledge that is being implemented into policy and procedures through group dialogue needs to be performed on a routine bases. If some individual or group of individuals refuse to learn, the learning environment becomes endangered and those individuals must be dealt with, either convinced to learn or encouraged to work for the competition (Thompson, 1995). The positive learning environment described has to be maintained to promote dialogue and thus, learning.

C.Communities of Practice

Learning can be described as a process of social participation in a Community of Practice (add citations). Key assumptions of the Community of Practice perspective as they apply to this research project:

  • Professionals organize their learning through the professional communities to which they belong.
  • Knowledge is integrated in the life of professional communities that share values, beliefs, languages, and ways of doing things.
  • The processes of learning and membership in a community of practice are inseparable.
  • Knowledge is inseparable from practice. It is not possible to know without doing. By doing, we learn.
  • To empower people, provide opportunities for them to contribute to the community and provide opportunities to engage in real action that impacts individuals and the communities. This empowerment creates the most powerful learning environments.

D.Summary Comments on the Literature Review

The literature provides rich and deep insights into the nature of learning as it applies to professors. While there are various theoretical perspectives, there are a number of common themes. These themes and how they connect with this study are.

  • Learning by experience is effective
  • Adult learners chose what they learn based on
  • Learning is social
  • Learning is organizational—

IV.RESEARCH QUESTION

. For adult learners who are improving their knowledge of assessment and their ability to effectively assess others (i.e. their performance), what factors or combinations of factors mediate in learning, attitudinal changes, and performance improvements?

SUBQUESTIONS

  • What challenges or barriers do learners face?
  • What are the present cultural norms for assessment practice?
  • What would ideal cultural norms for assessment practice be?
  • What are challenges to transfer beyond the workshop boundaries?
  • What motivates a faculty member to come to a workshop on assessment?
  • How does motivation change as a faculty member acquires knowledge of assessment?
  • How can extrinsic motivation change to intrinsic motivation?
  • What are the key intrinsic long term behaviors?
  • What factors of the environment are important?
  • How important is practice w/ feedback?
  • What is their conceptualization of assessment when thy enter the workshop
  1. Context
  2. Research question answered by gathering data at a faculty development workshop, UI, 35?, facilitated by Pacific Crest.The present study uses case study (add), one of the five traditions of qualitative social research (Creswell, 1998; Creswell, 2003.). The study was conducted at the University of
  3. Research Team participated in the workshop while conducting the research study--
  4. Research Team comprised on a engr. manufacturing manager , an operations management specialist with significant experience, and a learning specialists. POVs that are brought by the team
  5. Performance is what motivates people. True of students and true of faculty.
  6. People learn performances best by doing and then receiving feedback
  7. Focus on people and their learning is key. This leads to results, whereas a direct focus on results usually fails because the people expected to produce the results lack the performances.

V.DESCRIPTION OF WORKSHOP

Part of ELE grant

Solicitation @ UI, WSU, LCSC. Leads. Marketing

Description of facilitator.

Workshop Objectives

  • Learners will recognize the value of daily reflective practice. The way this will be manifest after the workshop is over is that the learners will be more inclined to, on a regular basis (several times per week), write down strengths of their present performance, ideas for improving their present performance, and insights. These writings will demonstrate processing of “what is important” and “why is this important.”
  • Learners will develop a desire for an academic culture in which there is effective assessment (see intro section for defn of effective assessment). The way this will be manifest after the workshop is over is that the learners
  • During the workshop, learners will improve their abilities to analyze their observations and produce meaning and learning from these observations. That is, learners will become more effective at experiential learning. The way that this will be manifest after the workshop is over is ……

SII—give handout in appendix A

Workshop Methods—use photos

VI.METHODS

We chose case study from the five traditions described by Creswell (1998) because ….. While the case student is largely qualitative, we selected mixed methods ….. We used six instruments to gather data

A.Pre and post survey Instrument

Appendix B contains the pre & post survey. Mixed. Rationale. Purpose of intrument

  • What did participants know about assessement when they came in
  • What were the changes that
  • Did the workshop have any impacts?

B.Workshop Interview Instrument

An interview protocol form was produced for the purpose of conducting individual, face-to-face, semi-formal interviews using a simple set of 5 open-ended questions (see Appendix C for the interview protocol instrument). This instrument was used in order allow the interviewee to bring out the richness of context surrounding particular factors involved in their learning. Six interviews were conducted during the second and third day of the event before it began, during breaks, and at the end of the day so that participation in the event was not impacted for the interviewee. The interviews lasted between ten and fifteen minutes.

The interview protocol instrument was designed with a focus on assuring validity and trustworthiness in the results obtained [1-3]. A substantial effort was made in terms of achieving trustworthiness in the data, or the removal of interviewer bias during the process of obtaining data. Trustworthiness was elevated through careful design of questions according to guidelines for social science research available in the literature, peer review of the questions used (and associated prompts), and the use of a practice interview with subsequent discussion of the line of questioning and techniques used [1-4]. Furthermore, the validity of the data, or the degree to which results measured what was intended to be measured, was increased through the use of two interviewers and transcription of the interview results from audiotape by a third party [1, 2]. During the interview one interviewer recorded responses in the case of equipment failure while the other asked questions and prompts to guide the conversation.

The interview protocol instrument (see Appendix C) was used for all interviews in an identical fashion. The research question was first introduced to the interviewee and permission was obtained to use and record their responses. This was followed by a short introductory story to orient the interviewee before the first question. During question phase of the interview, questions were asked with every effort put on minimizing intrusion by the interviewer and maximizing listening according to the guidelines for interview techniques in social science available in the literature [3, 4].

References for this section

[1-4]

1.Creswell, J.W., Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 2nd ed. 2002, Thousand Oaks, CA: Sage Publications.

2.Creswell, J.W., Qualitative Inquiry and Research Design: Choosing Among Five Traditions. 1998: Sage Publications.

3.Doyle, J.K., Handbook for IQP Advisors and Students: Chapter 11: Introduction to Interviewing Techniques. 2004, Worcester Polytechnic Institute.

4.Berry, R.S.Y. Collecting Data by In-depth Interviewing. in British Educational Research Association Annual Conference. 1999. University of Sussex at Brighton.