IST@IUB
R561: Evaluation in the Instructional Development Process (Section 7463)
Spring 2011
“We improve human learning and performance in diverse contexts”
Class Time: Thursdays at 4:00 – 6:45 pm (1/13 – 4/28) (No class – 2/24 and 3/17)
Location: Education 2275
Instructor: Dr. Yonjoo Cho (Education 2232)
Office Hours: Wednesday at 1:30 pm – 3:00 pm (appointment requested for other times)
Communications: choyonj @indiana.edu; 812-856--8144
Teaching Assistant: Ingu Kang () and Minkyoung Kim ()
Course Access: http://oncourse.iu.edu; https://www.indiana.edu/~istr561/cho11spring/
Course Description
R561: Evaluation centers on evaluation as an integral element of the instructional technology (IT), human performance technology (HPT), and human resource development (HRD) processes. Training, performance improvement, and HR professionals need information about the impact and effectiveness of programs in terms of: (1) the degree to which program results achieve intended objectives, (2) whether results are desirable, and (3) evidence that results are achieved in a cost-effective manner. Principles and methods for evaluating instructional and performance improvement programs during the stages of analysis, design, development, implementation, and utilization are covered. Frameworks and models for planning and conducting evaluations are also discussed and applied.
Course Objectives
At the completion of the course, students will be able to:
1. Understand basic concepts and terminology associated with instructional, performance improvement, and HRD evaluation.
2. Explain the purposes and uses of evaluation within different instructional, performance improvement, and HRD environments.
3. Use qualitative and quantitative data collection techniques in evaluation activities.
4. Analyze and interpret evaluation data and information.
5. Report the results of evaluation activities.
Course Outline
The course is divided into six units to reflect the importance of major evaluation perspectives.
Unit 1: Basics of Evaluation
Develop common understandings of basic concepts and definitions, underlying principles and theories, and perspectives of the field of evaluation.
Units 2 to 5: Four Levels of Evaluation
Address four well known perspectives for evaluating instruction and performance improvement:
· Gain knowledge of purposes, concepts, theories, cases, and major issues in using different levels of evaluation in varied types of organizations.
· Gain skills of developing instruments required for data collection in each level of evaluation, including interview questions, questionnaires, test items, and observation checklists.
· Gain skills of analyzing collected data using both qualitative and quantitative methods.
· Gain field work experience by applying knowledge and skills to various organizations including business, education, military, non-profit organizations, and simulated situations.
Unit 6: Evaluation Synthesis
Discuss evaluation frameworks and ethics that are applicable to a variety of organizational settings. This is developed through the student group’s evaluation project and a review of recent innovative approaches to evaluation.
Course Assignments
Weekly Participation
Students are expected to: (1) post discussion questions on required readings (i.e. one on each reading) by Thursdays at 9:00 am and the same number of comments on other students’ questions by the end of Sunday each week in the Oncourse Forum; (2) actively participate in class activities such as evaluating a movie (12 Angry Men); and (3) honestly respond to my check-in asking “where are you at right now?” either at the beginning of each class or near the end of class to see if students are doing okay in class. The instructor and TAs will assign grades by reviewing the quality AND the quantity of the participation.
Unit Exercises
Students will complete three unit exercises including levels one to three evaluation: survey, test development, and transfer scenario. Each exercise will be worth 10 points, assessing students’ abilities to analyze and synthesize what they have learned in each level of evaluation. Peer reviews are part of unit exercises evaluating other groups’ surveys and test development.
Case Study Presentation
Groups of students will select a case study related to their contexts and prepare for a presentation. In their presentations, students are expected to show their mastery of basic concepts and knowledge of evaluation. The case study presentation should include: (1) title (2) table of contents (3) introduction (4) explanation of the evaluation process in the case (5) results of the case (6) conclusion/implications (7) discussion points and (8) references. Your presentation is limited to 30 minutes while you effectively use the next 10 minutes after for a Q&A session. Your group is evaluated both for the content you deliver and for the effectiveness of your presentation.
Final Evaluation Project
Students will work as a team to complete an organization-specific evaluation project (e.g., IST’s Distance Education). Students are expected to utilize three unit exercises and add on the impact level of evaluation to this capstone project in order to display their ability to apply what they’ve learned to a real organization. Teams are required to do a pilot presentation on their draft of the report in the second to last week (April 21) and based on feedback they will have one more week to finalize their evaluation report. Instructor will evaluate progress in their final report.
Individual (Personal) Evaluation Project
Students are required to develop an individual (personal) evaluation project. Their evaluation topics can be anything to practice an individual level of evaluation to see progress based on the intended goal. Students work in pairs to monitor progress throughout the semester. We will have three informal presentations (proposal, mid-term, and final) on students’ progress in their individual evaluation project. For their final presentation, they are required to write a one-page report on what they’ve done in their individual evaluation project.
Reflection Paper
Individual students are required to write a (single-spaced, 3 page-limit) reflection paper. This end-of-class reflection paper reflects on the lessons learned from a perspective of what learning and changes have occurred with respect to the field and team work experiences as well as class activities and an individual evaluation project experience. The reflection paper should include: (1) title (2) introduction (3) key points and anecdotes (4) suggestions and (5) conclusions.
Grading Criteria and Due Dates
Assignment / Unit / Points / Due1. Weekly Participation / Individual / 10 / weekly
2. Unit Exercises 2 to 4 (10 x 3)
(Peer Reviews included) / Team / 30 / 2/10
3/3
3/31
3. Case Study Presentation / Group / 10 / your choice
4. Final
Project / Proposal / Team / 2/3
Mid-term Presentation / 3/24
Pilot Presentation / 10 / 4/21 (a draft)
Final Report / 20 / 4/28
5. Individual Evaluation Project (10) / Proposal / Individual/Pair / 1/20
Mid-term Presentation / 3/24
Final Presentation / 10 / 4/28
6. Reflection Paper / Individual / 10 / 5/1 (Sun)
100 points
Grading Policy
The following grading policy has been adopted for graduate courses in the School of Education (http://www.indiana.edu/~bulletin/iu/educ_grad/2005-2007/policies.shtml#grading).
A (95%) / = / Outstanding achievement. Unusually complete command of the course content.A- (90%) / = / Excellent achievement. Very thorough command of course content.
B+ (85%) / = / Very good achievement. Thorough command of course material.
B (80%) / = / Good achievement. Solid, acceptable performance.
B- (75%) / = / Fair achievement. Acceptable performance.
C+ (70%) / = / Not wholly satisfactory. Marginal performance on the course requirements.
C (65%) / = / Marginal achievement. Minimally acceptable performance on course assignments.
C - (60%) / = / Courses with a grade of C- or lower may not be counted in graduate programs.
Be certain that you understand the evaluation criteria before you begin any of the projects. The evaluation guideline sheets are the checklists for content to be included in all project assignments.
Plagiarism and Original Work
We expect that you will turn in original work for every part of every deliverable in this course. We also expect that you make every effort to acquaint yourself with both the IU Code of Student Rights, Responsibilities and Conduct, the concept of plagiarism (start with the required departmental tutorial "Understanding Plagiarism"), and the ways in which you must both credit the work of others and avoid presenting that work as your own (start with the resources from the Campus Writing Program and reference the APA style guide).
Both individual and group/team project work containing plagiarized material will be awarded a grade of F. At the discretion of the instructor, your original work may be turned back to the group/team for correction of the problem before a specified deadline and re-graded for a grade equivalent to or lower than the grade the project would have otherwise received. If your individual/team work is discovered to be plagiarized or to contain plagiarized material, you will receive a failing grade for the course. These policies cover written and graphical work, and all work assigned in the course.
Required Textbook
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco: Berrett-Koehler.
Required Readings
Week 2
Cho, Y., Park, S., Jo, S. J., Jeung, C.-W., & Lim, D. H. (2009). Developing an integrative evaluation framework for e-learning. In V. C. X. Wang (Ed.), Handbook of research on e-learning applications for career and technical education (pp. 707-722). Hershey, PA: IGI Global.
Shrock, S. A., & Geis, G. L. (2010). Evaluation. In J. L. Moseley & J. C. Dessinger (Eds.), Handbook of improving performance in the workplace, vol. 3: Measurement and evaluation (pp. 185-209). Silver Spring, MD: International Soceity for Performance Improvement.
Week 2 - optional
Arthur Jr., W. A., Bennett, W. Jr., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88(2), 234-245.
Christie, C. A., & Fleischer, D. N. (2010). Insight into evaluation practice: A content analysis of designs and methods used in evaluation studies published in North American evaluation-focused journals. American Journal of Evaluation, 31(3), 326-346.
Holton, E. F. III. (1996). The flawed four-level evaluation model. Human Resource Development Quarterly, 7(1), 5-21.
Week 3
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). The four levels; evaluating reaction. Evaluating training programs: The four levels (3rd ed.) (pp. 21-41). San Francisco: Berrett-Koehler.
Lee, S. H. (2006). Constructing effective questionnaires. In J. A. Pershing (Ed.), Handbook of human performance technology (3rd ed.) (pp.760-779). San Francisco: Pfeiffer.
Week 3 – optional
Maher, J. B. Jr., & Kur, C. E. (1983, June). Constructing good questionnaires. Training and Development Journal, 100-110.
Ritter, L. A., & Sue, V. M. (2007). Introduction to using online surveys. New Directions for evaluation, 115, 5-14.
Week 4
Gilmore, E. R. (2006). Using content analysis in human performance technology. In J. A. Pershing (Ed.), Handbook of human performance technology (3rd ed.) (pp.819-836). San Francisco: Pfeiffer.
Thomas, M. N. (2006). Quantitative data analyses. In J. A. Pershing (Ed.), Handbook of human performance technology (3rd ed.) (pp. 837-872). San Francisco: Pfeiffer.
Week 4 – optional
Lehrer, J. (2010, December 13). The truth wears off: Is there something wrong with the scientific method? The New Yorker, 52-57.
Moss, P. A., Phillips, D. C., Erickson, F. D., Floden, R. E., Lather, P. A., & Schneider, B. L. (2009). Learning from our differences: A dialogue across perspectives on quality in educational research. Educational Researcher, 38(7), 501-517.
Week 5
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating learning. Evaluating training programs: The four levels (3rd ed.) (pp. 42-51). San Francisco: Berrett-Koehler.
Morrison, G. R., Ross, S. M., Kemp, J. E., & Kalman, H. K. (2007). Instructional objectives. Designing effective instruction (5th ed.) (pp. 102-129). Hoboken, NJ: John Wiley & Sons.
Week 5 – optional
Morrison, G. R., Ross, S. M., Kemp, J. E., & Kalman, H. K. (2007). Developing evaluation instruments. Designing effective instruction (5th ed.) (pp. 264-305). Hoboken, NJ: John Wiley & Sons.
Week 6
Shrock, S., & Coscarelli, W. (2007). Create cognitive items; create rating instruments. Criterion-referenced test development (2nd ed.) (pp. 121-194). San Francisco: Pfeiffer.
Week 7: No class! I will present my study at the 2011 AHRD Conference in Chicago.
Week 8
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating behavior. Evaluating training programs: The four levels (3rd ed.) (pp. 52-62). San Francisco: Berrett-Koehler.
Marrelli, A. F. (2010). Data collection. In R. Watkins & D. Leigh (Eds.), Handbook of improving performance in the workplace, vol. 2: Selecting and implementing performance interventions (pp. 792-816). Silver Spring, MD: International Society for Performance Improvement.
Week 8 – optional
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63-105.
Hutchins, H. M., Burke, L. A., & Berthelsen, A. M. (2010). A missing link in the transfer problem? Examining how trainers learn about training transfer. Human Resource Management, 49(4), 599-618.
Week 9
Pershing, J. L. (2006). Interviewing to analyze and evaluate human performance technology. In J. A. Pershing (Ed.), Handbook of human performance technology (3rd ed.) (pp.780-794). SF: Pfeiffer.
Pershing, J. A., Warren, S. J., & Rowe, D. T. (2006). Observation methods for human performance technology. In J. A. Pershing (Ed.), Handbook of human performance technology (3rd ed.) (pp. 795-818). SF: Pfeiffer.
Week 9 – optional
Rodriguez, H., Trainor, J., & Quarantelli, E. L. (2006). Rising to the challenges of a catastrophe: The emergent and prosocial behavior following Hurricane Katrina. The Annals of the American Academy of Political and Social Science, 604, 82-101.
Week 10: Spring Break – Enjoy!
Week 11: Individual and final evaluation projects: Mid-term evaluation
Week 12
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating results; implementing four levels. Evaluating training programs: The four levels (3rd ed.) (pp. 63-74). San Francisco: Berrett-Koehler.
Phillips, J. A., & Phillips, P. P. (2010). The business of program evaluation: ROI. In J. L. Moseley & J. C. Dessinger (Eds.),Handbook of improving performance in the workplace: Measurement and evaluation(pp. 219-239). Silver Spring, MA: ISPI.
Week 12 – optional
Parry, S. B. (1996). Measuring training’s ROI. Training & Development, 50(5), 72-77.
Russ-Eft, D., & Preskill, H. (2005). In search of the Holy Grail: Return on investment evaluation in human resource development. Advances in Developing Human Resources, 7, 71-85.