Creating Valid and Reliable Reactionnaires That Predict Behavior Transfer

Introduction

This session will enhance your knowledge and skills in creating valid and reliable reactionnaires that can predict behavior transfer. These evaluation skills are important to learn because many practitioners use reactionnaire data as the only measure of training success. Even with its popularity, practitioners create instruments that are biased towards the positive, collect general information, lack content validity and are unreliable. This session shows you how to overcome these problems and create sophisticated evaluation tools.

Objectives

At the end of this training session, you will be able to:

 edit corporate reactionnaires to include six assessment guidelines;

 create a self-efficacy section within a reactionnaire;

 use the self-efficacy section to prove content validity and internal

consistency reliability: and,

 predict behavior transfer using post-training self-efficacy scores.

Outline

To meet the above objectives, the session outline is:

I. Edit corporate reactionnaires to include six assessment guidelines.

a. Practice

b. Debrief

II. Create a self-efficacy section within a reactionnaire.

a. Practice

b. Debrief

III. Use the self-efficacy section to prove content validity and internal

consistency reliability.

IV. Predict behavior transfer using post-training self-efficacy scores.

V. Final questions

I. Edit corporate reactionnaires to include six assessment guidelines.

I. Continued -- Edit corporate reactionnaires to include six assessment guidelines.

I. Edit corporate reactionnaires to include six assessment guidelines.

Practice:For each reactionnaire item, indicate which guideline is being violated.

This example of a poorly created reactionnaire has been modified from the original that appears in "Creating Evaluation Instruments That Predict Behavior Transfer: A New Theory And Measures In Training Evaluation" which is a Third House Inc. publication. Third House Inc.

( has given permission for one time use in this ISPI presentation (April, 2002). Any other use is in copyright violation.

II. Create a self-efficacy section within a reactionnaire.

For Example:

Practice:Create a self-efficacy instrument from this session's performance objectives.

 Edit corporate reactionnaires to include six assessment guidelines.

 Create a self-efficacy section within a reactionnaire.

 Use the self-efficacy section to prove content validity and internal consistency reliability.

 Predict behavior transfer using post-training self-efficacy scores.

III. Use the self-efficacy section to prove content validity and internal consistency reliability.

A content valid instrument is an instrument that: parallels the training objectives to the evaluation instrument items, emphasizes a proper balance between evaluation instrument items and training objectives, and, is free from prerequisites information or skills not taught in training (Lanigan, 2001b).

For example:

Let's say you are creating a self-efficacy instrument on the training session--How To Make Coffee.

Below are the performance objectives and the time it took to deliver the content for each performance objective.

Performance Objectives/Delivery Time

1. Open coffee can15 minutes3. Pour water into coffee pot15 minutes

2. Scoop coffee into filter30 minutes4. Turn coffee machine on15 minutes

EX: A content valid self-efficacy instrument for the "Making Coffee" training.

My confidence in …

1. opening a coffee can.5 4 13 2

2. scooping coffee into a filter. 5 4 3 2 1 (x2)

3. pouring water into the coffee pot.5 4 3 2 1

4. turning the coffee machine on.5 4 3 2 1

A Content Valid Instrument

Create a content valid, self-efficacy instrument based on the performance objectives and additional information. Training Session on: "How To Lay Sod"

Training ObjectivesDelivery Times

1.Fertilize soil10 minutes

2.Place sod on soil30 minutes

3.Roll sod with roller20 minutes

4.Water sod for one hour10 minutes

Internal consistency reliability is obtained by correlating each exam item to the total score. Internal consistency is illustrated by Alpha. A high alpha (for example .80 or higher) gives us confidence that the self-efficacy items are working together to generate reliable results. Measuring internal consistency requires you to take the data obtained from the self-efficacy instrument and load it into a statistical software package.

Step 1: Take data from self-efficacy instrument.Step 2: Enter data into software program.

Have software multiply item 2 results by 2.

Step 3: After entering all the data, run a reliability analysis.Step 4: Determine the alpha level.

IV. Predict behavior transfer using post-training self-efficacy scores.

There are studies to support the predictive validity of using self-efficacy scores to predict behavior transfer, and in using self-efficacy instruments as a substitute for knowledge and skills tests.

For Example:

Lanigan, M., Abbott, R. Barrett, T. and Smith, P. (in review). Making training evaluations easier by

examining the relationship between self-efficacy to knowledge, to skills and to simultaneous

pre/post instruments. PerformanceImprovement Quarterly.

Machin, M.A., & Fogarty, G.J. (1997). The effects of self-efficacy, motivation to transfer, and situational constraints on transfer intentions and transfer of training. Performance Improvement Quarterly,

10(2), 98-115.

Pajares, F. (1996). Self-efficacy beliefs in academic settings. Review of Educational Research, 66(4), pp.

543-578.

Work Referenced Within This Document

Lanigan, M. (2001a). Creating evaluation instruments to predict behavior transfer:

A new theory and measures in training evaluation. Tinley Park, IL: Third House

Inc.Recipient of a 2002 ISPI Award of Excellence for

Outstanding Instructional Communication.

Lanigan, M. (2001b). Evaluation course on-line: Reliability and validity. (Module 3).

Third House Inc. (O-line). URL:

V. Final questions

Appendix

Reactionnaire Example

This example of a reactionnaire has been modified from the original that appears in "Creating Evaluation Instruments That Predict Behavior Transfer: A New Theory And Measures In Training Evaluation" which is a Third House Inc. publication. Third House Inc. ( has given permission for one time use in this ISPI presentation (April, 2002). Any other use is in copyright violation.

Training Evaluation: Reactionnaire

Your input regarding the questions below is most appreciated. This survey is anonymous, so please answer honestly. Your input will be used to enhance future classes.

Section I: Self-Assessment

High 5 4 3 2 1 Low

Confidence Confidence

For each item below, please circle the response that best reflects your confidence in carrying out each skill upon entering and exiting this class.

Please use the following scale:

Entering this classExiting this class

My confidence in being able to...

5 4 3 2 1(1) define the four levels of the traditional5 4 3 2 1

training evaluation model.

5 4 3 2 1(2) name the three variables that influence 5 4 3 2 1

behavioral intentions.

5 4 3 2 1(3) name the one variable that directly 5 4 3 2 1

influences actual behavior.

5 4 3 2 1(4) name the five tiers of the new training5 4 3 2 1

evaluation model.

5 4 3 2 1(5) name the evaluation instrument that is the5 4 3 2 1

best predictor of actual behavior.

5 4 3 2 1(6) create a self-efficacy instrument.5 4 3 2 1

Section II: Assessment Regarding the Training Class

For each statement below, please circle the response that best represents your belief about the training session. Please use the following scale.

3 = Perfect 2 = Too much 1 = Too little

Facilitator Effectiveness

How would you assess the facilitator’s...

1.projection of voice?3 2 1

2.rate of voice?3 2 1

3.energy level?3 2 1

4.clarity in answering questions?3 2 1

Adequacy of Facilities

How would you rate the training room in terms of...

5.comfort of the chairs3 2 1

(3 = comfortable, 2 = too hard, 1= too soft)

6.temperature

(3 = comfortable, 2 = too hot, 1 = too cold)3 2 1

7.ability to hear people

( 3 = perfect volume, 2 = too loud, 1 = too soft)3 2 1

8.cleanliness of the restrooms

(3 = clean, 2 = too sanitary, 1 = too dirty)3 2 1

For each statement below, please circle the response that best represents your belief about the training session. Please use the following scale.

Excellent3 2 1 Poor

Training Manual

How would you rate the training manual in terms of...

The material in Chapter 1:

9.clarity3 2 1

10.visual appeal3 2 1

11.usefulness as a job aid when you return to the field3 2 1

The material in Chapter 2:

12.clarity3 2 1

13.visual appeal3 2 1

14.usefulness as a job aid when you return to the field3 2 1

The material in Chapter 3:

15.clarity3 2 1

16.visual appeal3 2 1

17.usefulness as a job aid when you return to the field3 2 1

Thank you for your participation.

Which self-efficacy instrument(s) is/are content valid based on the exercise information.

Create a content valid, self-efficacy instrument based on the performance objectives and additional information.

Training Session on: "How To Lay Sod"

Training ObjectivesDelivery Times

1.Fertilize soil10 minutes

2.Place sod on soil30 minutes

3.Roll sod with roller20 minutes

4.Water sod for one hour10 minutes

The scale for the three instruments below is: High 5 4 3 2 1 Low

ConfidenceConfidence

Self-Efficacy Instrument #1

My confidence in…

1.fertilizing soil.5 4 3 2 1

2.placing sod on soil.5 4 3 2 1

3.rolling sod with a roller.5 4 3 2 1

4.watering sod for one hour.5 4 3 2 1

Self-Efficacy Instrument #2

My confidence in…

1.fertilizing soil.5 4 3 2 1

2.placing sod on soil.5 4 3 2 1(multiply answer by 3)

3.rolling sod with a roller.5 4 3 2 1(multiply answer by 2)

4.watering sod for one hour.5 4 3 2 1

Self-Efficacy Instrument #3

My confidence in…

1.growing sod.5 4 3 2 1

2.placing sod on soil.5 4 3 2 1(multiply answer by 3)

3.rolling sod with a roller.5 4 3 2 1(multiply answer by 2)

4.watering sod for one hour.5 4 3 2 1