Project Evaluation Summary Report for The

IECPCP Project # 6804-15-2005/6880027

Project Evaluation Summary Report for the

Institute of Interprofessional Health Sciences Education (IIHSE)

Submitted June 2008

Patty Solomon1

Carole Orchard2

Sue Baptiste1

Pippa Hall3

Robert Luke4

Ellen Rukholm5

Lorraine Carter5

With contributions from:

Beckie Tingley

Gissele Damiani-Taraba

Susanne King

1 McMasterUniversity, School of Rehabilitation Science

2 University of WesternOntario, School of Nursing

3 University of Ottawa, Faculty of Medicine

4 GeorgeBrownCollege, Office of Research and Innovation

5 Laurentian University, School of Nursing

Table of Contents

Overview of IIHSE Project

The Dilemma

Purpose and Goals of the Project

Design of Project

Project Evaluation

Pre-Licensure Stream

Overview of Pre-Licensure Stream

Recruitment of Participants

Modules

Description of Participants

Data Collection and Analysis Methods

Results

A. Students’ Feedback on the Modules

1. Overall Look and Layout of the Module:

2. Educational Content:

3. On-line Discussions and Team Work:

4. On-line Facilitation:

5. On-line Support:

Interpretation of the SMFF Data:

General Conclusions about Students’ Feedback on the Modules

B. What We Learned from the Discussion Forum

Content:

Process:

Summary of Students’ Data from the Discussion Forum:

C. Students’ Perceptions of IPE: Using a Standardized Measure

Summary of IEPS Data:

D. Perceptions of On-line Facilitators and Students via Interviews

What We Learned from the Interviews with Students:

Summary of Students’ Interview Data:

What We Learned from the Interviews with On-line Facilitators:

Summary of Data from Interviews with On-line Facilitators:

Conclusions for Pre-Licensure Stream

Practice Stream

Description of Purpose, Sites and Team Development Modules

Recruitment and Description of Participants

Description of Project at Each Site

Data Collection and Analysis Methods

Results

A. Clinicians’ Feedback on the Modules

1. Overall look and layout of the modules:

2. Educational Content:

3. On-line discussion and team work:

4. On-line support:

B. Findings from Pre and Post Standardized Measures

C. Results from Project Evaluation and Feedback Form (PEFF)

D. Perceptions of Clinicians and IPEFs via Interviews

Conclusion of Interview Findings:

Practice Stream Conclusions

Clinical IPE Placements

Overall Summary/Conclusions/Comments

Reference List

List of Appendices

List of Tables

Table 1: Student Modules Enrolment – Breakdown by Discipline……………………………....5

Table 2: Demographic Characteristics of Students………………………………………………6

Table 3: Comparative Data of IEPS Factor Scores..……………………………………………14

Table 4: Analysis of Interdisciplinary Education Perception Scale (IEPS) ..…………………..15

Table 5: Composition of Clinical Teams at the 5 Practice Sites..………………………………28

Table 6: Practice Stream: Analysis of Standardized Measures for Overall Sample...………….33

Table 7: Summary of “Project Evaluation and Feedback Form for Team Members”.…………35

1

The Dilemma

Preparing health care students and health care teams to work collaboratively to enhance patient-centred practice and patient outcomes.

Scenario 1: Preparation of Health Care Professionals

Students in health care are trained as professionals with the technical skills and knowledge required to perform their roles. In addition, they need to develop other skills such as communication, collaboration and joint decision-making. Generally these students learn within uni-disciplinary programs. Their clinical placements may provide them with exposure to other professions but opportunities for interprofessional education (IPE) in the curriculum and interprofessional collaboration (IPC) in clinical placements are limited. One major programmatic barrier to IPE is scheduling. Their curricula are already full. It is a considerable challenge to coordinate the schedules of students from various programs within one institution to engage them as a group in IPE and in a location sufficiently large to accommodate large numbers of students. The scheduling problem is further exacerbated when learning with and from others from other universities and colleges is considered.

Scenario 2: Post-Graduation and Employment in a Clinical Setting

Once students graduate, some are hired onto teams for which they may not be prepared, if they have had little or no IPE. Additionally, the team may or may not function well: members do not know or appreciate the contributions of others’ expertise on the team; there are implicit or explicit territorial issues; decision-making is not shared; communication may not be open and honest. Even if students have previously participated in IPE, they are surprised, dismayed and discouraged as their training and enthusiastic embrace of interprofessional collaboration meets up with the real world.

Purpose and Goals of the Project

One solution to this dilemma was the creation of theInstitute of Interprofessional Health Sciences Education (IIHSE) through the Health Canada funding initiative, Interprofessional Education Collaborative for Patient-Centred Practice. The IIHSE is a virtual network of expertise designed to develop knowledge, skills and attitudes, and promote cultural change, in and among health science students and clinicians. The IIHSE is currently comprised of faculty from four Ontario universities and one college: McMasterUniversity, University of Ottawa, University of WesternOntario, Laurentian University and GeorgeBrownCollege.

The overall goal of this project was to facilitate interprofessional collaboration in educational and practice settings through the use of web-based and team-based learning activities.The specific objectives of the project include:

•to promote, implement and evaluate interactive web-based learning modules focusing on interprofessional knowledge, attitudes and skills.

•to develop and evaluate a process of team development that uses on-site facilitators and targets patient outcomes for patient-centered care.

•to pilot and evaluate an educational process for health professional students that is based on a continuum of learning from academic to practice settings.

•to create a sustainable collaborative consortium of educational institutions to support the Institute.

•to stimulate networking and sharing of best practice approaches

•to disseminate high quality interprofessional learning products

This report focuses on the first three objectives; the latter three are addressed in the Final Report to Health Canada(see citation in Reference list).

Design of Project

To address the dilemma of the scheduling problem and to take advantage of the appealof on-line learning, students were offered IPE experiences through asynchronous on-line modules. These were designed to provide opportunities to learn about, with and from each other, in a forum that is facilitated by a faculty member. The students worked on clinically relevant problems and engaged in team work and joint decision-making. Ideally, they were to then participate in a clinical placement with a group of health care professionals who had undertaken a team development process to improve their own team function so as to model IPC for and with the students.

Project Evaluation

As this project was designed to assess educational processes and outcomes, it was essential to have formative evaluation throughout the duration of the project and to capture summative information upon completion.A mixed-methods comprehensive evaluation using quantitative and qualitative measures was integrated throughout the project. Our experiences (Solomon et al., 2003) and the literature would support that it is important to incorporate qualitative methods in the analysis and evaluation of complex processes and outcomes related to interprofessional education.

There were two foci for this project: a Pre-Licensure Student Stream and a Practice (Clinician) Stream. The Implementation Phase of the project operated independently for these two streams, though these eventually merged as the students entered their clinical fieldwork placements. The evaluation results are presented for each stream separately. Data from the clinical inter-professional placements are reported after the two streams.

Overview of Pre-Licensure Stream

The student experience consisted of two elements: learning via interactive web-based learning modules followed by clinical practice experiences. The literature indicates one of the biggest barriers to interprofessional learning at the pre-licensure phase relates to time table conflicts. Web-based courses that can be completed asynchronously were chosen to allow students to access the learning modules at times that are most convenient to them. Web-based learning was complemented by interprofessional clinical placements with teams that were targeted to participate in a team development and facilitation process. The modules were designed to be facilitated by a faculty member experienced in interprofessional education.

Recruitment of Participants

Students were recruited at each of the four universities by a site research assistant. Various methods were used to make students aware of the project and the opportunity to participate in piloting the on-line modules. These included brochures, posters, notices via programs’ email lists, announcements to students in classes and personal invitation.Students were targeted in programs for medicine, nursing, occupational therapy, physiotherapy and speech-language pathology and audiology. However, the modules were not restricted to these disciplines. Generally, students could take only the module offered at their own university. The exception was at Laurentian where students from any of the four universities could enroll because it did not have as many health discipline programs from which to draw students. We received Ethics Board approval from all universities involved in the project.

Modules

Descriptions of the modules are provided in Appendix A. These modules are listed here with their shorter names which are used in Table 1. Also noted are the universities where the modules were developed.

Abor HealthAboriginal Health/La santé des Autochtomes, Laurentian University

Comm Part1Communication Part 1, McMasterUniversity

Comm Part2Communication Part 2, McMasterUniversity

CPHPCommunity Practice: Health Promotion, University of WesternOntario

EBPInterprofessional Stroke Care: An Evidence-Based Approach, McMasterUniversity

EthicsEthics, McMaster University

Rural HealthSoins de santé interprofessionnels en milieu rural: le patient et sa communauté,RuralHealthcare: The Patient and the Community,University of Ottawa

Total PainTotal Pain/Palliative Care, University of Ottawa

Most of the modules were delivered through WebCT, a learning management system for on-line courses. Two of the modules (the ones at University of Ottawa) used a different software called Thinkmap.

Description of Participants

We display in Table 1, the breakdown of the enrolment of students in each module by their disciplines. The five major targeted disciplines are listed first in column 1 with the other programs noted as well. The modules’ short names are displayed across the top of the table in alphabetical order. Some of these modules were run more than once. A total of 156 students enrolled in the project and of those, 83 (or 53%) completed the entire module.

Looking across the row at the bottom of Table 1, we see the percentage of students who completed each module. These ranged from 25% to 100%. There were a number of factors that contributed to this variation as reported by the students who dropped out and completed a Non-Completion Form.Time of year was a major factor in that several of the modules were run during the spring or summer months. Studentswere juggling their regular curricula, clinical placements, work schedules and personal/family life. For some, the amount of work was more than they expected; however the time commitment was approximately 3 hours per week. Students volunteered to participate and no academic credit could be given at this time (although this is being pursued). It is important to note that some of the modules were the length of a full semester (12 weeks) and it is commendable that students remained in the module to its completion.

Of the 83 students who completed a module, six of them took more than one module, resulting in a sample of 77 different participants from whom we collected basic demographic data. Information on a few characteristics was not collected from the 15 students who participated in the early pilots of the Total Pain module which accounts for some of the missing data. The features of the sample are presented in Table 2. Most students were female (87%). The age range varied widely from 19 to 61 with an average age of 26.0. Of the health profession programs, most students were from Occupational Therapy (30%), Medicine (19%), and Nursing (18%). Over a third of the students (38%) were in the first year of their programs.

1

1

Data Collection and Analysis Methods

At the time of enrollment into the study, students completed a consent form; an Information Sheet containing demographic questions and items about their interest in the modules; and a standardized measure (described below) about their perceptions of interprofessional education. They agreed to complete this same measure after they finished the module. They also consented to provide feedback on the module itself and on the whole project and to participate in a focus group (or individual interview) about their experience. An honorarium of $50 was sent to the students once all components were completed.

Evaluationsof the modules and the students’ experiences are outlined below. These data and perceptions of students are presented in this order in the next Results section.

Student feedback of web-based learning modules

Content analysis of on-line discussions

A standardized measure, the Interdisciplinary Education Perception Scale (IEPS; Luecht et al., 1990) was administered pre and post module experience

An overall project evaluation form was completed after the modules and prior to focus group sessions to avoid contamination by and of the other group members

Focus groups or individual interviews at completion of the project

Results

A. Students’ Feedback on the Modules

We wanted to gather information from the students about their perceptions of the module they chose and their on-line experience with it.One way to do this was through a questionnaire. We developed the Student Module Feedback Form (SMFF)that had five major sections:

1)Overall Look and Layout of the Module

2)Educational Content

3)On-line Discussions and Team Work

4)On-line Facilitation

5)On-line Support

The SMFF had 55 items,containing both categorical and open-ended questions. There were 39 questions that had a 5-point response option from “Strongly agree” to “Strongly disagree”; the middle position was “Neither agree nor disagree.” Six questions were yes/no in nature. There were also 10 open-ended items.

Most students completed the SMFF in an on-line format which was downloaded for analyses. For this report, data have been combined across all 8 modules and 68 students provided feedback. We report first on three of the yes/no questions that relate to students’ previous experiences. This is followed by findings from the categorical questions in each of the five major sections. The remaining yes/no and open-ended questions provided formative feedback which we used to modify the modules. The specifics of these are too lengthy to present in this report.

It was important for us to understand what kinds of experiences students may have had with on-line modules and with working with other students. From the first 3 yes/no questions we learned that:

• 66% had previous experience with on-line learning
• 66% had previous experience with on-line discussions
• 60% had previous experience working with students from other professions.

For the 39 categorical questions, we examined the how often each response option was used for each item. These frequency findings are presented in Appendix B (Tables B.1 through B.5) as the percentage of respondents using the top 2 (out of 5) agreement or disagreement categories. That is, the number of students who indicated agree or strongly agree (and similarly disagree or strongly disagree) were combined to provide the percentages reported in the tables. We summarize the findings below according to the five major sections on the questionnaire.

1. Overall Look and Layout of the Module: It was important to ask about the design and function of the modules’ features. The data in Appendix B, Table B.1 show that several of these aspects were rated fairly high(70% or higher) including those indicating that the modules were clear in indicating the estimated length of the course, the language used was easy to understand, the videos were realistic, and functions of icons were evident.Items with lower ratings were those where the participants thought the videos were too long or all special requirements (e.g., modem speed) were not identified.