Project Title:
Assessing the Impact of Labour Market Information (LMI)
Project Number:
9042797
Assessing the Impact of Labour Market Information:
Final Report on Results of Phase II
(Field Tests, Francophone Population)
Research Team:
Céline Rénald, Lynne Bezanson, Elaine O’Reilly, Sareena Hopkins, Réginald Savard, Bryan Hiebert, Kris Magnusson, Adam McCaffrey, and Sylvain Paquette
Canadian Career Development Foundation and
Canadian Research Working Group on Evidence-Based Practice in Career Development
Preliminary Report Authors:
Réginald Savardand Sylvain Paquette, in collaboration with Bryan Hiebert
Please direct any inquiries to:
Lynne Bezanson, CCDF
Suite202 – 119 Ross Street
Ottawa, ON
K1Y 5G3
e-mail:
1
FOREWORD
This research report takes stock of the results of a study designed to assess the impact of labour market information on a Francophone population. The study is based on the method and analytical framework developed during a research project that had the same objective but was conducted with an Anglophone population (Hiebert, 2012). For that reason, many sections of the Anglophone report have been reproduced here, sometimes in their entirety, but always in a manner adapted to the context and characteristics specific to the Francophone population.
EXECUTIVE SUMMARY
This study was designed to answer two research questions:a) To what extent is the independent method a sufficient process for clients to use LMI effectively? and b) To what extent does the assisted method offered by a counsellor enhance the effective use of LMI by clients? Supplementary research questions looked at the following points: a)For clients with a careerchoice or jobsearch need, what type of assistance in using LMI leads to what kinds of outcomes? and b) What demographic factors (e.g., age, gender, employment history) influence the effectiveness of an LMI intervention? These questions were addressed in a context where client needs were assessed to determine whether LMI for career choice or LMI for job search was the appropriate intervention.
This project isolated the use of LMI, making it an independent intervention unlike training workshops or career counselling.And since the intervention offered was limited to LMI use (with or without assistance), the term LMI interventionwill beused throughout this report. Consequently, during the employability assessment interview, counsellors selected only those clients who were likely to benefit from LMI when choosing a career or searching for a job. Clients who, for one reason or another, seemed to require more than an LMI intervention were not recruited for the study. To operationalize the intervention, LMI binders were developed for the two employability dimensions targeted in this study: one binder for career choice and the other for job search. In both cases, the binders included lists of available resources, instructions for accessing information, and tips for interpreting and applying LMI information to the client’s situation.The approach as a whole, regardless of the LMI delivery method, was designed primarily to promote career self-management, or self-guidance. The program lasted for three weeks.
To evaluate the effectiveness of the LMI interventions used in this study and their impact on clients’ lives, the evaluation framework developed by the Canadian Research Working Group on Evidence-based Practice in Career Development (CRWG) was used(see Baudouin, Bezanson, Borgen, Goyer, Hiebert, Lalande, Magnusson, Michaud, Renald Turcotte, 2007). To evaluate the impact of career-related interventions on clients, the CRWG evaluation framework uses an approach consisting of three main components: InputsProcessesOutcomes. Inputs are the resources available to help clients change (i.e., achieve the outcomes). Processes are the mechanisms involved in achieving the outcomes. Outcomes are the changes experienced by those receiving services (clients); in other words, the results of the inputs enacting the processes. These three main components (inputs, processes, outcomes) can be thought of as being related in a linear way, although their relationship is not strictly linear. Inputs feed processes, and processes result in outcomes. However, outcomes are also influenced by the inputs (resources) available, and the nature of the inputs, particularly the competencies of the intervention team, influence the process (intervention) that can be enacted.
The participative research approach, sometimes called action research,was adopted for this study (see Buerk, 1998; Hossack, 1997; Johnson Button, 1998). The service providers were employment counsellors and guidance counsellors working at their employment centres with clients who were part of their typical clientele. In Phase 1 of the project, a series of interviews was conducted to obtain a snapshot of how counsellors come to view LMI as a suitable intervention for clients and an overview of the available LMI resources they considered to be the most useful. The results of the interviews were used to develop a standard initial employability assessment protocol for identifying clients with a careerchoice or jobsearch need, the two needs studied in this research project.
The dependent measures used in this study came from researcher-developed questionnaires designed in accordance with the expected outcomes of the LMI intervention and utilizing procedures developed by the CRWG (see CRWG, 2009). The goal of Phase 1 was to review promising LMI intervention practices, augmented by the knowledge of the research team. Then,in accordance with the parameters surrounding the planned intervention (LMI only [not used in combination with any other intervention such as career counselling], used either independently or with minimal assistance), the outcomes that could be expected were identified, then becoming the objectives of the LMI interventions. Next, questionnaires were developed to assess the extent to which the expected outcomes were achieved.The LMI intervention outcomes could be mapped onto a 14-item questionnaire, with the items falling into three categories of client change: changes in LMIknowledge, changes in LMI skills, and changes in personal attributes (confidence and optimism) that might result from participating in the LMI interventions. All of the questionnaire items were rated on a five-point Likert scale ranging from 0 to 4. Since the LMI intervention as a whole targets career self-management, certain items suggest the development of this skill in the careerchoice and jobsearch interventions.
At a descriptive level, the results of this study show that the clients who used either of these methods, i.e., independent or assisted, experienced substantial positive change during the intervention.The self-assessments show that 45% of the pre-intervention responses to the statements relating to skills, knowledge, and personal attributes were “does not apply,” compared with 2% post-intervention. Saying the same thing, but in reverse, 55% of the pre-intervention responses were “does apply,” whereas 98% of the post-intervention responses were “does apply.”The amount of change was similar across all dimensions of the questionnaire. In fact, there was about the same amount of change for knowledge, skills, and confidence and optimism.
If we look at all of the statistical analyses done, we can see that, in most cases, the amount of client change was statistically significant (p0.01) and, at the clinical level, that the scores on the questionnaires doubledbetween the pre- and post-intervention periods.Thus, the changes experienced by the participants in this study were clinically meaningful as well as statistically significant.Moreover, 96% of the participants attributed at least some of the changes they experienced to their participation in the study (of whom 58% attributed the change mainly to the study). When the intervention was over, 30% of the clients were employed (compared with 19% before the intervention began), and two-thirds felt their jobs were a good fit with the career they had planned. Of those who had not found employment, 77% reported that they were sufficiently prepared to continue using the knowledge and skills developed through their participation in the study and did not feel the need for individual or group consultations in career development.
Overall, the answers to the main research questions were in the affirmative:
- LMI tailored to meet a client’s LMI need as part of an independent intervention is sufficient for promoting statistically significant and clinically important change,[1]and assistance from counsellors enhances the impact on many outcomes, especially when the client participates in a career-choice LMI intervention.
- LMI intervention for career choice or job search, whether independent or assisted, produced significant change in general ability to access and use LMI, in knowledge about how to use LMI, in skills for using LMI, in taking action on the information that was accessed, and in personal attributes, such as optimism about career future and confidence in ability to self-manage future career transitions.
- There was a 60% increase in the number of clients working after the intervention was over, compared with before the intervention, and the number of clients who thought their job was a good fit increased significantly.
- The assisted method seemed to produce greater LMI-related change than the independent method, and the amount of change seemed to be greater for the careerchoice intervention.
- Between pre- and post-intervention, the proportion of employed individuals increased in both groups (independent and assisted methods).
It would be reasonable to believe that the procedures used in this study played a role in the positive LMI gains experienced by the participants. A structured LMI needs assessment interview for the careerchoice and jobsearch interventions was essential. In this study, the initial needs assessment successfully identified clients for whom LMI intervention was the most suitable. In order for LMI interventions to be successful, it is important to ensure that the client’s needs are addressed. Often, employment centres develop programs and force their clientele to participate in them. We believe that the reverse is preferable, i.e., that client needs should be identified first and programs then developed to address those needs. The data in this study support this latter contention. The procedures used in this research project made it possible to identify the resources that were designed to address two frequently occurring categories of client need, i.e., LMI for careerchoice and LMI for jobsearch. LMI interventions were then developed, keeping in mind the characteristics of the target clienteles and the expected outcomes. Next, an assessment protocol was developed to evaluateclient needs and identify clients for whom the careerchoice or the jobsearch intervention was suitable, and appropriate procedures were developed to track the extent to which the LMI intervention was implemented as initially planned and document the degree to which clients were engaged in the LMI intervention process. Client outcomes could then be assessed in a manner that made it possible to link client changes to the LMI intervention process.
This is one of the first studies that has made it possible to demonstrate a clear link between the LMI intervention processes in which counsellors and clients are engaged and the outcomes that clients experience.
TABLE OF CONTENTS
FOREWORD
EXECUTIVE SUMMARY
LIST OF TABLES
LIST OF FIGURES
1.RESEARCH QUESTIONS
2.conceptuAl AND mEthodologiCAL FOUNDATIONS OF THE STUDY
2.1Research Methodology
2.1.1Evaluation Framework
2.1.2Research Design
2.1.3Intervention
2.1.4Sample Selection Process
2.1.5Other Relevant Methodological Considerations
2.2Dependent Measures
2.2.1.Evaluationas a Decision-Making Process
2.2.2.Retrospective Evaluation
2.2.3.Psychometric Properties of Survey Instruments...... 11
3.SAMPLE
4.REsults
4.1Self-Assessment Accuracy before LMI Intervention...... 19
4.2Intervention Effectiveness
4.3Supplementary Analyses
4.3.1Gender Differences
4.3.2Age
4.3.3Work History...... 35
4.3.4Unemployment History
4.3.5Attribution for Change
4.3.6Intervention-Specific Changes
4.4Outcomes Related to Impact of LMI Intervention...... 42
4.4.1Action Planning....... 42
4.4.2Employment Status....... 44
4.5Process Data...... 45
4.5.1Employability Assessment Interview....... 45
4.5.2Checklist for Advice and Information Sessions....... 46
4.5.3Tracking Sheets....... 48
5.SUmmarY OF RESULTS...... 49
6.Conclusions AND implications
6.1Research Methodology
6.2Implications for Practice
6.3Next Steps and Recommendations
6.4Conclusion
REFERENCES...... 58
LIST OF APPENDICES
APPENDIX AParticipant Information...... 61
APPENDIX BInitial Questionnaire...... 65
APPENDIX CFinal Questionnaire...... 68
APPENDIX DAnalysis of Initial Questionnaire Scores (pre-test) Compared with Final Questionnaire Scores (post-test after) 77
APPENDIX EChecklist for Employability Assessment Interview...... 79
APPENDIX FChecklist for Advice and Information Sessions...... 83
APPENDIX GCareer Choice Tracking Sheet...... 88
APPENDIX HJob Search Tracking Sheet...... 93
LIST OF TABLES
Table 1. Initial Questionnaire – Inter-Item and Inter-Total Correlations (n = 52)...... 13
Table 2.Final Questionnaire – Before: Inter-Item and Item-Total Correlations (n = 52)...14
Table 3.Client Sample Distribution by Age...... 15
Table 4.Client Sample Distribution by Education Level...... 16
Table 5.Client Sample Distribution by Need and Intervention Type...... 16
Table 6.Client Sample Distribution by General Area of Employment...... 17
Table 7.Client Sample Distribution by Employment Status...... 17
Table 8.Client Sample Distribution by Gender and Intervention Type...... 18
Table 9.Employment Counsellor Sample Distribution, Continuous Variables...... 18
Table 10.Employment Counsellor Sample Distribution, Categorical Variables...... 19
Table11.Differences in Mean between Scores on Initial Questionnaire and Final Questionnaire– Before 21
Table 12.Response Frequencies of Post-Pre Assessments...... 24
Table 13.ANOVAComparing Before and After Mean Total Scores for Two Delivery Methods 25
Table 14. ANOVAComparing Before and After Mean Scores on Knowledge Subscale for Two Methods 27
Table 15.ANOVAComparing Before and After Mean Scores on Skills Subscale for Two Methods 28
Table 16.ANOVAComparing Before and After Mean Scores on Personal Attributes Subscale for Two Methods 30
Table 17.ANOVA Comparing Before and After Mean Total Scores by Gender for Two Methods 33
Table 18.MANOVAComparing Before and After Mean Total Scores by Age Group for Two Methods 34
Table 19.Frequency of ResponsesRelated to Attribution for Change by LMI Delivery Method 36
Table20.Fréquences de réponses aux questions réservées au groupe en choix de carrière (n = 32) 37
Table21. Frequency of Responses to Questions Reserved for Job Search Group (n = 20)..38
Table 22.ANOVA Comparing Before and After Mean Scores, including Intervention-Specific Items 40
Table 23.Components of Client Action Plans...... 43
Table 24.Need for Assistance in Developing an Action Plan...... 43
Table 25.Current EmploymentStatus (Post-intervention)...... 44
Table 26. Job Fit...... 45
LIST OF FIGURES
Figure 1. Framework for Evaluating Client Change...... 4
Figure 2. Dependent Measures...... 9
Figure 3. Before and After Total Mean Scores for Two Methods...... 26
Figure 4. Before and After Mean Scores on Knowledge Subscale for Two Methods...... 28
Figure 5. Before and After Mean Scores on Skills Subscale for Two Methods...... 29
Figure 6. Before and After Mean Scores on Personal Attributes Subscale for Two Methods31
1
1
ASSESSING THE IMPACT OF LABOUR MARKET INFORMATION:
EXPERIMENTATION RESULTS WITH A FRANCOPHONE SAMPLE
This project consists of a program evaluation concerning the impact of labour market information (LMI) on careerchoice and jobsearch behaviour, as well as on the way that program evaluationscan be carried out in the field.The project has three phases. Phase I produced a snapshot of the stateof LMI practices used atvarious employment centres across the country, while PhaseIIcompared two methods of delivering LMI to clients:an independent method whereby clients accessed LMI on their own and an assisted method whereby clients again accessed LMI in their own but benefitted from the assistance of a counsellor.Both LMI delivery methods – independent and assisted – were used with adult clientswith careerchoice or jobsearchneeds. PhaseIIIconsists mainly of a detailed exploration of the nature of the clients’ experience in using LMI by means of semi-structured telephone interviews with a subset of the clients who used one of the two LMI delivery methods. The interviews focused on how the clients accessed and used LMI, with attention to the details of the clients’ experience in relation to how they processed the LMI they accessed. This report presents the results of Phase II of the project.
1.RESEARCH QUESTIONS
This research project was designed to answer two related questions:
Once clients’ needs are assessed and they have been given LMItailored to their needs,
- To what extent is the independent method a sufficient process for clients to use LMI effectively?
- To what extent does the assisted method offered by a counsellor enhance the effectiveness of LMI use by clients?
A number of sub-questions were addressed as well:
- What demographic factors (e.g., age, gender, employment history) influence the effectiveness of an LMI intervention?
- For clients with careerchoice or jobsearch needs, what type of assistance in finding and using LMI leads to what kinds of outcomes?
- How do clients process the LMI they access, and how do they use that information to come up with an action plan?
This project uses an approach that isolates the use of LMI and is independent of other interventions,such as training workshops and career counselling. It is therefore important for readers ofthis report to keep in mind that the main goal of both the assisted and the independent intervention methods is career self-management.Consequently, even though some clients used the assisted method, the objective was still to encourage them to engage in career self-management.
2.CONCEPTUAL AND METHODOLOGICAL FOUNDATIONS OF THE STUDY
In this study, we experimented with some innovative evaluation procedures in order to examine the efficacy of the LMI interventions and delivery methods. Consequently, to help readers understand the procedures used and the rationale and empirical support for using them, a summary of the philosophical and procedural foundations of our approach is provided below.
2.1Research Methodology
A participatory research (sometimes referred to as action research) approach was used in this study (see Buerk, 1998; Hossack, 1997; Johnson Button, 1998). The service providers were career and employment counsellors,[2]working at an employment centre with clients who were part of their typical clientele.The project began with a series of interviews designed to provide a snapshot of the following (Phase I): types of LMI resources the counsellors found most useful, how the counsellors determined that LMI would be an appropriate intervention for a client, and how the counsellors interacted with clients for whom LMI was their primary intervention (seeHiebert, Bezanson, O’Reilly, Hopkins, Magnusson McCaffrey, 2010). The information gathered during the interviews enabled the research development team to make existing practices the cornerstone of the interventions that were subsequently developed.The interview results were used to develop a standardinitial employability assessment protocol for identifying clients who might need assistance with their career-choice decision making or jobsearch efforts, which were the two needs addressed by this research.The commonly used LMI resources, augmented by other resources identified by the research team, were used to develop the LMI intervention binders used in the study.