Project Overview

Common Indicators: Transforming the Culture of Evaluation in Career and Employment Services

In May, 2012, Human Resources and Skills Development Canada (HRSDC) launched a national research agenda to better understand “what works” in career and employment services. This project is 1 of 3 that were approved under this initiative. It is an 11 month project focused on testing a comprehensive evaluation model that tracks:

  • changes in clients as a result of programs and services (new knowledge acquired, new skills acquired, changes in personal attributes such as self esteem and ability to self manage);
  • traditional client outcomes of employment and entrance into training or educational programs;
  • quality issues related to employment or training such as the fit of employment with client skills and qualifications, and fit of training with client preferred future vision; and
  • client employability issues and challenges as well as different employment opportunities in different locations.

The evaluation model has been developed over several years by the Canadian Research Working Group for Evidence Based Practice in Career Development (CRWG). This group, established in 2001, is a partnership among 8 researchers from both Anglophone and Francophone universities and the Canadian Career Development Foundation (CCDF). Depending on the project, all or some of the researchers take lead roles along with CCDF consultants.

The project team for this project is comprised of:

  • The Canadian Career Development Foundation (CCDF) (Lynne Bezanson & Céline Renald);
  • Dr. Guylaine Michaud (Lead Francophone Researcher, Université de Sherbrooke);
  • Dr. Dave Redekopp (Lead Anglophone Researcher, Life-Role Development Group, Edmonton, Alberta);
  • Dr. Reginald Savard (Université de Sherbrooke) and Dr. Liette Goyer (Université Laval);
  • Dr. Bill Borgen (University of British Columbia);
  • Ken Organ of Goss Gilroy Inc. (GGI), a nationally recognized firm known for its expertise in evaluation and the development of accountability frameworks; and
  • YOU – Career/Employment offices across Saskatchewan, Québec and New Brunswick who have volunteered to participate as part of the research team. Many of you have already helped to shape the evaluation model by participating in focus groups, which took place in June, 2012. You will be leading recruitment, field testing the model, and gathering all critical data for analysis. We hope this will be a learning opportunity for you as we work together, and we also know that we will learn immensely from your experience and your feedback. Together we believe that this project has the potential to make a significant contribution to the field—but of course we need to test it to find out if we are right!

THE PROJECT IN A NUTSHELL

The project will develop and test an online tool that will continue to gather traditional indicators including employment and training, include the newer indicators outlined above, and go further to include additional indicators and outcomes not yet tested. The indicators will be based on results of previous CRWG research, input from the literature review and focus groups with front line practitioners and managers who will be partners in the research itself.

All clients who are registered to receive services from a career development practitioner who is participating in the research will be eligible to participate in the research. There are no recruitment criteria.

  • For each client, participating practitioners will conduct a formal employability needs assessment as is normal practice. If the client is going to receive ongoing service from the practitioner (that is, will be seen more than for a single assessment interview), the practitioner will invite the client to participate in the research.
  • All participating clients must sign a consent form and will be asked to complete short questionnaires at the beginning and at the end of the study.
  • Participating practitioners will use the online tool to record client data, results of the needs assessment interviews, the kinds and duration of services given to the client, and incremental changes as appropriate after every client contact.
  • Participating practitioners will also continue to use the data collection system currently in use in their offices. This will require entering some data twice and will necessitate extra time being given to participating practitioners. As we are field testing a new system, we recognize that it is unlikely to be perfect and offices cannot compromise their ongoing data collection responsibilities. These details will be fully negotiated with office managers who are highly supportive of the research study.
  • The duration of the research will be 10 weeks (January to March 31, 2013);
  • Follow-up interviews will be conducted with participating practitioners and managers to explore their experience using the online tool, identify any anomalies and gather recommendations;
  • Participating practitioners will also be asked to complete a short questionnaire at the end of the study.
  • Clients who participate in the research and complete the questionnaires will receive a small honorarium or gift card equivalent ($20.00) in appreciation.
  • Afull report will be prepared on the research, its results, what has been learned, the implications for service delivery and recommended next steps.

Research Questions

The research questions to be addressed are:

1.What common indicators are applicable across different client contexts, different client groups, different agencies, and different interventions?

2.What statements can be made about service effectiveness by tracking common indicators of inputs, processes and outcomes? If numbers permit, it will be possible to aggregate data to begin to address the ultimate question, “What kinds of interventions in what contexts produce what kinds of outcomes?”

Project Objectives

  1. Identify an agreed upon set of common input, process, and outcome indicators to be collected across agencies;
  2. Develop, field test, and implement an online data management tool (English and French) that merges agreed upon common indicators with the CRWG evaluation framework;
  3. Revise and refine the data management tool in accordance with the results of the experimentation and the experience of end-users;
  4. Report on the connections among Inputs, Processes and Outcomes for diverse client employability profiles, specific groups with low labour market attachment and divergent opportunity structures; and
  5. Establish broader implementation plans with participating agencies in order for them to integrate the data management tool into their existing evaluation systems and evaluation into the everyday culture of their services on an ongoing basis.

Anticipated Learning

The tools and processes to be developed and tested in this research will respond directly to the demands from the career practitioner field.For many years, practitioners have asked for more robust data-gathering tools that capture more fully and accurately the results achieved with clients. Employment, reduced time on benefitsand successful retraining are currently measured but, with very few exceptions, other variables influencing success are not: skill acquisition; increases in confidence and optimism; motivation to pursue an employability goal; qualitative measures related to how well employment and/or training fits with individual ambitions and qualifications; and the actual quality of the employment itself.

The tools and processes will capture the growing complexity of career and employment services practices and outcomes and take into account factors that influence and, in many cases, severely limit and/or enhance individual choice. The tools will begin to map learning outcomes onto interventions and recognize in practice that “one size does not fit all.”

Because of funding requirements, the research must be completed by March 31, 2013. It is recognized in advance that a period of ten weeks is very short to be able to capture changes in knowledge, skill, and attitude as well as sustainable employment and long-term savings to benefit programs. However the researchers are convinced that it is long enough to begin to see trends in client change. If this proves true, we will be on solid ground to seek support for longer-term experimentation and implementation.

The tools and processes developed in this project will not answer all demands for more robust evidence, but will move the evidence base significantly forward. The common indices data to be gathered will include a considerably broadened range of outcomes, all of which will measure levels of client change. Participants in this research will be making an important contribution to building a stronger and more holistic culture of evaluation in career and employment services.


Employability Assessment Interview

The U-Name It tool is designed to capture the amount of client change which occurs as a result of career and employment services received over a period of time. In order to do this, the system needs to have a baseline of where the client is at the beginning of receiving services. This baseline will allow us to capture both client progress and client outcomes at the end of the research period.

Therefore the Assessment or Intake interview is very important to the study. We recognize that practitioners have different styles for conducting an assessment interview and, rather than provide a script, we have identified the factors that should be explored in order to assess a client’s employability needs and tailor services and programs directly to that client’s needs. We have also included an Assessment Checklist as a tool for your personal use should you find it useful.

The study is open to all clients who are going to see a practitioner more than once – that is all active clients. Existing as well as new clients are welcome into the study. In all cases however a baseline is needed so an Assessment Interview capturing where the client is NOW at time of entry into the study, not at time of entry into services, is essential. As long as the system has a client starting point or baseline, it will be able to track client change.

An Assessment/Intake Interview is the entry point into the research.

Protocol for Employability Assessment Interview

This is not a protocol in the strict sense of the word. We recognize that practitioners have different styles for conducting an interview and, rather than provide a script, we have identified the factors that should be explored in order to assess a client’s employability needs and tailor services and programs directly to that client’s needs. Generally speaking, the interview will follow the three steps below and address the content itemized in the “Points to Help in Determining Need” chart as well as the “Checklist for Employment Assessment Interview,” both in this section.

Step 1: Establish A Collaborative Relationship
  • Greet client and determine client objective
  • Establish the purpose of the interview
  • Confirm motivation and client commitment
Step 2: Gather employability information
  • Gather data on employability of the client
  • Job-readiness
  • Self-awareness and career/employment direction/goal
  • Ability to look for work
  • Meeting occupational requirements/getting qualified
  • Ability to maintain work
  • Verify client perspective
  • Obtain agreement on identified needs
Step 3: Develop an action plan
  • Come to agreement on a goal to be achieved
  • Come to agreement on what “success” for the client at the end of your work together will look like
  • Come to agreement on first steps to take towards the goal
  • Check client commitment
  • Establish follow-up plans
  • Close interview

Employability Dimensions

Client employability needs are organized into five broad employability dimensions: Pre-Employability/Job Readiness; Career Decision Making; Work Search; Skill Enhancement and Employment Maintenance. Each employability dimension lists a number of indicators which suggest specific needs within each dimension. No list of indicators can be complete; there must always be room for an “Other”category to capture the diversity of client needs. In general however, within career and employment services, needs fall into the following general dimensions:

PRE-EMPLOYABILITY/JOB READINESS: Has the intention to seek employment and/or enter training but faces challenges, vulnerabilities and/or skill deficits that need to be addressed before the client can be successful in achieving employment/training goals

CAREER DECISION MAKING: Not yet clear about work/learning goals; needs to understand personal skills, interest, values and personality and more about different potential occupations in order to develop a vision for a preferred future

SKILLS ENHANCEMENT: Has career goal, but needs to explore skills and qualifications needed for the work they want and how to acquire them

WORK SEARCH: Has career goal and necessary skills/qualifications for desired work; needs skills, strategies and tools necessary to conduct a successful job search

EMPLOYMENT MAINTENANCE: Has difficulty keeping work or may need to work on the skills/attitudes required for success at work

OTHER: Many circumstances beyond those covered by the employability dimensions may impact on the client’s employability outcomes. Some of these life circumstances may necessitate referral outside of career and employment services. Life circumstance issues may need to be addressed and resolved before progress toward employment can realistically be made.


Points to Help in Determining Need

Pre-Employability/Job-Readiness / Client needs assistance to:
  • identify and clarify future direction (training, education , employment or change in life circumstances goal)
  • identify personal strengths/resources that support future direction (training, education, employment or change in life circumstances goal)
  • resolve specific financial challenges/vulnerabilities that may impact on future direction (mortgage, public transit, childcare etc.)
  • access community resources/supports that address specific personal challenges/vulnerabilities and assist in helping move goal forward (mental health services; addictions counseling; public housing etc.)
  • develop necessary personal supports needed to move toward future direction (family, childcare, transportation)
  • acquire life/employment skills which support future direction
  • acquire attitudes which support future direction
  • Develop and follow a plan of action to move future direction forward

Career Decision Making / Client needs assistance to:
  • identify own strengths, skills, interests, values, transferable skills
  • connect strengths, skills, interests, values, transferable skills to career choices
  • research work opportunities for work using a range of sources (job boards, labour market information, internet, networks, employer and employee contacts)
  • research details specific to career goal (time needed in education/training; future employment prospects; types of work; places of work; local opportunities)
  • identify a career goal (training, education , employment or change in life circumstances)
  • identify personal resources that support achievement of career goal (support system, finances, motivation)
  • identify challenges, internal or external, which may interfere with achievement of career goal (mobility, local opportunities, finances, health)
  • develop and follow a plan of action to mitigate challenges and move toward career goal

Work Search / Client needs assistance to:
  • confirm employment goal
  • confirm that qualification and experience are consistent with employment goal
  • identify personal strengths that support successful work search
  • identify potential employers and potential employment opportunities
  • adjust/adapt employment goal with employment opportunities as needed
  • identify transferable skills
  • complete appropriate resume and cover letter
  • use networks to identify employment leads
  • use resources, tools and methods to support work search including internet
  • tailor resume and cover letter according to work possibilities
  • demonstrate appropriate job interview skills
  • develop and follow a plan of action for active work search
  • demonstrate work attitudes and behaviours to support successful job search
  • be active and persistent in work search

Skills Enhancement / Client needs assistance to:
  • confirm training/education goal
  • research future employment prospects related to training/education goal/program prior to pursuing education/training
  • research training/education options available to achieve training/education goal
  • evaluate and choose training/education options taking into consideration personal circumstances (supports, strengths and limitations)
  • identify potential challenges that may impact on achievement of training/education goal
  • proactively develop strategies to address identified challenges
  • acquire study and personal skills needed to be successful in education/training
  • identify resources that are available in training and education sites and/or community agencies to provide help and guidance supporting successful completion of program
  • sustain motivation to complete training/education program

Employment Maintenance / Client needs assistance to:
  • identify important skills and attitudes that improve chances of keeping employment
  • identify personal strengths and limitations with respect to these skills and attitudes
  • develop a learning plan to acquire/strengthen skills and attitudes before on the job problems arise
  • identify community and/or workplace resources that provide help and guidance related to keeping work
  • know job roles, responsibilities and expectations that support being successful on the job (who to report to; who makes decisions; approval processes, getting answers to job reacted questions)
  • assertively seek assistance when needed
  • develop and follow a plan to remain up to date with on the job changes in duties and competencies

Additional Life Circumstances /
  • improved housing is needed
  • improved transportation is needed
  • improved capacity to work and/or study is needed
  • Increase in sense of responsibility for own choices and behaviours is needed
  • Increase in ability to set short and long term goals is needed
  • Increase in understanding expectation and demands of employers is needed
  • Reduction in destructive behavior is needed
  • Following medical and medication protocols is needed
  • Increased openness to change is needed
  • Increased access to constructive and positive support systems is needed
  • Improved relationships with family and friends are needed
  • Increased trust in other people is needed

Personal Attribute Needs / Client needs assistance to:
  • Develop stronger self-management skills (able to take steps forward independently; open to using available resources; stay motivated to follow steps in action plan)
  • Develop stronger self-esteem (optimism; confidence; sense of self as competent)
  • Develop stronger sense of personal well-being (good energy to pursue goal; willing to use support systems and available resources; open to learning and suggestions)
  • Develop stronger self-efficacy (belief in own capacity to overcome challenges; confident in a positive outcome despite obstacles)

Checklist for Employability Assessment Interview:You may find this a useful checklist to keep you on track in your Employability Assessment Interviews and to confirm that you have not omitted some important areas for exploration. Many professions (surgeons; pilots) use checklists constantly and they have been proven to reduce mistakes and errors. It is not required that you complete the checklist; it is for your own professional growth.