Virtual Outcomes College

Virtual OutcomesCollege

National ROMA Peer-To-Peer Training Program

Results-Oriented Management

and Accountability for

Community Action Agencies

and CSBG Subcontractors

ROMA Trainer Manual

Version 3.3 – Day Two

Teaching Guide – Learning Objectives

FY 2004-2005

A Community Action Network Peer-to-Peer Project

Funded by the U.S. Department of Health and Human Services

Office of Community Services

Assignment #5 – Analysis of the Family Development Matrix64

Module Eight A – Carter Questions and Return-On-Investment67

Reginald Carter Seven Questions for Developing an Outcome Framework……..68

An Example Using the Carter Methodology71

The Building Blocks for Return-On-Investment: Nine Key Questions77

The Building Blocks for Return-On-Investment: Nine Key Questions – Example79

Sample Outcome Scale Using ROI Calculations81

Sample Outcome Scale Using ROI Calculations – Scenarios 82

Module Eight B – Implementing A ROMA System84

Implementing a ROMA System in Your CAA85

Implementing a ROMA System in Collaboration with your State………... 87

Appendices

Located in Participant Manual

Estimated Timing

Module Six90 minutes (9:00AM-10:30AM)

Break15 minutes (10:30AM-10:45AM)

Module Six Continued30 minutes(10:45AM-11:15AM)

Module Seven45 minutes (11:15AM-12:00PM)

Lunch60 minutes (12:00PM-1:00PM)

Module Seven Continued90 minutes(1:00PM-2:30PM)

Break15 minutes (2:30PM-2:45PM)

Module Eight A45 minutes(2:45PM-3:30PM)

Module Eight B30 minutes (3:30PM-4:00PM)

Please use this timetable as a guide to structure your day. Stay close to the schedule.

Module Six

Logic Model

Time: 120 Minutes (2 hours)

Logic Model

Introduction to the Logic Model – Page 2
Learning Objectives:

Learn how to use the ROMA Logic Model.
Be able to distinguish between short, intermediate, and long-term outcomes.
Demonstrate that analysis of the data in a logic model can lead to improved program management and decision-making.

The logic model is a tool that integrates programoperations and programaccountability. The logic model can be used to support planning, monitoring, evaluation, and other management functions of the CAA.
The logic model links programoperations, Columns 1-5 of the logic model (mission, need, intervention, projected results, actual results), and programaccountability, Columns 6-8 of the logic model (measurement tool, data source, and frequency of data collection and reporting including personnel assigned to function).
The logic model, Emergency Housing 2.0A, on page 4, documents a specific need, “families are at risk of being evicted or are homeless.” A review of the results in Column 5 indicates that the intervention, an emergency rent payment, produces a 100% positive outcome in the first 30 days. Follow-up data reveals a different story. After 60 days, families increasingly lose their housing. The same pattern exists after 90 days.
This CAA used the logic model to document the emergency housing program, and monitor performance. Collecting and analyzing follow-up outcome/performance data revealed the limits of the intervention and suggested that other factors may adversely be affecting families in the community.
The logic model, Housing Assistance 2.0B, on page 6, documents an expanded need. As a result, the CAA revised its mission, added interventions or changed referral patterns, and would expect different outcomes. These outcomes are characterized with the added dimension of time represented by short, intermediate and long term. Use of this logic model can demonstrate a CAA’s responsiveness to a changing environment and its effectiveness in handling change and obtaining results. This is “community action!”
The logic model, Adult Basic Education 2.0C, on page 8, is used to document outputs and outcomes. A CAA can use this adapted logic model to account for all the activity needed to obtain a positive outcome. In this logic model, the only reportable outcome is in Column 5 – Long Term, “30 of 100, or 30%, of clients pass A.B.E. class and demonstrate competence within 1½ years of enrollment.” Documentation of the activity in Columns 3, 4, and 5, allows the CAA to demonstrate the effort needed to “produce” a positive outcome, and can provide documentation for the fiscal and human resources necessary for the delivery of services and achievement of outcomes. In this manner, it can be demonstrated that activity and achievement of outcomes is similar to that experienced in the business world, page 34.

Logic Model

ROMA Logic Model 2.0A, 2.0B, 2.0C – Page 3

Instructions – Definition of Terms

Column 1 –Identified Problem, Need, Situation: The general statement of need that provides the rationale for the service, activity, or intervention. Planning
Column 2 –Service or Activity: The service or intervention provided in response to the problem, need, or situation. Always indicate the number of people or number of services offered and a timeframe. Intervention
Column 3 – Outcome: A positive benefit, behavior, or a change in condition, functioning, or problem accruing to individuals, families, and communities resulting from a service or activity. Intervention
Column 4 – Outcome/Indicator: Outcome/Indicators are numerical measures characterizing the results or impact of a program activity, service, or intervention and are used to measure performance. Indicators are typically represented with both the # and the %. Column 4 is used to project the expected outcome. Intervention
Column 5 – Actual Results: The same outcome/indicators used in Column 4 but updated to reflect actual results. Impact
Column 6 – Measurement Tool: The tool, form, or other medium where raw data is collected, e.g. survey instrument, attendance log, case record, pre and post test, waiting list, etc. Accountability
Column 7 –Data Source, Collection Procedures, Personnel: Place where data is maintained, e.g. central database, individual case records, specialized Access database, other, and location, e.g. on-site, subcontractor, other. Describes method(s) for retrieving data from data source(s), e.g. data from case records is retrieved manually, data is maintained in an automated database, and personnel assigned to the task. Accountability
Column 8 –Frequency of Data Collection and Reporting: How often data is required to be collected, how often data is reported. Accountability

Mission: Provide a mission statement for the logic model.

Proxy Outcome: Identify if a proxy outcome is used in the logic model.

Logic Model
1 / 2 / 3 / 4 / 5 / 6 / 7 / 8
N / S / O / OI / R / M / D / F
Mission:

Logic Model

Background

The logic model links programoperations, Columns 1-5 of the logic model (mission, need, intervention, projected results, actual results), and programaccountability, Columns 6-8 of the logic model (measurement tool, data source, and frequency of data collection and reporting, including personnel assigned to function). All eight components of the logic model are necessary for the successful delivery of services.

The ROMA Logic Model can be a “blueprint” for any of the CAAs programs or services.

The ROMA Logic Model can also be used in evaluation and monitoring and can provide some of the preliminary documentation for a CAA’s management information system.

The above page 2 contained in the Participant Manual is the script that you will need to deliver the Logic Model module.

Learning Objectives

Participants learn how to use the ROMA Logic Model.

Participants learn to distinguish between short, intermediate, and long-term outcomes.

Participants learn that managers could improve program management and decision-making by analyzing the information in a logic model.

Script

Introduce the participants to the ROMA Logic Model by stating that they are already familiar with it having completed 5 of the 8 columns in Day One. Refer them to the ROMA Logic Model from Day One that is posted on the wall.

Tape on the wall or across two easels the poster of the ROMA Logic Model 2.0A – Emergency Housing from the Day Two poster kit. This poster is the same as page 4 of the Participant Manual, ROMA Logic Model 2.0A – Emergency Housing.

Logic Model

Have the participants open to pages 3-4 of the Participant Manual and “walk” them through the sample logic model. Have the participants pull out page 3, Instructions – Definitions of Terms and place it alongside page 4, the ROMA Logic Model 2.0A – Emergency Housing.

Begin with a general introduction of the logic model, page 2 of the Participant Manual. Read the first two paragraphs and point to the poster on the wall/easel.

Using the Instructions – Definition of Terms on page 3 of the Participant Manual, (which is also in front of the participants). Review each column using the list below and go over the ROMA Logic Model 2.0A – Emergency Housing, page 4. As you review each column, review the actual example and numbers in the logic model.

Start with the statement of mission at the bottom of the form and read it to the participants.

Column 1 –Identified Problem, Need, Situation: The general statement of need that provides the rationale for the service, activity, or intervention. Planning

LM: Families are at risk of being evicted. Families are homeless.

Column 2 –Service or Activity: The service or intervention provided in response to the problem, need, or situation. Always indicate the number of people or number of services offered and a timeframe. Intervention

LM:200 families receive emergency housing assistance, January 1, 2004-June 30, 2004. One month emergency rent payment for 150 families. Emergency shelter for 50 families.

Logic Model

Column 3 – Outcome: A positive benefit, behavior, or a change in condition, functioning, or problem accruing to individuals, families, and communities resulting from a service or activity. Intervention

LM:Families remain in their own residence. Homeless families obtain emergency shelter.

Column 4 – Outcome/Indicator: Outcome/Indicators are numerical measures characterizing the results, or impact, of a program activity, service, or intervention and are used to measure performance. Indicators are typically represented with both the # and the %. Column 4 is used to project the expected outcome. Intervention

LM: 100 of 150, or 67%, of families remain in their own residence 90 days. 50 of 50, or 100%, of homeless families obtain emergency shelter lasting no longer than 30 days.

Column 5 – Actual Results: The same outcome/indicators used in Column 4 but updated to reflect actual results. Impact

LM: 90 of 150, or 60%, of families remain in their own residence 90 days. 50 of 50, or 100%, of homeless families obtain emergency shelter lasting no longer than 30 days.

Column 6 – Measurement Tool: The tool, form, or other medium where raw data is collected, e.g. survey instrument, attendance log, case record, pre and post test, waiting list, etc. Accountability

LM: Housing application, housing activity log, shelter log, client case record.

Column 7 –Data Source, Collection Procedures, Personnel: Place where data is maintained, e.g. central database, individual case records, specialized Access database, other, and location, e.g. on-site, subcontractor, other. Describes method(s) for retrieving data from data source(s), e.g. data from case records is retrieved manually, data is maintained in an automated database and personnel assigned to the task. Accountability

LM:Case Record. Data entered into automated case record at time of encounter. Data entered by CAA case-manager. Data entered by shelter case-manager.

Logic Model

Column 8 –Frequency of Data Collection and Reporting: How often data is required to be collected, how often data is reported. Accountability

LM:Data collected at time of encounter. Summary report generated to supervisor daily. Weekly report generated to department head each Monday. Monthly report generated for executive director. Daily electronic report emailed to CAA at daily close of business.

After you have reviewed the logic model, Emergency Housing 2.0A, on page 4, it is important to close with the analysis of the logic model. Read the text on page 2 along with the participants:

The logic model, Emergency Housing 2.0A, on page 4, documents a specific need, “families are at risk of being evicted or are homeless.” A review of the results in Column 5 indicates that the intervention, an emergency rent payment, produces a 100% positive outcome in the first 30 days. Follow-up data reveals a different story. After 60 days, families increasingly lose their housing. The same pattern exists after 90 days.

This CAA used the logic model to document the emergency housing program and to monitor performance. Collecting and analyzing follow-up outcome/performance data revealed the limits of the intervention and suggested that other factors may adversely be affecting families in the community.

Complete “construction” of the logic model at the end of this page. Add Column 6 – Measurement Tool (M), Column 7 – Data Source, Collection Procedure, Personnel Results (D), and Column 8 – Frequency of Data Collection and Reporting (F) labels. At the completion of columns 6-8, your logic model should look like this:

Logic Model

1 / 2 / 3 / 4 / 5 / 6 / 7 / 8
N / S / O / OI / R / M / D / F

Mission:

Remind participants that there is a blank logic model on page 5 which they can copy and use as long as the footer remains intact.

When you have completed the review of the Emergency Housing 2.0A Logic Model, repeat the same scenario with the Housing Assistance 2.0B, Logic Model on page 6.

Logic Model

Begin by reading the text on page two, the fifth paragraph along with the participants:

The logic model, Housing Assistance 2.0B, on page 6, documents an expanded need. As a result, the CAA revised its mission, added interventions or changed referral patterns, and would expect different outcomes. These outcomes are characterized with the added dimension of time represented by short, intermediate, and long term. Use of this logic model can demonstrate a CAA’s responsiveness to a changing environment and its effectiveness in handling change and obtaining results. This is “community action!”

Tell participants that this form of the logic model is similar but uses three dimensions of time characterized by short, intermediate, and long term outcomes. Indicate that the Emergency Housing 2.0A Logic Modelis now the short term component of the Housing Assistance 2.0B Logic Modeland that this logic model has been expanded to accommodate a changing need. A changing need could result in a changed mission, a change in the delivery of services, changes in projected and actual outcomes, and changes in the flow and documentation of data.

Remind participants that there is a blank logic model on page 7 which they can copy and use as long as the footer remains intact.

The final logic module of the presentation is Adult Basic Education 2.0C on page 8. This logic model is used to document outputs and outcomes. A CAA can use this adapted logic model to account for all the activity needed to obtain a positive outcome. In this logic model the only reportable outcome is in Column 5 – Long Term, “30 of 100, or 30%, of clients pass A.B.E. class and demonstrate competence within 1½ years of enrollment.” Documentation of the activity in Columns 3, 4, and 5, allows the CAA to demonstrate the effort needed to “produce” a positive outcome and can provide documentation for the fiscal and human resources necessary for the delivery of services and achievement of outcomes. In this manner, it can be demonstrated that activity and achievement of outcomes is similar to that experienced in the business world, page 38, Day One.

Walk participants through the logic model on page 8, Adult Basic Education 2.0C. The logic model has been modified to accommodate the incremental steps that are needed in order to achieve the long-term outcome: pass A.B.E. course and achieve competency in basic math, reading, and writing skills by receiving a TABE or GED.Short and intermediate outcome boxes have been modified to include outputs such as placement, attendance, and completion of the course. It is helpful for a CAA to document these incremental activities for internal management purposes since they consume resources, staff time, and

Logic Model

occur over an extended time period. However, only the long term outcome should be reported. It is important to note that from screening to achievement of the outcome there is a 70% dropout in the 1½ year’s time period. If the CAA consistently tracks this experience they may be able to establish the 30% success rate as a target, benchmark, or standard.