The Heller School for Social Policy and Management, Brandeis University
Master of Science in Global Health Policy and Management
HS278f-3 Monitoring and Evaluation
Spring 2016 (Module 1)
Friday, 9:00 – 12:00
Room TBD
Instructors: Candace Miller
E-mail:
Office Hours: Friday 12-1 pm and by appointment
Welcome to the spring 2017 Monitoring and Evaluation course!
University Notices
1. If you are a student with a documented disability on record at Brandeis University and wish to have a reasonable accommodation made for you in this class, please see me immediately.
2. You are expected to be honest in all of your academic work. The University policy on academic honesty is distributed annually as section 5 of the Rights and Responsibilities Handbook. Instances of alleged dishonesty are subject to possible judicial action. Potential sanctions include failure in the course and suspension from the University. If you have any questions about my expectations, please ask.
Academic integrity is central to the mission of educational excellence at Brandeis University. Each student is expected to turn in work completed independently, except when assignments specifically authorize collaborative effort. It is not acceptable to use the words or ideas of another person – be it a world-class philosopher or your roommate – without proper acknowledgement of that source. This means that you must use footnotes and quotation marks to indicate the source of any phrases, sentences, paragraphs or ideas found in published volumes, on the internet, or created by another student. If you are in doubt about the instructions for any assignment in this course, you must ask for clarification.
Course Description
Monitoring and evaluation activities include a wide range of formal and informal data collection and analysis techniques developed to help determine whether a project, program, or policy is working and why. The processes of ongoing monitoring and evaluation are critical to program planning and improving the impacts and outcomes of a given program. Also, monitoring and evaluation processes are an essential tool for ensuring accountability across public programs.
This course introduces students to the field of evaluation, including their purpose, design, methodologies for data collection and analysis, and utilization. We will also explore the context in which evaluations are carried out, frequent challenges and pitfalls in conducting evaluation. The instructor will share materials and experiences across a range of sectors and real world evaluations. Students should leave the course informed consumers of evaluations, capable of thinking through evaluation design and contracting qualified specialists for the more technical aspects of evaluation, and familiar with how to tap into the wealth of additional resources (many web-based).
Core Competencies
Conceptual: On a conceptual level, students will understand
· The range and purposes of project and program monitoring and evaluation
· The evaluation process and the links between planning, implementation and evaluation and how all are affected by implicit or explicit theories of change
· Types of evaluation
· Evaluation designs and methods
· Important considerations, such as cost, validity, value of information, and participation among stakeholders
Skills
· Identifying a program’s theory of change
· Developing a program logic model
· Informed by the theory of change and logic model, determine appropriate study design (such as a process, outcome, or impact evaluation) and methodologies.
· Methods for mainstreaming gender into evaluations and/or doing gender-focused evaluation
· Planning for data collection methods
· Design a simple evaluation, based on a project of your choosing or one from among several options provided in class.
Course Grading:
Evaluation Portfolio 50%. The in- and out-of-class exercises that accompany each lecture will combine to create an evaluation portfolio at the end of semester that should be housed online on student’s website, such as WordPress or another site.
Throughout the course, students may work individually or AS A GROUP. If working alone, you will submit materials individually for comment and suggested revisions. If you work as a group, you will submit pieces as a group for comments and suggested revisions. The products (ONE PER GROUP) must be submitted by Thursday 9 am on the day before the course meets (we meet on Fridays). There will be no grades on weekly submissions, however students will be notified of the grade that the portfolio will receive so revisions can be made.
The final, end-of-course portfolio will be submitted as INDIVIDUAL EVALUATION PORTFOLIOS and assignments from each week (that have been revised and improved over the semester.) Each product in the evaluation portfolio will be approximately 1 to 2 pages. Most groups will have the same products except have an individual:
1. Introductory cover sheet
2. CV
Class participation 20%. Class attendance is required and students are encouraged to contribute to class discussion. Participation is the key to a lively class. 15% of the course grade will depend upon contributions to our class sessions. Class participation provides the opportunity to practice speaking and persuasive skills, as well as the ability to listen. Comments that are vague, repetitive, unrelated to the current topic, disrespectful of others, or without sufficient foundation will be evaluated negatively. What matters is the quality of one's contributions to the class discussion, not the number of times one speaks. Also, your classmates are relying upon your attendance for group work. If a student misses a lecture, he/she is expected to use class notes and readings, and work with group members and the TA to cover missed material.
Weekly reading reflections 20%. Students will write weekly reflections on the assigned readings prior to class. These are due electronically by Thursday at 6 pm. The expectation is that students will have read all materials assigned for the week prior to the class and will relate the reading to their project. The weekly reflection will be an opportunity for students to demonstrate their mastery of the material, or, to articulate why they are unable to apply reading topics to their project. Reflections should be posted on Latte.
Final presentations and portfolio 50%. The evaluation portfolio will be presented by each group. They should be structured as if being presented to relevant ‘stakeholders’.
Student grade 10%. In the professional world, future projects, job recommendations and other opportunities come to those who work well with others, offer fresh ideas, and get things done. We will ask students to provide constructive references for their classmates.
Course textbook: As a basic reference book, I recommend:
Handbook of Practical Program Evaluation (Essential Texts for Nonprofit and Public Leadership and Management) 4th Edition by Kathryn E. Newcomer (Author), Harry P. Hatry (Author), Joseph S. Wholey (Author) 2015. This text is available at Amazon for rent, on a kindle, or to purchase a used or new copy. The price ranges from $26 to $73 depending on how you obtain it.
There are many M&E textbooks available. If you have access to another textbook that you prefer, please let see the book or send the URL and I will review it to ensure that it contains the essential aspects of the course. Online handbooks include the UNDP’s handbook that provides step-by-step guidance (http://web.undp.org/evaluation/handbook), while the European Commission has prepared one for humanitarian evaluation http://ec.europa.eu/echo/files/evaluation/2007/humanitarian_guide.pdf)
Additional resources you may want to reference are:
UNICEF evaluation database www.unicef.org/evaldatabase
World Bank www.worldbank.org (click onto learning on upper right hand side)
Institute Development Studies www.ids.ac.uk/info/index.html
Active Learning Network for Accountability and Performance www.alnap.org
International Household Survey Network www.surveynetwork.org
Handbook on Planning for M&E http://web.undp.org/evaluation/handbook/
American Evaluation Association http://www.eval.org/
International Development Evaluation Association http://ideas-global.org/
International Initiative on Impact Evaluation http://www.3ieimpact.org/
Poverty Action Lab at MIT https://www.povertyactionlab.org/
Mathematica Policy Research http://www.mathematica-mpr.com
Network of Networks on Impact Evaluation (NONIE)
UN Evaluation http://www.uneval.org
World Bank Independent Evaluation Group
MIT Development Economics Events
CDC Webinars on Evaluation
American Evaluation Association Career Center http://www.eval.org/programs/careercenter.asp
Class Schedule
YOU MUST COMPLETE ALL READING PRIOR TO CLASS.
Week 1 (1/20) – Overview of Monitoring and Evaluation for Global Health and Development
This class will focus on introducing and discussing a framework for monitoring and evaluation; how monitoring and evaluation fits into the planning cycle; the difference between monitoring and evaluation; different approaches to M&E; and the importance of a theory of change.
Before class please read:
· “Leave your laptops…” http://www.nytimes.com/2017/01/02/opinion/leave-your-laptops-at-the-door-to-my-classroom.html?smprod=nytcore-iphone&smid=nytcore-iphone-share&_r=0
· “Understanding Types of Evidence: A Guide for Educators.” October 24, 2016. Gregory Chojnacki, Alexandra Resch, Alma Vigil, Ignacio Martinez. Available on Latte
· Choose at least one blog post from the Evidence in Action blog (on the right of the page): https://cire.mathematica-mpr.com/
o “The mandate of effective government.” https://cire.mathematica-mpr.com/commentary/the-mandate-of-an-effective-government
o “Could there be too much evaluation?” https://cire.mathematica-mpr.com/commentary/could-there-be-too-much-evaluation (or others)
· Newcomer et al. Chapter 1 and 2
Class presentation:
· Evaluation of the Malaria Elimination Initiative (Global)
Assignment due 1/26:
In groups or individually, identify a program or policy that you will use for class assignments. You will create a monitoring and evaluation plan for the program. It may be real or a proposed program that you would like to see be implemented. Write a 2 page paper describing the program. Include details on the who funds and implements the program; the program beneficiaries (age, gender, socio-economic status etc.); what the program is intended to do (what outcomes); identify the program timeline; and describe the evidence needed to demonstrate or prove that the program works or can work.
Week 2 (1/27) From program to evaluation: theoretical frameworks, logic models, and evaluation questions
This class will describe what questions to ask about the program in order to understand the theory of change (if the program has one) and how to develop a logic model from the theory of change that will guide program M&E. Students will also learn how to critique a logic model and determine whether revisions are needed prior to designing an evaluation.
Before class please read:
· Newcomer, Chapters 3 and 4
· Grant Craft, “Theory of Change: Using a Theory of Change to Guide Planning and Evaluation,” 2006
· OECD Glossary of terms, 2010 (esp. DAC criteria of relevance, effectiveness, efficiency, sustainability, and impact; and terminology related to evaluation design – baseline, benchmark, ex-ante vs. ex-post evaluation, etc)
Class presentation:
· Early childhood education and development Fursa kwa Watoto (Opportunities for Children) Tanzania
Assignment due 2/2:
Students will develop a logic model for the program or policy for their project.
Week 3 (2/3) – Evaluation study designs and economic evaluation
This class will focus on study designs and approaches as well as the importance of matching the evaluation approach to program design and needs. We will identify different forms of experimental, quasi-experimental, and non-experimental designs, as well as discuss the importance of economic evaluations.
Before class please read:
· Newcomer, chapters 6-8, 24
· World Bank, chapter 12. http://www.who.int/water_sanitation_health/publications/2012/ch12.pdf
Class presentation:
· Evaluation of the Millennium Challenge Corporation Projects (Infrastructure that improves public health and wellbeing) Moldova, Namibia, and Tanzania
Assignment due 2/9:
Students will develop a study design for their project.
Week 4 (2/10) Program monitoring, mixed method evaluations and the benefits of multi-method approaches
This class will focus on program monitoring to understand how well the program is being implemented, what course corrections are needed, and how to improve program performance. Additionally, the class will also focus on using mixed data collection approaches to obtain all the information needed about program performance, outcomes, impacts, and costs.
Before class please read:
· Newcomer, chapter 5 and 13
· ACAPS, “Qualitative and Quantitative Techniques for Humanitarian Needs Assessment,” May 2012
· Jessica Shortall, “Cost-effective Household Surveys: Key Lessons for Implementing a Household Livelihood Survey on a Budget,” USAID, 2009.
Class presentation:
· Evaluation of poverty reduction programs in Liberia, Palestine, and Malawi
Assignment due 2/16:
Students will continue working on their study designs for their project, ensuring the evaluation has mixed method approaches.
Week 5 (2/17) Quantitative instrument development and qualitative methods including in depth interviews and focus group discussions
Before class, please read:
· Newcomer, chapters 19-22
· Ellen Taylor-Powell, “Questionnaire Design: Asking questions with purpose.” University of Wisconsin. Found at http://s142412519.onlinehome.us/uw/pdfs/G3658_2.PDF and on LATTE
Optional reading
· Interviews, “Closing the Loop – Responding to People’s Information Needs from Crisis to Recovery to Development: A case Study of Post-Earthquake Haiti,” January 2010. Read especially the methodology sections.
· Andrew Fisher and James Foreit, “Chapter 8 Sampling,” Designing HIV/AIDS Intervention Studies: An Operations Research Handbook, 2002.
Class presentation:
Students will conduct in class focus group discussions.
Assignment due 2/23:
Students will also develop an instrument for their project.
Week 6 (2/24) – From a terms of reference (TOR) to results
This week we will review the process from an organization identifying the need for an evaluation and developing a TOR, to evaluation design, the financing and logistics of implementing an evaluation, the need for periodic feedback and check in, to data analysis and validation, to disseminating results.
Before class, please read:
· Newcomer, chapters 27, 28, 29
· Bamberger, et al. excerpt from chapter “Pulling it all together”, pp. 395-402 on LATTE
Class presentation:
· The Karnataka Cash Transfer and families affected by HIV, India or
· Women for Women International
· Armenia’s Integrated Social Support Services
After class:
Students will develop a communication plan for their project.
Week 7 (3/3) Student presentations and feedback; vote on which projects to fund
Before class, please read:
· Newcomer, chapters 26
Class presentation:
During the final class session, students will present their full portfolio of work.
1