Assignment A5a
Formative Evaluation Plan
Group D: Vis-à-vis
Alonzo Delaine
Jane Dougan
Jeff Getchell
ITDE 8001: CRN 33289
Introduction to Instructional Design
Nova Southeastern University
April 2, 2006
Formative Evaluation Plan
Background
Grant funding can provide organizations with a significant opportunity to create new programs or to enhance existing programs with the infusion of monetary resources. However, the grant application process can be a difficult one to navigate, particularly for individuals with little to no grant application experience. The purpose of this training module is to provide instruction in the essentials of completing a a grant application. After completing this training module, learners with education or experience equivalent to a college or university undergraduate degree but with limited or no experience in completing grant applications will be able to complete a grant application.
The lesson will be primarily paper-based and delivered at the learner’s home or place of employment. Learners will use word processing and spreadsheet application software as well as the Internet to supplement the paper-based lesson materials. The entire lesson module should take no longer than three hours to complete. The learner will also have supplemental reference materials to take with them after completion of training which can be used when completing future grant applications.
Purpose
The purpose of the formative evaluation for this learning module is to determine whether the module materials meet the needs of the lesson and whether the materials are effective in their application. Feedback received from this formative evaluation will be used to further revise the instructional materials for this module.
Expert Review (Subject Matter Expert)
Subject: The Expert Reviewer is currently the grants coordinator for a medium-sized community college. This person has six years of experience as a grant writer and grant administrator and has authored or co-authored grants worth several million dollars. The Expert Reviewer currently assists community college faculty and staff in both identifying grant funding opportunities as well as the grant application processes for public and private grant opportunities.
Instruments: The survey instrument for the Expert Review consists of four questions designed to ascertain the efficacy of the goals and objectives and the lesson content. The questions are Yes/No and includes a comment line.
Procedure: The Expert Review formative evaluation will take place in the office of the Subject Matter Expert (SME). The interviewer will begin the meeting by explaining that this is a formative evaluation designed to gain feedback on the efficacy of the lesson materials and the appropriateness of the goals and objectives. The SME will then be given a two-page handout. The first page of the handout will describe the goals and objectives of the lesson. The second page will have four questions that ask for the opinion of the SME on the appropriateness of the material. When the first handout is completed, the SME will be given a second handout. This handout will contain the lesson content materials as well as an evaluation comment sheet. The SME will first be instructed to read the lesson materials. When finished reading, the SME will then be instructed to complete a survey form that consists of four questions dealing with the appropriateness and effectiveness of the lesson content. Each question is in Yes/No format and each question will include a comment line. When the survey form is complete the Interviewer will ask the SME if there are any other comments regarding the lesson materials or assessments that the SME would like to add. All comments will be recorded for later review.
Learner review (one male and one female)
Subject: Both the female and male Learner Reviewers are learners meeting the criteria of the target population for this training module. Each learner has a four year college degree and will have affirmed that they may be asked to write a grant proposal sometime in the future. Both the male and female learners have computer experience and both feel comfortable using word processing and spreadsheet software and are comfortable in the use of the Internet and Web browsers.
Instruments: The survey instrument for the learner review consists of four questions designed to ascertain the efficacy of the goals and objectives and the lesson content. All four questions are a Yes/No format and each question includes a comment line.
Procedure: The formative evaluations of the male and female Learner Reviewer will take place in the Learner’s offices on separate days. All other aspects of the interview process will be identical. To begin the interview, the interviewer will explain to the Learner that this is a formative evaluation designed to gain feedback on the efficacy and appropriateness of both the lesson materials and the assessment instruments. Each Learner will then be presented with the lesson content and assessment instruments. The Learner will be instructed to read the lesson materials and complete the assessments to the best of their abilities. When they are finished reading the materials, the Learner will be presented with a survey form and instructed to answer the four questions dealing with the appropriateness and effectiveness of the lesson content and the effectiveness of the assessment methods. All questions are in a Yes/No format and each question includes a comment line. When the survey form is complete the Interviewer will ask the Learner if there are any other comments regarding the lesson materials or assessments that the Learner would like to add. All comments will be recorded for later review.
Learner review (group of three learners)
Subject: The group of three Learner Reviewers are learners meeting the criteria of the target population for this training module. Each learner has a four year college degree and has affirmed that they may be asked to write a grant proposal sometime in the future. All learners in this group have computer experience and all feel comfortable using word processing and spreadsheet software, as well as being comfortable in the use of the Internet and Web browsers.
Instruments: The survey instrument for the learner review consists of four questions designed to ascertain the efficacy of the goals and objectives and the lesson content. All four questions are in a Yes/No format and each question includes a comment line.
Procedure: The formative evaluations for the Learner Reviewers group will take place in a computer lab located on the main campus library. To begin the interview, the interviewer will explain to the group Learner Reviewers that this is a formative evaluation designed to gain feedback on the efficacy and appropriateness of both the lesson materials and the assessment instruments. Each Learner Reviewer will then be presented with the lesson content and assessment instruments. The Learner Reviewers will be instructed to read the lesson materials and complete the assessments to the best of their abilities. When finished reading the materials, the Learner Reviewers will then be presented with a survey form and instructed to answer the four questions dealing with the appropriateness and effectiveness of the lesson content and the effectiveness of the assessment methods. All questions are in a Yes/No format and each question includes a comment line. When the survey form is complete the Interviewer will ask the Learner Reviewer group if there are any other comments regarding the lesson materials or assessments that the Learner Reviewer group would like to add. All comments will be recorded for later review.
Appendix A: Overall Formative Evaluation Plan
Expert reviews / Subject matter expert / April 10, 2006 / · Goals and objectives
· Lesson content / · Survey
· Followup interview
One-to-one evaluations / Learner
(male) / April 11, 2006 / · Lesson content
· Assessments / · Survey
· Followup interview
One-to-one evaluations / Learner
(female) / April 12, 2006 / · Lesson content
· Assessments / · Survey
· Followup interview
Group evaluation / Group of three learners / April 13, 2006 / · Lesson content
· Assessments / · Survey
· Followup interview
Appendix B: Specific Formative Evaluation Protocols
Lesson: Five Easy Steps to Completing a Grant Application
Facilitator: Getchell
Evaluator: Subject Matter Expert (Del Mar College Grants Administrator)
Date: April 10, 2006
Targets
· Goals and objectives
· Lesson content
Technique
· Survey
· Interview
Purpose
· To verify the validity of the goals and objectives of the lesson and the associated lesson content.
Written Questions
· Do you believe the goal statement represents an accurate outcome for this lesson?
o Yes
o No
o Comment (optional)
· Do you believe the learning objectives are achievable for the audience described in the preceding goal statement?
o Yes
o No
o Comment (optional)
· Does the course content effectively cover the learning objectives?
o Yes
o No
o Comment (optional)
· Do the learning resources provided with the lesson add value to the lesson?
o Yes
o No
o Comment (optional)
Oral Interview
· Do you have any additional comments regarding any of the questions you have just answered?
Procedure
· Discuss the overall purpose of the lesson and the expected outcomes with the SME. Describe the intended audience and the resources to be made available to the learners.
· Discuss the purpose of the formative evaluation process and the uses of the feedback that may be provided by the SME.
· Present the goals and objectives to the SME; have the SME complete the survey form and add any narrative comments in the appropriate places.
· Review the survey results and narrative comments and discuss any results that may not be clear.
· Present the lesson materials to the SME; have the SME complete the survey form and add any narrative comments in the appropriate places.
· Review the survey results and narrative comments and discuss any results that may not be clear.
· Ask the SME if there may be additional comments about any aspects of either the goals and objectives or the course content
· Thank the SME and determine if there is interest in additional evaluations in the future.
Materials
· Goals and objectives statement
· Course materials
· Supplemental handouts
· Survey forms
· Interview questions
Lesson: Five Easy Steps to Completing a Grant Application
Facilitators: Getchell
Evaluator: Learner (male)
Date: April 11, 2006
Targets
· Lesson content
· Assessments
Technique
· Survey
· Interview
Purpose
· To verify the accuracy and validity of the lesson content and associated assessment instruments.
Written Questions
· Does the course content effectively cover the learning objectives?
o Yes
o No
o Comment (optional)
· Do the learning resources provided with the lesson add value to the lesson?
o Yes
o No
o Comment (optional)
· Do you believe the assessments provide an accurate gauge of the expected learning outcomes?
o Yes
o No
o Comment (optional)
· Do you believe the lesson content and associated materials will allow the learner to achieve the desired learning outcome?
o Yes
o No
o Comment (optional)
Oral Interview
· Do you have any additional comments regarding any of the questions you have just answered?
Procedure
· Discuss the overall purpose of the lesson and the expected outcomes with the Learner. Describe the intended audience and the resources to be made available to the Learner.
· Discuss the purpose of the formative evaluation process and the uses of the feedback that may be provided by the Learner.
· Present the course content to the Learner; have the Learner complete the survey form and add any narrative comments in the appropriate places.
· Review the survey results and narrative comments.
· Ask if there are any additional comments the Learner would like to add.
· Present the assessment materials to the Learner; have the Learner complete the survey form and add any narrative comments in the appropriate places.
· Review the survey results and narrative comments.
· Ask the Learner if there may be additional comments about any aspects of either the course content or the assessment materials.
· Thank the Learner and determine if there is interest in additional evaluations in the future.
Materials
· Course materials
· Supplemental handouts
· Assessment materials
· Survey forms
Lesson: Five Easy Steps to Completing a Grant Application
Facilitators: Getchell
Evaluator: Learner (female)
Date: April 12, 2006
Targets
· Lesson content
· Assessments
Technique
· Survey
· Interview
Purpose
· To verify the accuracy and validity of the lesson content and associated assessment instruments.
Written Questions
· Does the course content effectively cover the learning objectives?
o Yes
o No
o Comment (optional)
· Do the learning resources provided with the lesson add value to the lesson?
o Yes
o No
o Comment (optional)
· Do you believe the assessments provide an accurate gauge of the expected learning outcomes?
o Yes
o No
o Comment (optional)
· Do you believe the lesson content and associated materials will allow the learner to achieve the desired learning outcome?
o Yes
o No
o Comment (optional)
Oral Interview
· Do you have any additional comments regarding any of the questions you have just answered?
Procedure
· Discuss the overall purpose of the lesson and the expected outcomes with the Learner. Describe the intended audience and the resources to be made available to the Learner.
· Discuss the purpose of the formative evaluation process and the uses of the feedback that may be provided by the Learner.
· Present the course content to the Learner; have the Learner complete the survey form and add any narrative comments in the appropriate places.
· Review the survey results and narrative comments.
· Ask if there are any additional comments the Learner would like to add.
· Present the assessment materials to the Learner; have the Learner complete the survey form and add any narrative comments in the appropriate places.
· Review the survey results and narrative comments.