Evaluation Plan 1

The Online Academy (TOA)

Evaluation Plan

TOA Project Team:

John Baek

Michelle Didier

Dan Feinberg

Lisa Herbert

Danalyn Robinson

Heather White

February 20, 2004

Table of Contents

Need for Evaluation

Purpose/Goals

Evaluation Strategy

Objectives

Social and Cultural Factors

Desired Results

Evaluation Levels with Formality Levels

Data Collection Methods

Data Analysis

Evaluation Plan Details (Logistics)

Table of Illustrations

Table 1. TOA Evaluation Strategy Desired Results

Table 2. Summary of Desired Results, Learning Activities, Evaluation Methods, Formality Level, and Data Collection Methods

Table 3. Evaluation Plan Details (Logistics)

Table 4. Possible Questions and Tasks for Evaluations

Evaluation Plan 1

The Online Academy (TOA) EVALUATION PLAN

Through the evaluation plan of The Online Academy (TOA) team, we intend to examine the effectiveness of the materials and activities for TOA and of the effectiveness of the transfer of the materials from classroom to online. The TOA Evaluation Plan contains the following information:

  • Need for evaluation
  • Evaluation strategy
  • Evaluation purpose and goals
  • Evaluation strategy objectives
  • Relevant social and cultural factors
  • Desired results
  • Kirkpatrick’s levels incorporated
  • Formality levels of evaluation methods
  • Evaluation methods and related learning activities
  • Data collection methodology

Need for Evaluation

The need for the TOA Evaluation Plan is to:

  1. Inform and direct current and future development
  1. Examine the usability of the checklist (Expert)
  2. Organization and self-regulation
  1. Review the course modules for the following:
  2. Correlation to the Virginia Standards of Learning (SOL)
  3. Clarity of instruction
  4. Engagement and motivation of instruction (Distance learner, one-on-one-teacher)
  5. Content presented at the appropriate level
  6. Adequacy of the resources, skills, and content-adequate for challenge
  7. Pedagogical approach and format
  8. Pedagogical re-engineering
  1. Determine effectiveness of navigation including: (one-on-one, distance learner)
  2. Locating resources
  3. Intuitiveness—do the learners know where to go next

Purpose/Goals

The purpose of the TOA evaluation plan is based on the following goals:

  1. Analyze instructional strengths (pedagogical re-engineering)
  2. Engagement / Motivation
  3. Target audience perceived use (Do you like it?)
  4. Multiple learning modalities and learning styles
  5. Does it adhere to expert guidelines?
  6. Relevant to learners
  7. Will the student know the references (cultural as well as prior knowledge) made in the content?
  1. Determine the effectiveness of the design and transfer of the materials from classroom to online. Does the current design take advantage of online learning affordances?
  2. Individualized instruction and guidance
  3. Asynchronous – reflection
  4. Multimedia capabilities
  5. Self-paced
  6. Learner control
  7. Type of student and their interaction with the module
  8. Does not mimic “classroom” materials
  9. Evaluate the usability and the benefits of the checklist to the learner
  10. Self-regulation
  11. Organization
  12. Understanding assignments
  13. Scheduling of time
  14. Locate ease of use problems
  15. Clarity
  16. Self-navigation
  17. Can learners move around on their own, without assistance?
  18. Use of directions
  19. Do learners understand the directions?

Evaluation Strategy

The purpose of the TOA evaluation strategy is to determine the effectiveness of the initial modules of the seven courses being developed for TOA. Using expert and one-to-one reviews, we will investigate:

  • The strengths of the instruction and pedagogically re-engineered instructional materials
  • How users utilize and work with the checklist
  • How users navigate through and find items in TOA.

The results of this evaluation will be used to inform and direct the current and future design and development of TOA.

Objectives

The objectives for conducting the TOA Evaluation Plan are as follows:

  1. Expert review
  2. Find a way to determine whether the site adheres to Clark and Mayer criteria and Gardner criteria.
  3. Recruit experts.
  4. Prepare questions.
  5. Design data collection tools.
  6. Set up testing area.
  7. Conduct evaluation.
  8. Analyze results.
  9. Write evaluation.
  10. One-to-One review
  11. Recruit students.
  12. Prepare evaluation questions.
  13. Design data collection tools.
  14. Write script.
  15. Set up testing area.
  16. Conduct evaluation.
  17. Analyze results.
  18. Write evaluation.

Social and Cultural Factors

The TOA team has identified the following social and cultural factors related to the evaluation:

  • Priscilla Norton, PhD, TOA Project Director

Dr. Norton is interested in the:

  • TOA project meeting its deadline and coming within budget
  • Courses adhering to the COPLS model
  • Policy board

The Policy Board is interested in student learning performance

  • School districts

The school districts are interested in the:

  • TOA project meeting its deadline and staying within budget
  • Computer constraints of the school districts.
  • Students’ restricted access to TOA in schools
  • Subject Matter Experts (SMEs)

The SMEs are concerned about how their technical learning curve will affect the design of the content.

  • Students

The students are interested in how they will use TOA; they will typically use TOA at home or at a library.

Desired Results

Based on the evaluation strategy of the TOA Evaluation Plan, we have identified desired results that correlate with our evaluation strategy goals, as well as the intended participants and their needs. Table 1 presents the desired results for the TOA evaluation strategy.

Table 1. TOA Evaluation Strategy Desired Results

Instruction / Participant and Need / Desired Results
Engagement / Motivation / Students.
Need to be engaged and motivated to complete assignment.
Activities need to be engaging and motivating. / Increased interest and engagement in learning.
Relevant to Learners / Students.
Will the student know the references (cultural, as well as prior knowledge) made in the content? / Students will recognize the references in relation to their world
Ease of Use
  • Clarity
  • Use of directions
/ Students.
Need to navigate to activities, understand what they need to do, and perform activities without assistance. / Students will be able to navigate to activities, understand what they need to do, and perform activities without assistance.
Online Learning Affordances
  • Individualized instruction and guidance
  • Asynchronous – reflection
  • Multimedia capabilities
  • Self-paced
  • Learner control
  • Type of student and their interaction with the module
  • Does not mimic “classroom” materials
/ Online Instructional Design Expert.
Need to determine if the instruction satisfies good design principles and best practices. / Effective use of online learning environment to support student learning
Checklist /
  1. Self-regulation expert.
Does the checklist support the student ability to self-regulate?
  1. Students.
Do students understand assignments and deadlines? / Checklist design enables students to manage assignments.
Multiple learning modalities and learning styles / Instructional Design Expert.
Accommodates different learning styles using effective modes of presentation. / Courses use appropriate instructional strategies to accommodate varying learning modalities and styles.

Evaluation Levels with Formality Levels

We will conduct a multi-level evaluation to explore as many perspectives as possible. To achieve our desired results, we will conduct evaluations at Levels 1, 2, and 3 of Kirkpatrick’s Levels of Evaluation. The level of formality of the evaluations will vary from informal to formal, depending on the type of evaluation.

Table 2 maps the desired results to the learning activities, evaluation levels, formality levels, and data collection methods.

We have identified learning activities that are congruent with what students will be doing when they take a module or course in TOA. For example, since students will be completing a module online in a setting where their mentor is usually not available for immediate feedback (desired result 3 in Table 2), we have a learning activity to complete part of a module.

Data Collection Methods

To collect our data, we will use observations during the one-to-one evaluations with the students and through interviews during the expert reviews. We will develop and follow a protocol during the one-to-one evaluations and expert reviews. During both the one-to-one evaluations and expert reviews, we will follow a script with a list of prepared questions, allowing for the introduction of additional questions or probing questions when needed. The one-to-one evaluations will be recorded on either audiotape or videotape. We will document our observations and interviews in field notes.

See table 2 for a summary of our data collection methods.

Data Analysis

After we collect our data from the one-to-one evaluations and expert reviews, we will analyze our data by reviewing our field notes and transcripts to look for common themes, patterns, and trends.

Evaluation Plan Details (Logistics)

To complete our evaluations, we have identified whom we might use as evaluators and expert reviewers, who will conduct the different evaluations and reviews, when and where we might conduct the evaluations, and how the evaluations will be conducted. We have also developed an initial set of questions to ask for each type of evaluation. We will develop a protocol and script for each type of evaluation.

Table 3 presents the high level details of how we plan to complete our evaluations. Table 4 presents our initial set of questions for each type of evaluation.

Evaluation Plan 1

Table 2. Summary of Desired Results, Learning Activities, Evaluation Methods, Formality Level, and Data Collection Methods

Desired Results / Learning Activities / Evaluation Levels/
Evaluation Method / Level of Formality / Data Collection Methods
1 / Increased interest and engagement in learning. / Complete two lessons from a module. / Level 1
Document observations Interview student / Formal / Field notes
Interview
2 / Students will recognize the references in relation to their world / Read challenge, lessons, and activities. / Level 1
Document observations Interview student / Formal / Field notes
Interview
3 / Students will be able to navigate to activities, understand what they need to do, and perform activities without assistance. / Complete defined navigational path.
Eg. Find challenge, find resource pages, or find activity x in lesson x. / Level 1
Document observations Interview student / Formal / Field notes
Task completion checklist
4 / Effective use of online learning environment to support student learning / Review two modules and provide feedback. / Level 2
Interview expert / Semiformal / Field notes
Interview
5 / Checklist design enables students to manage assignments. / Review checklist and provide feedback. / Level 3
Interview expert / Semiformal / Field notes
Interview
6 / Courses use appropriate instructional strategies to accommodate varying learning modalities and styles. / Review two modules and provide feedback.
Eg. Does it meet standards? / Level 2
Interview expert / Semiformal / Field notes
Interview
7 / Improve learning environment based on mentor’s/SME’s comments / Iterative design feedback / Level 1:
Feedback from SMEs / Informal / Field notes

Table 3. Evaluation Plan Details (Logistics)

Evaluation Planning Factors / One-to-One Evaluation / Expert Review
Who will be the evaluators? / Two evaluators
Tentative possibilities are:
  1. Monisha’s nephew
  2. Homebound friend of Monisha’s nephew—10th grader
  3. High school students from private Edmond Burke high school where John’s friend works
  4. Home schooled student of high school age from LoudounCounty (LIFE Home School Support Group)
Student for World History —10th or 11th grade
Student for Chemistry —10th grade
Students should be articulate and moderate ability level. / Tentatively two expert reviewers:
  1. Dr. Nada Dabbagh
  2. Dr. Anastasia Kitsantas,

Who will conduct the evaluations? / Evaluator / Researcher / Evaluator / Researcher
Student 1 / Michelle/Lisa / Dr. Dabbagh / Heather
Student 2 / Dan / Dr. Kitsantas / Danalyn/Lisa
All will help with transcription / All will help with transcription
Where will the evaluations be conducted? /
  1. Monisha’s nephew at home, public library, or GMU
  2. Homebound friend of Monisha’s nephew at home, public library, or GMU
  3. High school students from Edmond Burke at their school
  4. Home schooled student at their home or public library
/ GMU
When will the evaluations be conducted and for how long? / They will be conducted in the month of March and April and each session will last approximately 60-90 minutes. / They will be conducted in the month of March and April and each session will last approximately 60-90 minutes.
What are the details of the data collection? / How Conducted
Observation of student with script and structured questions at defined points in the lesson during the session. A computer is needed for the student to complete the activities.
Strategy
For each session, one student will look at one different module.
Example: One student will look at the Chemistry module
One student will look at the World History module
Modules
  • World History—Silk Road
  • Chemistry
Data Collection Tools
  • Audio recording of interview
  • Field notes
  • Checklist of completing tasks, for example, finding an activity or using the checklist or navigating through the site.
Script Elements
  • Welcome
  • Background about TOA
  • Mention anonymous use of results
  • Use of results
  • Explain purpose of interview
  • Explain what students will be doing
  • Read from script. Have students complete activities or navigate through course/module.
  • Final comments
  • Thank you
/ How Conducted
Interview with structured questions and probing follow-up questions.
Modules
  • Checklist—show different designs and discuss them
  • Show entire module or selected lessons
  • Same as one-to-one modules—show most or all of module
  • World History 1 module
Data Collection Tools
  • Audio recording of interview
  • Field notes
Script Elements
  • Welcome
  • Explain purpose of review
  • Show evaluator checklist.
  • Reading from script, ask questions.
  • Show evaluator module(s).
  • Reading from script, ask questions.
  • Final comments
  • Thank you

How will the data be analyzed? / Transcribe results
Look for common themes and patterns / Transcribe results
Look for common themes and patterns

Evaluation Plan 1

Table 4. Possible Questions and Tasks for Evaluations

One-to-One / Expert
Need to complete specific tasks / Does the module include multiple modalities?
Think aloud as user completes / Does the module accommodate different learning styles?
What is your overall impression? / Does the module take advantage of the affordances of online learning?
What would you change if you could? / Review checklist
Is it relevant and understood?
Do the directions make sense?
Is it easy to get around?
How would you start the assignment?
How would you organize yourself to work in this module?
Does the organization of the module make sense?