Deliverables Report
IST-2001-33310 VICTEC
Evaluation of Demonstrator in Schools
AUTHORS: Carsten Zoll, Sibylle Enz, Sarah Woods, Lynne Hall, Kerstin Dautenhahn,
STATUS: DRAFT
CHECKERS:
PROJECT MANAGER
Name: Ruth Aylett
Address: CVE, Business House, University of Salford, University Road,, Salford, M5 4WT
Phone Number: +44 161 295 2922 Fax Number: +44 161 295 2925
E-mail:
TABLE OF CONTENTS
1 Purpose of Document……………………………………………………………..3
2 Executive Overview…………………………………………………………………4
3 INTRODUCTION…………………………………………………………………………..5
4 The VICTEC DEMONSTRATOR “FEARNOT” 6
5 Evaluation OF FEARNOT IN SCHOOLS 8
6 Conclusions 16
References:
- 1 -
Deliverable7.2.1/draft
Deliverables Report
IST-2001-33310 VICTEC
1 Purpose of Document
The purpose of this document is to report the evaluation of the VICTEC software “FearNot”. It describes the application very briefly, details the evaluation design, provides information on activities already carried out as well as on activities that are yet to be completed. Moreover, first results are reported.
The document focuses on decisions made by the VICTEC team in order to adapt the evaluation activities to the development of the application. In changing the evaluation schedule and protocol, these decisions consider both the altered circumstances related to the software development and the specifications that resulted from the last project review.
The This document targets aims to give provide an overall idea of the planned evaluation activities with a special focus on alterations made to adapt the design to current requirements.
2 Executive Overview
The goal of work package 7 is to develop an appropriate evaluation methodology (as has been previously outlined in Deliverable 7.1.1) and to evaluate the VICTEC toolkit as well as the VICTEC Demonstrator. The current document reports on the latter. It again outlines briefly the design for the pedagogical and psychological evaluation of the Demonstrator (for detailed description see Deliverable 7.1.1), puts special emphasis on the adaptations made to this design due to the ongoing software development process, and reports results from already completed evaluation activities.
Before describing the evaluation process, chapter 4 provides some information on FearNot (Fun with Empathic Agents to Reach Novel Outcomes in Teaching), as the VICTEC demonstrator has meanwhile been named. Since the final version of the application has not yet been reachedis not yet available, the current evaluation process uses a prototype version needs to be introducedwhich is fully explained in this document.
In chapter 5, three different evaluation approaches are detailed. First, an evaluation study of a prototype version depicting one physical bullying scenario is reported. Second, the Psychological Evaluation event in Herts (University of Hertfordshire, June 2004) is briefly described (again, for detailed description see Deliverable 7.1.1) and preliminary results are discussed. Third, the Psychological Evaluation activities are briefly portrayed with a special focus on organisational adaptations, also including a short discussion of preliminary results.
3 Introduction
“FearNot” is a software application that addresses the problem of violence in school – also referred to as bullying. It depicts bullying incidents between autonomous agents in a virtual school. By the use of these autonomous agents the application aims at triggeringto trigger empathy within the child users and therefore helpshelp to reach the educational goal: reducing bullying among pupils.
To investigate whether its the Victec goals can have be been reached, a broad and differentiated evaluation design has been developed by the VICTEC team – taking into consideration empathy between child users and virtual agents and , knowledge transfer about bullying and empathy. – a broad and differentiated evaluation design has been developed by the VICTEC team. This evaluation design has already been detailed in Deliverable 7.1.1 (Operational Evaluation Methodology).
However, some changes have been made to this evaluation design, partly due to the development process, partly due to the recommendations resulting from the last project review. These adaptations mainly concern the schedule of the pedagogical evaluation.
Other adaptations?
Since the evaluation is currently ongoing, preliminary results are reported in this document.
4 The VICTEC Demonstrator “FEARNOT”
The VICTEC team has decided to name the main product of the project FearNot (Fun with Empathic Agents Reaching Novel Outcome in Teaching). It depicts bullying incidents in the form of an episodic virtual drama. The child user views the bullying incidents between autonomous agents in a virtual school and acts as the ‘invisible friend’ of the victimised character in between episodes, providing help and advice. Every Each episode is framed by an introduction segment at the start of the episodes, and a reflective interactive segment at the end.
For this document – , dealing with evaluation in schools, – two versions of FearNot are important. As outlined in chapter 5, specific constraints related to the software development brought about an alteration of the original pedagogical evaluation design, resulting in an evaluation phase being carried out using a scripted prototype before summer, and a second evaluation phase being carried out using a more elaborate version of FearNot after the summer. In the following, the scripted prototype will be described.
The FearNot Prototype
In this application, the child user views one physical bullying scenario and one relational scenario. Each child user initially provides their personal information (name, gender and age) and a unique personal code that matches the their off-line questionnaires. After the introduction of the characters, school and situation, users view the first bullying episode, followed by the victimised character seeking rescue in the school library where it starts to communicate with the user. Within the initiated dialogue the user selects an advice from a list of given coping strategies (realized shown as a drop down menu) and also explains his/her selection and what he/she thinks will happen after having implemented the selected strategy, by typing it in. Then , theThe next episode then starts. The content of the last episode depends on the choices made by the user concerning the coping strategies: Paul, the bystander, might act as a defender for John (the victim), in case the user has selected a rather successful strategy, i.e. “telling someone”; or Martina (the bystander) might offer Frances (the victim) help, but Frances rejects it, in case the user has selected a rather unsuccessful strategy, i.e. “run away”. At the end of an episode, an educational message is displayed pointing out that “telling someone” is always a good choice.
All episodes of the prototype are scripted. The end of an episode depends on the response selected during the second interaction phase. If that response is in the “Tell someone” category, a “good” third episode is shown. Otherwise, a “bad” third episode is shown alternatively.
ð Description of the final version in September
5 Evaluation of FearNot in Schools
5.1 Evaluation of a single physical bullying episodethe physical bullying scenario (3 episodes) (UK)
The VICTEC team feels that it is essential to involve the target group in the evaluation process in at an early stage of the development, since they children have social and cognitive expectations and capabilities different from the an adult perspective. As part of the pedagogical evaluation,To early involve children in the further development of prototypes, a small scale evaluation study has been carried out with 59 school children aged 8-11 years (24 boys and 35 girls) in the UK. They Children interacted individually with a prototype version of FearNot that only depictsdepicted one a physical bullying scenario comprised of 3 episodes (including John the victim, Luke the bully, and Paul the bystander), but apart from this, equals the prototype described in chapter 4 in the current document.. After the interaction they participated in a Classroom Discussion Forum (Hall, Woods & Dautenhahn; forthcoming), a method designed to assess child users’ views and perspectives on interactive applications.
Method: Classroom Discussion Forum (CDF)
For reasons of classroom logistics, the CDF approach was designed to obtain child users’ verbal feedback about their views, expectations and needs concerning the software and additionally considering teachers preferences for a small group discussion approach. The 15-minutes – CDFs were lead by two trained researchers who ensured that everyone participated and created an encouraging and enjoyable atmosphere. The topics tackled covered by the facilitators included:
ð cover
ð Levels of interest and enjoyment that the children experienced from interacting with FearNot.
ð Interacting with FearNot and the synthetic characters - design, information provision, navigation approaches, and interaction style.
ð Design of characters, focusing particularly on emotions and children’s emotional responses to the events in the episode.
ð Changes to improve FearNot particularly related to types of advice, endings and educational goals.
Results
The analysis of the qualitative data obtained has yielded the following results.
1) Interacting with FearNot
No hesitation or difficulty in exposure to the technical equipment (mouse, tracker pad, and keyboard) could be observed, albeit the typing speed was generally slow. Children had experience with computers, mainly from recreational games and the internet. Concerning the control of the pace of the interaction, children showed impatience and a desire to participate, voting for ana preference for interaction with FearNot to compared to a teacher-led session such as Circle Time. During interaction phases, it took them children far more timelonger than has been expected to select a coping strategy and provide an explanation for it, indicating that they thought hard on about which strategy was the appropriate one and why.
2) Engagement
Children were pleased to use the application and found it “interesting, fun, better than books”. They highlighted the privacy and the “real” feeling of FearNot as special benefits and expressed the desire to further use the application “for different stories” as well as for knowing more about the characters’ future. They also indicated that FearNot has helped them to learn more about bullying and that, compared to a video, the possibility of giving advice to the victim was of special value to them. Besides, children reported to sensehaving feelings towards the characters, i.e. “feeling sad for John” (see below).
3) Emotions
Children reported to have feltfeeling several emotions towards the characters, ranging from “sad for John” to “angry with Luke” and various reactions to Paul.
4) Movement and appearance of characters
Children reported problems to rememberremembering the names of the characters, while they had no problem to rememberproblems recalling the bullying roles: They favoured to referreferring to the characters as “the bully”, but hand problems to recallrecalling the bully’s name. Besides, graphicsGraphics scored rather low, and children indicated that they found the embodiment of both objects and characters of rather low standard.
Implications for the design process
The implications derived from the results and translated for the design process can be summarized as follows.
ð Concerning interaction: Child users favour point and click and the use of texting. They furthermore want more control over the interaction pace and more background information on the characters and the story.
ð Concerning engagement: Appreciation of the project’s main idea to create a software application on bullying as children prefer the interactive application to a teacher-led session on the topic, mainly because of matters of privacy.
ð Concerning emotions & character appearance: Empathic engagement was quite high, despite the negative rating of characters’ physical appearance and emotional expression, indicating that these characteristics are not indispensable ingredients when trying to design believable agents. The children seem to infer the emotions from situational cues rather than to rely on the characters’ facial and body expression.
Overall, the CDF method proofed to be an appropriate instrument to evaluation interactive applications with child users. Children were keen to participate and teachers also found the format appropriate. It seems to be a rather economic way to obtain rich and valuable data in a rather short period of time which can be translated very quickly into design recommendations. Some implications should be kept in mind when using CDFs to gather data:
ð All children of a class should participate.
ð There should be no technical problems with the application: This seems to lead to highly negative data obtained with the CDF method.
ð Testing should take place in the classroom/school setting (logistical difficulties, preferences of teachers, pupils, parents).
5.2 Psychological Evaluation
o Short description of goals and activities planned
o Alterations to the design?
o First results? Deliverable is not due before end of July, maybe we could include something in a couple of weeks
5.3 Pedagogical Evaluation
Objectives
The evaluation in schools mainly concerns what has been outlined in previous deliverables as Pedagogical Evaluation. Within these specific evaluation activities the team aims at the assessment of cognitive, behavioural and affective effects of the child users’ interaction with FearNot. It is carried out with four school classes in each country.
Cognitive effects of FearNot: These relate to whether the child user has learned something about bullying due to the interaction with FearNot, whether there has been knowledge transfer concerning bullying situations, reasons for bullying incidents, and strategies are more or less successful for victims and bystanders to apply. Cognitive effects are partly assessed with questionnaires, but also by qualitative means as interviews.
Affective effects of FearNot: Regarding affective effects of the FearNot interaction, empathic reactions on behalf of the users are measured with the Empathy Questionnaire. Also, during interviews affective outcomes of the interaction are assessed.
Behavioural effects of FearNot: We also want to know whether there are more, less or other types of bullying taking place in school classes after the interaction with FearNot. These behavioural aspects are evaluated from a pupil’s (Bullying Questionnaire) and from a teacher’s (Teacher Rating) perspective.
Along the way, the VICTEC team has a strong desire to include the teachers into our work. To insure high teacher cooperation provides the project with information on how to improve the application and how to integrate it in school curricula at a later stage of the development.
Design
The evaluation will be carried out according to a pre-/posttest design: 3-4 weeks prior to and after the interaction with FearNot, questionnaires will be completed by the pupils and their teacher. The pupils complete the Empathy Questionnaire, Bullying Questionnaire, and Picture-Story; additionally, the teacher fills in a rating of every pupil’s bullying behaviour (Teacher Rating). These four diagnostic instruments will be applied in pre- and posttest (for a detailed description of the instruments see D:7.1.1). The evaluation will be conducted as a field study in the three participating countries (Portugal, UK and Germany) in four school classes each. To meet the requirements of the target group the subjects will be aged eight to twelve and will be from rural as well as from urban schools. Overall approximately 300 pupils will participate in the Pedagogical Evaluation.