Deliverables Report
IST-2001-33310 VICTEC
PEER REVIEW REPORT
AUTHOR: Paul Brna
STATUS: Discussion/final
DATE: 1st August 2002
PROJECT MANAGER
Name: Ruth Aylett
Address: CVE, Business House, University of Salford, University Road,, Salford, M5 4WT
Phone Number: +44 161 295 2922
Fax Number:+44 161 295 2925
E-mail:
TABLE OF CONTENTS
1. EXECUTIVE OVERVIEW 3
1. Clarity 3
2. Accuracy 3
3. Completeness 3
4. Relevance 3
5. Depth 3
2. CLARITY 5
3. ACCURACY 6
4. COMPLETENESS 7
5. RELEVANCE 8
6. DEPTH 9
7. OTHER COMMENTS 10
1. EXECUTIVE OVERVIEW
This document lays out some suggested evaluation criteria for the peer review of the first of deliverables of the VICTEC project subject to this process. The deliverables in question are:
D1.1.1 Vision document (end May 2002)
D2.2.2 Evaluation Methodology (end Jan 2003 )
D3.3.1 VLE Toolkit version 1 (end April 2003)
D6.4.1 First prototype of demonstrator (end Jan 2004)
D7.2.1 Final evaluation of demonstrator (end July 2004)
D7.3.1 Final evaluation of toolkit (end August 2004)
It can be seen that some of these are documents and some are software, so the criteria to be applied are likely to be different. In addition, the first deliverable, that is the first version of the Vision document, had a particular purpose: to articulate a vision of where the project team would like the project to go in order to promote close co-operation between the geographically separated teams.
The following review criteria are therefore suggested. This is in no way intended to inhibit the peer reviewers from adding other criteria or from making comments not covered by particular headings: the purpose of peer review is to help the project to improve what it is doing and all comment to this end is welcome.
1. Clarity
Is the document well-organised, clearly written and easy to follow, given its target audience of project members?
2. Accuracy
Where the document makes assertions of fact, are they correct? Are inferences logical, are there any jumps in reasoning?
3. Completeness
Are all the relevant topics covered, or is there other material that ought to be discussed in the next update of the Vision document?
4. Relevance
Are the topics covered relevant to the scope of the project as outlined in its Technical Annex?
5. Depth
Does the document cover topics with adequate thoughtfulness and depth?
2. CLARITY
Generally the document is well organised and clearly written.
Issues that might be examined to improve the organisation:
· The purpose of the document briefly mentions project outputs and policy decisions. It might help if these are referenced at the end of the document by summarising them?
· The issue of the toy interface pops up out of the blue - project members will know about this, but some idea of the importance of this part of the work would be helpful. Structurally it might help if the intelligent toy interface is related back to the goals of the project in the overview.
· The deliverables are outlined clearly - the project is to produce an authoring system and a demonstrator. Some idea of the other PSE topics might be a good move to provide face validity to the utility of the authoring system. I guess this must have been done already somewhere – perhaps the proposal? This point touches on how you go about arguing that, at the end of the project, you have delivered a broadly useful authoring system (i.e. useful in principle for other areas of PSE). This might then help the project members understand how important emotions are outwith the domain of bullying. (But I can easily believe that the partners understand this already.) This issue comes also under the heading of completeness in relation to the point of how you are going about evaluating the work as you go.
· Maybe worth spelling out briefly how the narrative issue relates to the “main purposes of VICTEC” – even if this relation might be simply to provide a platform for educators in PSE to trial with children. Personally, I’d see the ‘something” the child takes away physically as being closely linked to the goals of the project in relation to the educational justification for VICTEC.
3. ACCURACY
The document is broadly accurate. Places where there are some problems are:
In the Introduction, it is difficult to “create empathy”, it seems to me you are trying to evoke empathy.
Potentially, there is a great deal hidden in the phrase “background history”. Is this background history simply a record of physical interactions, and/or is it a record of the ‘emotional interactions’? It is even possible that some account is imagined in terms of a “bullying-theoretic language”? I imagine not the latter since this would undermine the generic nature of the authoring tool. However, a PSE-theoretic language would be an interesting idea though almost certainly out of the scope of the proposal.
I am not sure that the notion of background history has been closely defined so perhaps all that is needed is a disclaimer to the effect that the background history is to be discussed further.
This point also relates to the issue of Scenario Authoring, when it becomes possible that some elements injected into the scenario will be used to build the background history.
In section 4.1.3, interaction is not just about language used but surely more about the ways in which individual agent behaviours are linked. If, as suggested in 4.1.4, there is no intention to control the course of events, it becomes difficult to understand how the system will ensure that incidents of interest to the child actually do happen. The “devil is in the detail” of how “motivating situations” are to be described, how situations will be indexed and how social relationships will be described etc. The process of bullying (cf section 5) requires some further elucidation but this notion does need clarification.
Personally I am very interested in how the project team will develop these key “data structures”. Since the document is a vision of how things will progress then we can hope for an interesting time ahead.
In 4.2, it would help if the SAFIRA project was given a reference.
4. COMPLETENESS
Generally the document is “complete” but some other issues could do with discussion in some later report in addition to the issue of evaluation mentioned earlier which, no doubt, will be discussed in D2.2.2.
Cover Story
Rerunning scenarios: re the point about being able to rerun scenarios would reduce believability . The question is “believability in what”? If it is “I believe what I am seeing is happening now” then the proposition is plausible but if it is “I believe I am seeing what happened a few minutes ago” then the proposition is much less plausible. So, it depends on the way in which the viewing of events is embedded into the “cover story” provided to the child when using the software. While the project is not committed to developing materials, from an interaction perspective, some thought has to be given to how a clear and convincing cover story is conveyed to the child using the system.
Introduction to Characters and Scenarios
Why only a non-interactive intro to the characters and the scenarios? There may be low cost ways of giving a child the opportunity to interact with the individual characters - as to whether this is useful or not, that is not clear at this stage. The key issue would be “in what way does the child interact”? My initial thoughts suggest that this is non-trivial in that I assume you don’t want to detract from the main purpose - ie the examination of bullying (or other PSE events). Getting involved in a lot of curiosity driven interaction with more or less realistic characters would not necessarily focus the mind on the problems faced by those being bullied (or bullying). This thinking also suggests that making too complex a non-interactive intro may be self defeating... I’d be looking for some notion of simplicity in the way this is done.
The Role of the Invisible Friend
The notion of the child being an “invisible friend” seems very sound. The ethical issues are outlined carefully and the point about getting too absorbed in activity is persuasive. Hot seating is one way of getting reflection into the process but one of the points that has to be decided is to whether to try and get the child to examine alternative possibilities for behaviour as events unfold. So is there a role here for timeouts/freezing the action while the child offers alternative actions to the bully or bullied? The document suggests this may be possible but it looks like there is no settled view as when to get the child involved with the action and when to get the child to stand back.
5. RELEVANCE
The topics discussed are all relevant.
6. DEPTH
Some further deeper discussion would have helped in relation to:
Scenarios
As mentioned, there is a need to go into the definition of a scenario, especially when there is a need to understand how scenarios are linked – especially about the ways in which relationships and back-story events are carried forward/traced backwards.
Speech Generation
Comments in the report are quite brief, seem accurate but don’t give any insight into how realistic it would be to develop an application using speech generation. The problem with generating Portuguese is known to me and the option of using recordings is useful but has its own set of problems. Some discussion would be useful to map out the pros and cons of each decision.
Toy
The relationship of the toy to the interaction issue is opaque at present. Options have been outlined but no clear guiding principles can be detected (yet)! It would have helped to go deeper into this issue.
Log
To give children a narrative to take away is an excellent idea – but more is needed on this since the kind of narrative needs to be thought about. The options vary from a straightforward story about physical events to a more complex story featuring motivations, character descriptions and reflections on the situation.
7. OTHER COMMENTS
Movement within the Environment
An excellent decision, it seems to me, to seek to ‘anchor’ children to the vicinity of the bullying in some way. There is at least one other way than those mentioned - use sound to make sure that wherever the child is looking/wherever the child is there is something going on to make them aware of problems. This might avoid the need to reproach children for not looking at/being near the action or to force the child back to the location of the bullying.
Physical Bullying
One thought about physical bullying... it surely isn’t the case that this is simple. It cannot just be reduced to physical activity. For it to be bullying suggests there are other important things going on. Some discussion from a more “psychological” viewpoint might clarify this issue.
Interaction Design
Indeed, I would like to see some discussion of the core interactions that relate to the social dimension. These are the heart of the matter and I would hope that someone in the project somewhere is working on what might constitute a good interaction between an animated character and the child – i.e. what kinds of exchange are regarded as “helpful”? If these helpful exchanges can be described then the design decisions relating to invisible friends etc can be clarified. Of course this is a “chicken and egg” situation and it will be hard to do without understanding the scenario in which the interaction takes place so I hope this point will be
- 9 –
Peer Review Paul Brna/report1.1.1/version1