contract no 031517 / / Co-funded by the European Commission
TRAIN-ALL
Integrated System for driver Training and Assessment using Interactive education tools and New training curricula
for ALL modes of road transport
Contract no. 031517
Driving simulator functional validity
assessment methodology
Dissemination Level / Public
Workpackage No. / WP2 / Workpackage Title / Towards a single training and assessment platform
Activity No. / A 2.5 / Activity Title: / Simulator validity assessment methodology
Workpackage Leader / Wim Huiskamp (TNO)
Authors (per company, if more than one company provide it together) / Bjorn Peters, Mats Lidström (VTI), Katerina Touliou (CERTH/HIT)
Status (Final; Draft; Revised Draft): / Final updated
File Name: / TRAIN-ALL Deliverable 2.3_V2.doc
Project start date and duration: / 01 November 2006, 38 Months
Submission date: / December 2009
Version Number: / V2
Distribution / All partners
Pages Number: / 64
V1- Page 2 of 46
TRAIN-ALL Deliverable 2.3contract no 031517 / / Co-funded by the European Commission
V1- Page 2 of 46
TRAIN-ALL Deliverable 2.3contract no 031517 / / Co-funded by the European Commission
Version history
Version / Date / Modifications0.1 / 22 May 2008 / First drafted version.
0.2 / 3 June 2008 / Draft document for comments by the partners.
0.3 / 17 June 2008 / Inclusion of some responses.
0.4 / 4 July 2008 / With corrections and adjustment based on peer review.
V1 / 9 July 2008 / End of the Peer Review; ready to be delivered.
V2 / December 2009 / Updated version, with the inclusion of the test results performed by CERTH/HIT (new Chapter 7).
V2 - December 2009 Page 20 of 60
TRAIN-ALL Deliverable 2.3contract no 031517 / / Co-funded by the European Commission
Executive Summary
The following Deliverable provides an outline of a method aimed at assessing driving simulator realism.
The ‘realism’ for the TRAIN-ALL driving simulation purposes (education and training), could be resumed to a unique measure of “goodness” that describes how “well” or closely the simulated driving represents the environment, the vehicle and their interactions for the trainee teaching benefit.
Simulator realism means how close driving is to real driving in terms of the driver’s subjective impression and in actual driving behaviour. Even if simulator realism per se does not ensure simulator validity in terms of training impact it is believed to have an influence on training transfer. A realistic simulator implementation might simplify the transfer from simulator to real driving. However, this is just one aspect of simulator based training and there are several other aspects that are equally important.
The aim was to develop a method that can be used to compare driving behaviour and perceived realism in different driving simulators. Furthermore, it should be possible to compare the collected data of driving behaviours to real driving on the road. Both questionnaires and recorded vehicle data (driving performance) should be collected.
26 driving sub-tasks (short artificial driving tasks) are described with the following headings: rational, description for implementation, instructions to the driver and a selection of vehicle metrics. The selection sub-tasks was based on the experience gained from many years of driving simulator experiments at VTI and also experiences for other simulators. The list of sub-tasks is a first attempt and has to be empirically tested and possibly revised. A description of a test procedure is provided. The main focus here was on how to use the different questionnaires of which two are described in Annex A and B.
Objective data should be recorded automatically during the test. Some details on objective metrics assumed to be used for the test are provided but also references to other relevant work. A Matlab tool to calculate some vehicle data is described. This tool can be provided by VTI. However, the current version does not include e.g. headway metrics.
CERTH/HIT conducted a test with some selected sub-tasks. The results supported the fact that easier tasks with free driving and highway environments may be the more valid for transferability of findings; however, larger cohorts are necessary to be included in order to further investigate relations of validity, learnability and transferability of measurements.
Table of content
Executive Summary 3
Table of content 4
List of Figures 6
List of Tables 6
Abbreviations List 7
TRAIN-ALL applicable documents 8
1 Introduction 9
1.1 A method to assess driving simulator realism 9
1.2 How to assess simulator realism 10
2 A proposal for a method to assess simulator realism 11
2.1 A method based on specific driving situation and test-driver interviews 11
2.2 Definition of the sub-task 11
2.3 Test-driver interviews through questionnaire 12
3 Driving sub-tasks for driving simulator validation 13
3.1 Sub-tasks for the driver’s perception of vehicle position 13
3.1.1 Perception of the lateral position on the road 13
3.1.2 Perception of the longitudinal position on the road 14
3.2 Sub-tasks for the driver’s perception of vehicle speed and acceleration 15
3.2.1 Perception of vehicle lateral acceleration 15
3.2.2 Perception of vehicle longitudinal speed 15
3.2.3 Perception of vehicle longitudinal acceleration and deceleration 16
3.3 Sub-tasks for the driver’s perception of vehicle control 16
3.3.1 Vehicle response to steering wheel manoeuvres and braking in a curve 16
3.4 Sub-tasks for the driver’s perception of the environment 18
3.4.1 Perception of stationary objects 18
3.4.2 Perception of moving objects - vehicles 19
3.5 Free driving 20
3.6 Overview of metrics per sub-task 20
4 Test procedure for simulator realism assessment 23
4.1 Example of simulator specification 24
4.2 Data to be recorded 25
4.3 Test leader 25
4.4 Selection of drivers – entry questionnaire 26
4.5 Questionnaires during the test 26
4.6 Final questionnaire 26
4.7 On the road 26
5 Objective measures for driving simulator realism assessment 27
5.1 Lateral position 27
5.2 Time to line crossing (TLC) 27
5.2.1 Calculation of TLC 27
5.2.2 TLC Minima and its distribution 28
5.2.3 A first approximation of TLC 28
5.2.4 A second approximation of TLC 28
5.2.5 Calculations of TLCmin – some considerations 29
5.2.6 Further reading 29
5.3 Longitudinal Distance 29
5.4 Distance to lead vehicle, Time Headway (THWY) and Time To Collision (TTC) 30
5.5 Brake jerks 30
5.6 Reaction Time (RT) and Brake Reaction Time (BRT) 30
5.7 Sampling rate 30
6 Tools to analyse driving behaviour data 31
7 Driving simulator realism assessment tests 32
7.1 Introduction 32
7.2 Method 32
7.2.1 Participants 32
7.2.2 Design 32
7.2.3 Procedure 32
7.2.4 Statistical analysis 33
7.3 Results 33
7.3.1 Following highway 33
7.3.2 Free highway 37
7.3.3 Following Rural 41
7.3.4 Free Rural 44
7.4 Discussion 47
8 Conclusions 48
9 References 49
Annex A Entry questionnaire 50
A1 Pre-test questionnaires (mandatory for all) 50
Annex B A realism questionnaire for TRAIN-ALL 52
B1 General realism (overall) 52
B2 Vehicle control 52
B3 Sensory impression – ordinary drivers 53
B4 Sensory impression – alternative for the expert evaluators! 55
B5 Car body (driver’s cab) 56
B6 Suggestions for improvements 56
Annex C Display System Performance 57
C1 Fidelity Driving Performance: armoured vehicle pilots’ point of view 57
C2 Introduction 57
C3 Resolution 58
C4 Displayed Surface Resolution performance 58
C5 Image perspective and geometry accuracy 59
C6 Image stability 59
Annex D Thales TRUST simulator specification 60
List of Figures
Figure 1: Double lane change manoeuvre. Lane width in section A: , in section B: and in section C: , where is the width of the car (ISO, 1999) 17
Figure 2: Definition of Landolt C 18
Figure 3: An outline of test procedure for simulator realism assessment in TRAIN-ALL 23
Figure 4: The VTI truck driving simulator; top left shows the cabin and the forward visual scene, bottom left shows the test leaders control, and right shows the lateral motion base system 25
Figure 5: Road geometry and road as seen by the driver 27
Figure 6: Time history of TLC (upper half) and how in relates to lateral position (lower half) (from van Winsum, W., Brookhuis, K., & de Waard, D. (1998)) 29
Figure 7: GUI of VTI’s Matlab software that can be used to calculate some driving behaviour measures 31
Figure 8: Speed instructions per environment and condition 32
Figure 9: Mean min THWY (sec) (Following Highway) 34
Figure 10: Mean THWY values (sec) (Following Highway) 34
Figure 11: Time to Headway (sec) (Following Highway: min/max/mean) 35
Figure 12: Mean values of speed (m/sec) (Following Highway: min/max/mean) 35
Figure 13: Mean values for lateral position (m) in the simulator 36
Figure 14: Mean values of LP (m) in the vehicle (Following Highway: min/max/mean) 37
Figure 15: Mean values for THWY (sec) (Free Highway: min/max) 38
Figure 16: Mean values of mean THWY (sec) (Free Highway) 38
Figure 17: Mean values for all THWY measurements (sec) 39
Figure 18: Mean values of speed (m/sec) (Free Highway: min/max/mean) 39
Figure 19: Mean values of LP in the simulator (Free Highway: min/max/mean) 40
Figure 20: Mean values for LP in the vehicle (Free Highway: min/max/mean) 41
Figure 21: Mean THWY (sec) values (Following Rural: min/max/mean) 42
Figure 22: Mean values of speed (m/sec) (Following Rural: min/max/mean) 43
Figure 23: Mean values of LP (m) for simulator (Following Rural: min/max/mean) 43
Figure 24: Mean values of LP for vehicle (Following Rural: min/max/mean) 44
Figure 25: Mean values of THWY (sec) (Free Rural: min/max/mean) 45
Figure 26: Mean values of speed (m/sec) (Free Rural: min/max/mean) 45
Figure 27: Mean values of LP (m) for vehicle (Free Rural: min/max/mean) 46
Figure 28: Mean values of LP (m) for simulator (Free Rural: min/max/mean) 46
List of Tables
Table 1: Overview of metrics per sub-tasks 21
Table 2: Technical data for the VTI truck driving simulator 24
Table 4: Displayed Surface Resolution performance 58
Table 5: Image perspective and geometry accuracy 59
Table 6: Technical data for the TRUST3000 truck driving simulator 60
Abbreviations List
Abbreviation / Definition /BRT / Brake Reaction Time
HMI / Human Machine Interaction
ISO / International Standardisation Organisation
MSDLP / “A Modified way to calculate Standard Deviation of Lateral Position”
Modified lateral position variation (see AIDE Deliverable 2.2.5)
NA / Not Applicable
RT / Reaction Time
SDLP / Speed, Lateral Left Position
SWRR / Steering Wheel Reversal Rate
THWY / Time Headway
TLC / Time to Line Crossing
TR / Technical Report (ISO/TR 3888)
TTC / Time To Collision
UC / Use Case
TRAIN-ALL applicable documents
Title / Deliverable / Dissemination /“Description of Work”, TRAIN-ALL project, 23, Feb.-06 CN. 031517, Sixth Framework Programme. / [DoW] / TRAIN-ALL DoW 230206.doc / Consortium
Project Management and Quality Manual:
TRAIN-ALL Project Deliverable 8.1 / [D8.1] / Project Quality Manual / Public
Benchmarking and classification of CBT tools for driver training:
TRAIN-ALL Project Deliverable 1.1 / [D1.1] / TRAIN-ALL Deliverable 1.1_final.doc / Public
Training Needs, Scenario and Curricula Definition and Specification of Tools and Curricula:
TRAIN-ALL Project Deliverable 1.2 / [D1.2] / TRAIN-ALL Deliverable 1.2_final.doc / Consortium
Common System Architecture for driving simulators based on interoperable federates:
TRAIN-ALL Project Deliverable 2.1 / [D2.1] / TRAIN-ALL Deliverable 2.1_final.doc / Public
Knowledge Management Tool, TRAIN-ALL:
Project Deliverable 2.2 / [D2.2] / Under production / Public
Simulation sickness aversion checklist / [D3.8] / TRAIN-ALL Deliverable 3.8.doc (under peer review) / Public
Verification Pilot Plans Framework, Questionnaires and Template:
TRAIN-ALL Project Deliverable 5.1 / [D5.1] / TRAIN-ALL Deliverable 5.1_final.doc / Public
1 Introduction
This deliverable was written as the result of the work in WP2/A2.5 “Simulator validity assessment methodology”. The objective according to the Description of Work [DoW] for this activity is to:
“Develop and evaluate a methodology for assessing the ’realism’[1] of training simulators for road driving; and the relation of the measured data to actual driving data”.
For more information see the DoW.
The focus in the work presented here has been on developing a method to assess simulator realism with the assumption that simulator realism has an impact on training transfer. The method does not aim at validating simulators with respect to training impact. The term functional validity mentioned in DoW was interpreted as a method that can be used to determine the level of simulator realism. The approach used is to utilise the driver as a probe to assess the perceived simulator realism (questionnaire) and actual driving behaviour (vehicle data). Even if there is no one to one correlation between simulator realism and training impact, realism can have an effect (Allen et al., 2007).
1.1 A method to assess driving simulator realism
A simulation is always an approximation of a real world. A diving simulator should be sufficiently realistic to ensure that a driver behaves in a way similar to real driving. Validation can have many different meanings but what is meant here (in the Activity2.5) is functional validity in terms of how close driving in the simulator is to real driving. Another aspect of validity also relevant to TRAIN-ALL is how valid a driving simulator is in terms of driver training. These two forms of validity are not interchangeable, e.g. realistic driving in the simulator does not insure the validity in terms of training impact. This is not to say that the level of realism in a driving simulator is irrelevant on the contrary perceived realism is important and thus there is a need to assess simulator realism. However, it might be confusing to use the word validation here specifically when considering the definition made by IEEE:
Validity: The property of a model, simulation, or federation of models and simulations representations being complete and correct enough for the intended use. (IEEE 1516.4)
The intended use (of simulators) within TRAIN-ALL is driver education and training but the aim of the present work was to assess simulator realism with the assumption that it will have indirect impact on the driver training efficiency. Thus, it was decided to avoid the word validation and rather talk of a method to assess simulator realism. Simulator realism assessment is a complex and difficult task which require both subjective and objective data. However, within the scope of A2.5 the ambition was to focus on the human aspect of the simulator realism and not so much on the technical aspects. There is of course a need for a technical specification of the simulator to go with the validation method: the produced data by TRAIN-ALL demonstrators have been studied and selected for recording in the Deliverable D2.2 [D2.2] for further analysis. But such a specification does not necessarily say very much about the perceived realism of simulator e.g. how much will an elaborated vehicle model contribute to the perceived realism. However, it is believed that the outlined validation method will disclose the lack of a sufficiently advanced vehicle model and if it does not then we can question the need for it.