1O th - 11th May 2005 Nice
Meeting minutes / Ref : MM_ETI_2_008
Date : 27/10/2018
May, 10th
------
- 10-12h: presentation of past and ongoing evaluation programs: PETS
(J. Ferryman) - CAVIAR (B. Fisher) - ILids (L. Sands and S. Walker) -
VACE-ARDA (R. Bowers) - VERAAE (S. Guler).
This presentation has presented issues in performance evaluation and shown the strong interest of the scientific community in defining a precise framework for performance evaluation.
- 14h-16h: presentation of the ETISEO evaluation protocol. This
discussion has highlighted the different expectations from the different
types of partners: end-users, industrials and researchers. End-users
and industrials want a competition oriented towards the evaluation of
particular applications. For instance, softwares should be tested on a
large amount of videos depicting all situations relative to the events
to be detected (e.g., abandoned baggage). Researchers are not kind for
competition but are looking forward to having a large amount of diverse
video and associated ground truth for conducting their own research
activities. Besides that, everybody agree on the great benefit of having
a data set of ground truthed videos illustrating specific technical
challenges. In addition, everybody agreed that the ETISEO evaluation
protocol will enable a fair comparison, even if some partners would have
prefered that competitors submit their softwares instead of the software
results. Several partners would like to have a large amount of ground
truthed videos to tune their algorithms through a learning stage.
However, it has been acknowledged that ground truth genereation is a
tedious task and that Silogic has volunteered to define as much ground
truth as possible. Some partners have proposed aROC curve to
illustrate algorithm performances. In order to realize it, each
participant will have to submit several XML results illustrating the
performances of their algorithms in function of some of their parameter
variations (wrt precision and sensitivity).
- 16-18h: presentation of some video, which are representative of the
ones that can be recorded for the purpose of the ETISEO project. To get
an evaluation protocol as fair as possible, it has been proposed that
ETISEO videos will be new ones (specially acquired for ETISEO) and that
already published videos (such as CAVIAR ones) could be used with the
ETISEO evaluation tool but on the side of the ETISEO project. Based on
the questionnaries filled up by ETISEO partners, it has also been
proposed that the evaluation will be oriented in five directions by
acquiring five series of videos. These five directions are:
Influence of camera position and mono/multi camera
Influence of camera resolution and compression rate
Influence of static (clutter) and dynamic (crossings) occlusions
Influence of shadows and illumination changes
Influence of event complexity
While studying these challenges on algorithms, other issues won't be
addressed: the influence of object posture, strong weather conditions
and of displacement of contextual objects. For these other issues, we
will make sure that the videos won't contain any of these difficulties.
Industrials and end-users have requested that videos should be taken
within an application context. RATP has potentially agreed on helping
with providing such videos. The ETISEO core group will think about these issues and will propose videos following these requirements within
September.
May, 11th
------
- 9h-17h: the ETISEO document on the definition of the video processing
tasks, concepts (what is a blob, mobile,) and evaluation metric has
been described and commented. It has been decided to store the ground
truth in a XML format since MGEP7 is still in an early stage of
standardisation. R. Bowers has proposed to give a new version of Viper
for Silogic to acquire ground truth. Several issueshave been raised.
For instance, while evaluating the impact of low resolution videos,
should the ground truth be realized on the initial high resolution video
or on the low resolution one?
Next ETISEO important dates:
------
- June: specification of the video data
- September: first video data set creation and feedback from participants
- December: ETISEO core group meeting in Lille
Page 1 of 2