SSPNet

Social Signal Processing Network

project no. 231187

Annual Public Report 2011

Project Description

The ability to understand and manage social signals of a person we are communicating with is the core ofsocial intelligence. Social intelligence is a facet of our cognitive abilities that has been argued to be indispensableand perhaps the most important for success in life. Although each one of us understands the importance ofsocial signals in everyday life situations, and in spite of recent advances in machine analysis and synthesisof relevant behavioural cues like blinks, smiles, crossed arms, head nods, laughter, etc., the research effortsin machine analysis and synthesis of human social signals like empathy, politeness, and (dis)agreement, arefew and tentative. The main reasons for this are the absence of a research agenda and the lack of suitableresources for experimentation.

The mission of the SSPNet is to create a sufficient momentum by integrating an existing large amountof knowledge and available resources in Social Signal Processing (SSP) research domains including cognitivemodelling, machine understanding, and synthesizing social behaviour, and so:

  • enable creation of the European and world research agenda in SSP,
  • provide efficient and effective access to SSP-relevant tools and data repositories to the research community within and beyond the SSPNet, and
  • further develop complementary and multidisciplinary expertise necessary for pushing forward the cutting edge of the research in SSP.

The collective SSPNet research effort is directed towards integration of existing SSP theories andtechnologies, and towards identification and exploration of potentials and limitations in SSP. More specifically,the framework of the SSPNetrevolves around two research foci selected for their primacy andsignificance: Human-Human Interaction (HHI) and Human-Computer Interaction (HCI). A particular scientific challenge that binds the SSPNet partners is the synergetic combination of human-human interactionmodels, and automated tools for human behaviour sensing and synthesis, within socially-adept multimodalinterfaces.

Project Goal and Objectives

The goal of the SSPNet is to establish a virtual distributed institute for research on Social SignalProcessing (SSP), i.e. on conceptual modelling, detection, interpretation, and synthesis of social signals:

  • Conceptual modelling means that social signals will be described by cognitive scientists based on the experimental observations of social interactions.
  • Detection means that social signals will be extracted automatically from sensor data (e.g., video and audio recordings).
  • Interpretation means that social signals will be mapped automatically into socially relevant information (e.g., presence or absence of conflict).
  • Synthesis means that social signals will be generated artificially with computers to elicit desired social perceptions (e.g. to make artificial agents more friendly).

The goal is articulated into 10 concrete objectives:

  1. Building the framework for structuring and sharing the SSPNet knowledge.The SSPNet is defining and discussing the agenda for research on SSP, providing the resources for experimentation (see Objective 9), and establishing a virtual SSPNet knowledge repository through which the SSP-relevant information is accessible to theresearch community within and beyond the SSPNet (
  1. Describing the principles and laws of social interaction and social relations. The SSPNet studies the laws underlying social interactions, makes them explicit, and investigates how are these laws expressed and influenced by social signals.
  1. Detecting, interpreting, and synthesizing the phenomena observed and described by social scientists. Social scientists have studied social interactions for the last seventy years and they have shown the existence of social signaling patterns stable at least for specific cultures and social situations.The patterns are the observable evidence of interaction quality and as such they can be detected, interpreted, and synthesized automatically. The development of such automatic, multimodal tools is a very important objective of the SSPNet.
  1. Focusing on group interactions.The investigation of group interactions is particularly challenging from the technological point of view because it requires algorithms capable of taking into account multiple signal sources involved in complex interaction patterns. The SSPNet addresses such a major challenge for all modelling, detection, interpretation, and synthesis.
  1. Assessing the effect of the context on social interactions. The SSPNet addresses several problems relevant to the effect of context on social signals, including the discrimination between posed and spontaneous behaviour (e.g., politeness vs. irony vs. enjoyment), the communication effectiveness, the persuasiveness, and the quality of information delivery.
  1. Improving technology with social signals. The SSPNet investigateshow social signalling patterns can be used as a-priori information in technologies dealing with human-human and human-machine interaction (e.g., how to improve speaker diarization taking into account turn-taking regularities, how to make synthetic voices more socially appealing, etc.).
  1. Addressing cross-cultural issues.The SSPNet fosters activities addressing explicitly cross-cultural issues to interpret correctly data collected in different cultures, produce synthetic social signals perceived in the same way in different cultures, and assess the effect of culture on social situations.
  1. Identifying relevant applications. The SSPNet explores the identification of concrete applications. The efforts will include in particular the contact with companies active in domains like call centres, multimedia analysis, production and indexing, marketing analysis, communication, etc.
  1. Collecting datasets annotated in terms of Social Signals. The SSPNet is collecting existing repositories of relevant data including TV and radio shows, meetings, human-agent interactions, etc., and makes the data available to the research community via the SSPNet portal (
  1. Establishing performance assessment criteria. The SSPNet contributes to the definition of evaluation and benchmarking procedures accepted by the community via the organization of national and international SSP centred challenges.

The SSPNet Portal

The main outcome of the SSPNet is the web portal ( an instrument that aims at providing any interested researcher with the most important and yet difficult to obtain resources for working in SSP, i.e., the knowledge, data, and tools[1].

Figure 1. Some of the presentations currently available on the portal.

In particular, the portal hosts four repositories of publicly available material that allow virtually anybody to work on SSP related problems:

  • Bibliography: a collection of references (including the publications of the project) that provide not only the background of the domain, but also state-of-the-art works on the most important SSP trends.
  • Data: a collection of annotated corpora containing audio and video material annotated in terms of social signals.
  • Tools: a collection of software packages addressing some of the most important needs in SSP experiments and research.
  • The Virtual Learning Centre: a collection of lecture and presentation recordings covering a wide spectrum of SSP related subjects (see Figure above).

The repositories are enriched with continuity and they are fully searchable and browsable. In its first 30 months of life, the portal has been visited 55741 times by 27711 unique visitors.

Dissemination and Training

The SSPNet organizes a large number of events aimed at gathering and fostering a vibrant SSP research community. The events of 2011 have been (in chronological order):

First International Workshop on Emotion Synthesis, Representation, and Analysis in Continuous Space

Santa Barbara (SA) – March 21st, 2011

FG 2011 Facial Expression Recognition and Analysis Challenge

Santa Barbara (USA) – March 24-25, 2011

COST 2102 International Training School on Cognitive Behavioural Systems

Dresden (Germany) - February 21-26, 2011

International Workshop on Voice and Speech Processing in Social Interactions.

Glasgow (UK) - April 18-19, 2011

Fourth International Workshop on CVPR for Human Communicative Behavior Analysis.

Colorado Springs (USA) - June 25, 2011

International Summer School in Affective Sciences

Geneva (Switzerland) – August 22-31, 2011

First International Audio/Visual Emotion Challenge and Workshop

Memphis (USA) – October 9th, 2011

Second IEEE International Workshop on Socially Intelligent Surveillance and Monitoring.

Barcelona (Spain) - November 12, 2011

Third International Workshop on Social Signal Processing.

Scottsdale (USA) - December 1, 2011

The Consortium

/ University of Glasgow (UK)
Alessandro Vinciarelli, Coordinator
/ Imperial College London (UK)
Maja Pantic, Scientific Coordinator
/ Idiap Research Institute (CH)
Fabio Valente
/ University of Edinburgh (UK)
Steve Renals
/ University of Twente (NL)
Dirk Heylen
/ University of Roma Tre (I)
Isabella Poggi
/ Queen’s University Belfast (UK)
Roddy Cowie
/ DFKI (D)
Marc Schroeder
/ CNRS/TELECOM ParisTech (F)
Catherine Pelachaud
/ University of Geneva (CH)
Marc Mehu
/ Delft University of Technology (NL)
Emile Hendriks
/ University of Gothenburg (S)
Jens Allwood

[1]The SSPNet portal has been featured in the “Best of Web” column of the IEEE Signal Processing Magazine (A.Vinciarelli and M.Pantic, “A Web Portal for Social Signal Processing”, IEEE Signal Processing Magazine, 27(4):142-144, 2010).