How machines sense emotions

There are days when it just doesn't work. There are countless emails to deal with. Then there are conference calls, and then maybe there's an argument in the office. When you get into the car in the evening, the tension is there too. You drive more briskly, are inattentive - and there's a crash. Assistance systems that recognise the driver's condition would be helpful in such moments - a camera that detects the tension in the face, the reddened skin or the sweat on the forehead; or a sensor in the steering wheel that detects a high pulse rate. The car could then automatically switch to a calming programme and set the air conditioning to a comfortable temperature or play soothing music.

This scene is not a dream of the future. Fraunhofer IIS is working intensively on using sensors to analyse people's emotions and moods - not only for intelligent driver assistance systems that detect stress, but also, for example, to determine how people react to a particular product. Researchers are now even able to recognise pain in people's faces - for example in people with dementia who can no longer communicate how they are feeling. We are talking about EmotionAI, the linking of sensors that record body data and artificial intelligence (AI) that reads a person's emotional state from this data. ‘Our work is special because we link the information from many different sensors,‘ says EmotionAI expert Jaspar Pahl - such as the ECG, which measures the pulse, cameras that observe the test subjects’ faces or even electrogastrography, which measures intestinal activity from the outside. This can also be a measure of tension.

Higher quality thanks to many sensors

Fraunhofer IIS offers emotion recognition as a service. Firstly, for research teams that are working on scientific issues but do not have such a variety of sensors themselves. Secondly, for companies developing new products such as systems for monitoring the condition of vehicles. Market research, on the other hand, wants to test how test subjects react to products - such as a detergent with a new fragrance. Jaspar Pahl and the team rely on many different sensors because this compensates for the weaknesses of a single one. ‘Cameras are sensitive to backlighting, ECGs can be disturbed by radiation from electrical devices. If you use several sensors, the analysis by the AI is more reliable, less prone to interference and better.’ In recent years, the institute has developed the ‘ExpoBox’ for the tests - a small room in which test subjects can be measured with the full range of sensors. Here, manufacturers can also test which sensor is both practical and reliable in everyday use. To measure the stress level of car drivers, for example, it might make more sense to install a heart rate monitor in the steering wheel than a camera that is irritated by the sun or the headlights of other cars.

Discovering unknown correlations

In recent years, the experts at Fraunhofer IIS have developed self-learning algorithms to analyse the sensor data. These are able to extract certain patterns from the variety of information that indicate the emotional state of the test subjects. ‘A human would never be able to recognise correlations in such a volume of data,’ says Jaspar Pahl. For him, analysing emotions is one of the most important future tools for human-machine collaboration, because there is an enormous amount that can be done with it once the technology has been introduced. A camera in the car, for example, can recognise whether the person behind the wheel is tired - but also whether they are sweating or stressed. In the future, it should even be possible for the machine to predict what needs to be done based on the person's condition - before the user has to give an order, says Jaspar Pahl.