The starting point for Emotional AI is the work of Paul Ekman.  Ekman was inspired by Charles Darwin, who wrote about the universality of emotions. He was also inspired by Duchenne de Boulogne, the French Neurologist best known for his grotesque photographs of people with their faces contorted into grimaces by the application of electrical current.  Duchenne was interested in physiognomy – the relationship between our physical being and our internal being.  Ekman was interested in the idea that our faces could show a specific range of emotions.  He also worked with an isolated tribe in New Guinea to gain evidence that emotions are universal.  

Ekman identified six universal emotions: fear, anger, joy, sadness, disgust, and surprise.  These have been used for a number of Emotional AI models including by Affectiva, one of the largest Emotion AI companies, now a subsidiary of a vison tech company, Smart Eye.

This is not the only model used. For example, some tools also label data based on the intensity or valence of the emotion. Some researchers also use more than the face, for example, looking at gait. It is also true that the underlying models will be improving over time.

Is AI better than humans at recognising emotions?

According to an article by (Dupré et al., 2020) the answer is “no”.  Or, as we should say with a developing field, it was “no” in 2020.  Dupré et al tested eight different commercially available classifiers and none was as good as a group of humans.  Classifiers were better at identifying simulated emotions, but still not as good as humans.  This could well be because simulated emotions may be over-acted (see images below).

The Dupré et al article gives a very good summary of the problems with classifying emotions:

• The test used a standard set of 6 basic emotions, but in reality people have a much wider range of emotions.  (One could argue that they are combinations of the basic emotions, but that would not necessarily increase accuracy of classification).
• Classifiers assume that there is one-to-one correspondence between emotions and facial expression.  But people display emotions in a range of social contexts which will call for different expressions and in some contexts those expressions are simulated – just because someone is smiling does not actually mean they are happy.
• In addition, there is considerable individual difference in emotional expression
• More recently classifiers have started to include non-effective categories e.g. interest, pain, boredom, frustration.  However, this does not address the underlying issues fully.  What it may enable classifiers to do is to provide some useful information e.g. to detect drowsiness in drivers.  But this seems a long way from providing accurate information on what people are thinking or feeling.

What does an Emotion AI training set look like?

Here are stills of video image used in ADFES – a data set of images used to test models of emotion produced by the Amsterdam Interdisciplinary Centre for Emotion (AICE) at the University of Amsterdam.  

https://psyres.uva.nl/content/research-groups/programme-group-social-psychology/adfes-stimulus-set/stimulusset.html

To my eye these look poorly acted.  The first and second expressions, starting from the left, are pretty strange.  Little wonder that models trained on images like these struggle to identify emotions.

How is Emotional AI used?

“MorphCast Interactive Video Platform can offer benefits to:

• Digital ADV, to effectively capture attention through personalized and interactive videos, increase the emotional engagement of customers and select real people from BOTS
• Digital learning, to personalize the learning path by monitoring students’ attendance, mood and attention, simplify and personalize the learning process
• Entertainment, to create interactive videos, films and video clips
• Retail and OOH applications, to personalize in-store video communication in real time and automatically collect in-depth customer data
• e-Commerce, to personalize video communication in real time and automatically collect in-depth data from new / unregistered users
• There are many other industries that can use MorphCast VPaaS to customize and simplify the user experience.

Affectiva Media Analytics

• We help businesses understand how their customers and consumers feel when they can’t or won’t say so themselves. By measuring unfiltered and unbiased responses, businesses can act to improve customer experience and marketing campaigns.
• Human Perception AI. Our software detects all things human: nuanced emotions, complex cognitive states, behaviors, activities, interactions and objects people use.

5     References 

Ekman, P. ( 2007). The directed facial action task. In J. A. Coan and J. J. B. Allen (Eds.), Handbook of Emotion Elicitation and Assessment (pp. 47-53). Oxford University Press.

Ekman, P., & Friesen, W. V., & Hager, J. C. (2002). Facial action coding system: The manual on CD-ROM. Instructor’s Guide. Salt Lake City: Network Information Research Co.

Dupré, D. et al. (2020) ‘A performance comparison of eight commercially available automatic classifiers for facial affect recognition’ In: PloS one 15 (4) p.e0231968.