Affective Avatars & The Voice AI Solution

Soroosh Mashal

Avatar in Hinduism refers to a manifestation of a deity or released soul in bodily form on earth. Nowadays, people are more likely to associate it with an icon or figure representing a particular person in a video game, internet forum, social media, etc. We use avatars to show our identity or assume other identities, and when it comes to identity, we want to ensure that we are expressing ourselves as we like. 

As normal as paying for a T-shirt

In the video game industry, it’s common to refer to 3D avatars as “skins”. If you are a gamer, paying for a skin is as normal as paying for a t-shirt. However, if you are not, it might sound unearthly. To put it into perspective, let’s take a look at Fortnite. This free-to-play game earned roughly 70% of its $5.1 billion annual revenue through selling skins. For example, Epic just generated $50 million ONLY from NFL skin set, and it was not even the bestseller. 

NFL game with avatars makes billion annual revenue

The market knows the value of self-expression and is capitalizing rapidly on it. This business model resembles life, and people unconsciously understand it better. Playing (-living) is free, but you pay for extra items like skins (-branded clothing) to better express yourself. 

Next steps into the Metaverse

With the advent of Metaverse and moving from 2D to 3D experiences, the need for better avatars is more pronounced; and the ones who provide the best solutions and mediums win this round. Let’s see why people are paying billions of dollars for these avatars and what can be the next step. 

Almost any F2P (Free-to-play) like Fortnite and Dota generates the majority of its revenue through selling these skins. Even games like World of Warcraft also offer skins that are not obtainable in-game. And how do they market it: “Collect, mix, and match your favorite transmogrification (read avatar skin) pieces to personalize a look that is uniquely you.” 

blogpost AA 2

It sounds like my closet, but for my digital self. 

 So, we are already at a level where we can buy clothes for our digital representation, and the freedom of the digital world allows us to be anything. In this round, I can play as Batman and in the next round as Harley Quinn. 

The range of avatars from expressive to photorealistic 

Before we go further, it’s necessary to take a step back and see the spectrum of avatars. On one end of this spectrum, you want to be exactly yourself, so it makes sense to target photorealism. On the other end of this spectrum, you want to be something that expresses you the best. Mark Zuckerburg beautifully pointed this out in this conversation with Lex Fridman. 

People who are not from this domain might assume that if you are completely represented like yourself, that’s also the best way to be as expressive as yourself. However, they forget that people go to the digital world “exactly” for that reason – to be beyond their physical appearance and experience things that are impossible in the physical world. 

The best fitting representation for every situation

Clearly, there are different use cases that favor different points on this spectrum. For example, if you are creating a VR experience for training and phobia therapy, it’s likely that photorealism would provide the users with better results. However, if you are making a VR game like Fortnite, then the expressiveness of the avatars is more likely to make your game a success. 

Focusing on the “expressive” side of the spectrum, the question is: Can our avatar change based on our emotional expression?

 If so, how? And is it useful? 

The scientific research behind affective avatars 

There is an extensive body of science that delves into this topic and tried to validate the effect of emotionally expressive avatars in virtual environments. In this section, we would only mention some notable ones and their key results. 

The scientific research behind affective avatars

MIT Media-Lab

Researchers at MIT Media-Lab created something called Emotional Beasts: Visually Expressing Emotions through Avatars in VR. This paper (PDF download) posits: 

“… We argue that the use of collaborative virtual 

environments which incorporate emotionally expressive 

avatars has the potential to engender expression of 

emotions amongst the users of such environments.” 

MIT Media-Lab

Leed Becket University

Marc Fabri from Leed Becket University writes in his thesis Emotionally Expressive Avatars for Collaborative Virtual Environments

“The findings suggest that emotional expressiveness in avatars increases involvement in the interaction between CVE (Collaborative Virtual Environments) users, as well as their sense of being together, or copresence. This has a positive effect on their subjective experience…” 

As mentioned in the beginning, we clearly don’t want to undermine a simulation (e.g. VR Phobiatherapy) by adding a feature that turns the user into a dragon when they are angry. However, the findings in this study clearly indicate a positive effect on their subjective experience. 

Sejong University

Taehoon Cho and his colleagues at Sejong University also tested something similar. However, instead of changing the body like MIT study, they focused on augmenting the facial expression. If we use the same notion of the aforementioned spectrum, this study lies somewhere closer to realism by mixing realistic facial features and facial painting.  

Each painting is associated with one emotion, namely: neutral, happy, angry, surprised, and sad.

Sejong University - audEERING neutral, happy, angry and sad

Basque Research

Basque Research also conducted a study called Avatars As Communicators of Emotions. In this Ph.D. thesis, Ms. Amalia Ortiz proposes that avatars are one of the best ways for computer systems to emit non-verbal information. Amalia created a VR system in which the users experience a journey through history. This application, aimed at e-learning showed the following results: 

“…Dr. Ortiz concluded that the users prefer interaction using avatars because they considered it to be more pleasant, user-friendly and entertaining. Moreover, she observed that users were capable of recognising most facial emotions or expressions and they believed that information to be better explained if an emotional avatar is providing it…” 

University of Copenhagen

On the other side of the globe at the University of Copenhagen, Aske Mottelson and Kasper Hornbaek from the Department of Psychology and Computer Science joined forces to study the interplay between affect and the ownership of a virtual body for emotional avatars. The results from this study show that some avatar manipulations can subtly influence affect. Facial manipulation emerged as most effective in this regard, particularly for positive affect. Also, body ownership showed a moderating influence on affect. Results indicate that body ownership varied with valence but not with arousal, and it varies with positive but not with negative affect 

Our brain reacts to emotions in avatars

The question you might ask yourself is “why should we care about the emotions in avatars?”. Fortunately, affective neuroscientists and social-cognitive researchers were active to help us answer this question: 

How our brain responds to emotions in avatars as opposed to humans? 

The study posits that fearful expressions in human and avatar faces evoked similar responses in the amygdala. Amygdala is the region of the brain responsible for emotions. This can explain why we are touched so much by what happens to the characters in video games. 

The social impact of gaming

This pandemic taught us that things we take for granted can be taken from us in no time. The lack of social interactions has plunged many souls into the valley of depression during this pandemic. However, evidence shows that playing video games can ease loneliness during the coronavirus pandemic. Why? Because as much as your brain and amygdala is concerned, humans and their avatars are not that different. 

Thus the more natural these interactions and avatars, the more enjoyable these experiences can be. With the advent of the Metaverse and the promise of spending more time in the virtual worlds, we should ensure that such interactions are as enhanced as possible. Such enhancements can help us express our individuality better and in these expressions go beyond the limits of our physical world. 

Voice AI as the key factor

Here in audEERING we strive daily to ensure such use cases that are proven by science in the labs are not confined to articles and dreams. The key factor in expression is emotion. Without detecting the emotions of the players, we have no way to properly change their avatar to demonstrate their expression and show their individuality. 
Using entertAIn play, every developer for VR and Metaverse can use the microphone on any VR Headset to detect the emotions of their players. It works seamlessly on the device without any need to record and send the audio data to an external server, thereby, preserving the privacy of the players. 

Voice AI is the missing link in this long chain of expression in Metaverse, and we are proud to present our technology as a solution for this use case. We offer a multitude of AI models that can be used out-of-the-box or be calibrated for the highest accuracy. 

For more information, please check our entertAIn play product page.
Reach out to the entertAIn play team and request a trial version:

Sylvia Szylar | Customer Experience Manager |