Insights from a Remote Multimodal Study
Excitement doesn’t always mean people will buy. In advertising research, this distinction can be easy to miss. Traditional ad testing typically relies on surveys: participants watch an ad and then report whether they liked it or would consider buying the product. But emotional reactions often become visible in how people respond, through their voice, facial expressions, and other signals, not only in what they say.
A recent remote study conducted together with iMotions explored how multimodal expression analysis can complement traditional ad testing.
Looking Beyond Survey Answers
In the study, participants watched two video game advertisements: one for Grand Theft Auto and one for The Sims. After watching each ad, they answered four open questions: What did you like about the ad? What did you dislike about the ad? Would you play the game? Would you buy the game?
While participants responded, multiple signals were recorded via webcam and microphone:
- Voice expression features such as valence and activation
- Facial expressions measured as adaptive valence
- Respiration features such as respiration rate and cycle duration
- Spoken language analyzed through speech-to-text.
This multimodal setup allowed researchers to observe how participants expressed their reactions, not only what they reported.
Why a Baseline Matters
A key methodological step in the study was the use of an individual baseline. At the beginning of the session, participants read a short neutral text aloud. This allowed researchers to account for natural differences in how expressive people are. By comparing later responses to this baseline, the analysis focused on changes in emotional expression rather than individual speaking styles.
What the Data Revealed
The results showed several interesting patterns.
First, emotional expression aligned clearly with participants’ evaluations. When participants talked about what they liked about an ad, their voice and facial expressions tended to show more positive valence. When discussing dislikes, more negative expression patterns appeared.
Second, the analysis revealed measurable differences between the two advertisements. This shows how multimodal emotional data can support A/B testing by highlighting which creative version resonates more strongly with viewers.
Third, the study revealed an important distinction between different types of intentions. When participants discussed whether they would play the game, emotional activation tended to be higher. When discussing whether they would buy the game, responses were noticeably more neutral. This suggests that enthusiasm for an experience does not automatically translate into purchase intent.
Beyond Surveys: Adding an Emotional Layer
These findings highlight how expression-based analysis can complement survey methods.
Traditional surveys capture participants’ conscious evaluations. Expression analysis adds another layer by observing emotional signals that occur while people reflect on their experience. This approach can also be applied beyond advertising studies, for example in UX research, product and packaging evaluation, or entertainment content testing. In each case, emotional expression can help researchers understand not only what participants say, but how they react while saying it.
Remote Emotion Measurement at Scale
Another key insight from the study is methodological. All measurements were conducted remotely, using participants’ own webcams and microphones. With appropriate study design, including baseline normalization and quality checks, remote multimodal data collection proved to be a robust and scalable approach. This opens new possibilities for large-scale consumer research without requiring laboratory settings.
A Broader Way to Understand Audiences
The study demonstrates how combining voice analysis, facial expression analysis, respiration data, and sentiment analysis can deepen our understanding of consumer reactions. Beyond the specific findings, the report also uses this study as an example to illustrate methodological guidelines and potential analysis approaches that researchers can apply to their own data.
By adding an emotional layer to traditional surveys, researchers can identify patterns that might otherwise remain invisible, from subtle differences between ads to the gap between excitement and purchase intention. As remote research technologies continue to evolve, multimodal expression analysis is likely to become an increasingly important tool for understanding how people truly respond to content, products, and experiences.
