Emotion Recognition Demystified: The Expert Guide

Ready to resonate with consumers beyond discounts and special offers? Understanding the emotional drivers behind consumer behavior can help embed your product in their hearts and minds for the long haul. 

Expressing emotions verbally poses challenges for everyone, with surveys often highlighting this difficulty most prominently. This renders surveys alone potentially misleading. Nonetheless, alternative research approaches offer solutions. In addition to asking the right questions, like in real-life interactions, we meticulously analyze facial expressions. This is why relying on Emotional Recognition as a research method becomes imperative, allowing us to generate both more accurate and more profound insights.

That is why we have compiled a list of the most frequently asked questions to assist you in successfully understanding and utilizing such a powerful driving force as emotions.

1. What are the advantages of using emotional recognition to test advertisements over traditional methods?

Emotional Recognition delves into non-verbal reactions, unveiling responses that respondents might not consciously acknowledge, articulate, or be inclined to disclose in traditional survey or focus group settings. Furthermore, it provides insight into the underlying "reason why" behind an ad's performance, beyond mere measurement, thereby enabling businesses to make better informed decisions.

2. What are the advantages of emotional recognition compared to neuroscientific methods?

Marketing research relies on both behavioral and neuroscientific methods to overcome the say-feel gap, as, oftentimes, respondents are not able or willing to verbalize their emotions regarding stimuli. Combining explicit methods like surveys and implicit ones like FC allows us to access both parts of "the dual process thinking," popularized by Daniel Kahnman as System 1 and System 2 thinking. 

While EEG, fMRI, and GSR provide deeper insights into neural and physiological processes that emotional recognition does not directly capture, emotional recognition complements these methods by focusing on the observable manifestations of emotions and behaviors. Still, certain facial expressions are often associated with specific physiological states, and by observing facial expressions, researchers can infer potential physiological reactions. For example, expressions of fear may coincide with increased heart rate and perspiration; on the other hand, a smile might indicate activation in brain regions associated with reward and positive emotions.

In summary, while emotional recognition doesn't directly capture physiological or neurological processes, it can indirectly provide valuable information about these processes by analyzing observable facial expressions. It offers insights into how individuals experience and express emotions, providing a behavioral lens through which to understand underlying physiological and neurological responses. emotional recognition has three main advantages, comparing to neuroscience methods:

1. It is far less intrusive - the need for controlled laboratory settings and the onsite equipment (like EEG Cap, Amplifier and Recording device for both EEG and GSR, GSR sensors, MRI scanner and other associated devices), can make neuroscientific methods considerably intrusive. They require direct physical contact with the participant's body, or its exposure to strong magnetic fields (as in fMRI). On the other hand, emotional recognition primarily relies on observing and analyzing facial expressions. It involves capturing video recordings of respondents’ faces while they view stimuli (or engage in tasks).

2. It is more scalable and cost-effective - the low intrusiveness leads to higher participant compliance and reduces the need for specialized training to administer the technique. FC can be done online, and it is applicable across the global market since basic expressions are universal to humankind. It can also be easily scaled to accommodate large sample sizes, as it does not require significant additional resources for each additional participant (e.g. limited availability of EEG and GSR scanners). With all that in mind, emotional recognition is relatively cost-effective and accessible, compared to neuroscientific methods.  

3. Its output is more readable - although EEG, GSR and fMRI offer baseline complementary information about physiological arousal, neural correlates, and cognitive processes underlying emotional experiences, FC measures more direct and observable indicators of emotional expressions, providing us with output with far better readability that can easily be integrated into research studies. 

3. What is the process for the coding of facial expressions?

Respondents are recruited remotely, and by following the testing link, with no need to download any app, they are directed to the testing page. After successfully being screened, they are then invited to provide their consent for facial coding. Subsequently, they are exposed to the stimulus being tested.

4. How high is the response rate for facial coding studies?

The response rate may differ depending on the market and other sample specifications, but when dealing with the general population, it is on average 18% ( Brazil 30%, USA 20%, Germany 15%, Japan 8%).

5. What are the specific emotions our facial coding can record?

Facial coding can be used to recognize a variety of facial expressions, each associated with specific emotions. The Facial Action Coding System (FACS), developed by Paul Ekman and Wallace V. Friesen, is one of the most widely used systems for coding facial expressions. FACS identifies specific muscle movements, called Action Units (AUs), that correspond to different emotions. EyeSee’s framework relies on FACS and Ekman’s seven basic emotions list, having it adapted to market research needs; by grouping three of the negative emotions (disgust, contempt, anger) into one (dislike), and expanding the list by two cognitive emotions, interest and confusion. In total, EyeSee’s framework is comprised of 7 emotional categories (3 positive and 4 negative): 

1. Happiness: Indicated by the activation of the zygomatic major muscle, causing the corners of the mouth to lift (AU12), and the orbicularis oculi muscle, leading to "crow's feet" around the eyes (AU6). 

2. Surprise: Displayed through raised eyebrows (AU1 and AU2), widened eyes (AU5), and optionally an open mouth.  

3. Interest: Manifested by lowered eyebrows (AU4 and AU7), 

4. Confusion: Characterized by a downward pull of the corners of the mouth (AU15) and the upward movement of the chin (The Chin Raise Action Unit AU17), optionally accompanied by an inner brow raising (AU1). 
   
5. Dislike: 
   a) Disgust: Shown by a wrinkled nose (AU9), raised upper lip (AU10), and a downturned mouth. 
   b) Contempt: Reflected in a slight one-sided mouth raise, often with the corner of the lip lifted (AU12). 
   c) Anger: Manifested by lowered eyebrows (AU4 and AU7), narrowed eyes (AU4), and a tense jaw (AU24). 
   
6. Fear: Evident by raised eyebrows (AU1 and AU2), wide-open eyes (AU5), and horizontally stretched mouth (AU27). 
   
7. Sadness: Characterized by a downward pull of the corners of the mouth (AU15) and the inner eyebrows raising (AU1). 

6. Is tracking a limited number of emotions sufficient?

The seven emotional categories allow for a generalized understanding of the emotional landscape without needing to delve into highly specific emotions, which can be more subjective and challenging to interpret consistently. However, while emotional categories offer a useful starting point, they may not capture the full complexity of individual experiences and nuanced emotions. In EyeSee, we open a window into the individual's experience and specific emotions by analyzing facial expressions within their specific contexts.

7. What specific metrics can be derived from the facial coding method?

Emotional Reach - the percentage of viewers who have had any emotional reaction to the stimulus. 

Emotional Engagement - the average of total reactions during the whole time of the stimulus. 

Happiness - the percentage of respondents who have expressed a smile at least once during the stimulus. 

Surprise - the percentage of respondents who have conveyed an expression of surprise at least once during the stimulus. 

Interest - the percentage of respondents who have conveyed an expression of interest at least once during the stimulus. 

Confusion - the percentage of respondents who have conveyed an expression of confusion at least once during the stimulus. 

Dislike - the percentage of respondents who conveyed an expression of dislike at least once during the stimulus. 

Fear - the percentage of respondents who have conveyed an expression of fear at least once during the stimulus. 

Sadness - the percentage of respondents who have conveyed an expression of sadness at least once during the stimulus. 

Positive emotions - the percentage of viewers who have had any positive emotional reaction to the stimulus.

Negative emotions - the percentage of viewers who have had any negative emotional reaction to the stimulus. 

8. How can we tell what triggered an emotion?

Facial coding second-by-second output for video stimuli is very informative on emotional triggers since it allows us to know what scene from the stimulus triggered an emotion. In cases where the scenes are more complex with multiple potential triggers - the facial coding data can be crossed with eye tracking data as well as with the survey response, allowing us to have a full context and thus be able to pinpoint the emotional triggers and convey an actionable insight. 

9. What devices and browsers can be used for Facial Coding testing?

Facial coding tests can be run on both desktop and mobile environments, with no need to download any app. Our Web Platform supports multiple browsers for mobile: Google Chrome, Mi, Samsung Internet, and majority of integrated browsers, covering 70% market share, and when it comes to desktop - Google Chrome, Opera, Edge, Firefox, Safari, and AVG Secure Browser, covering over 99% of the browsers available.

10. How are emotions connected to brand impact?

In EyeSee, we conducted research with 18 short ads (from 7-15 seconds) with the objective of finding out how facial coding correlates with some of the most relevant survey and retention KPIs (see the list visual below). The significance level taken into account was 0.05 (equivalent to 95% significance) and Spearman value correlation being above 0.3 and below -0.3*.  

All three facial coding KPIs were significantly correlative with Likability (on average 0.32) and Uniqueness (avg. 0.46). Besides the two mentioned correlations, the Happiness KPI is also positively correlative with Stopping power, ad sharing potential, and the statement “It makes me think that this brand is really different from its competitors.” As for the negative correlations (statistically speaking), respondents who expressed higher happiness tend to rate the stimulus as less ordinary (-0.42).   

*Two main types of correlations are positive correlation and negative correlation (0.3 for positive correlation and -0.3 for negative), and they describe the direction of the relationship between the variables, as two variables may move in the same or opposite direction.

11. How are emotions connected to purchase decisions?

Although rational appeal in advertising is very important, it is not appropriate for certain products, especially those that are not that different from the products of their competitors in functionality, olfactory, or technical characteristics. That is when emotional appeal, which consists of creating certain emotional associations in the consumer’s mind towards the advertised product, fosters great results in purchasing decisions (in those cases, a unique selling proposition is replaced by a unique emotional proposition).

The word emotion comes from Latin and is derived from the word emotio/emovere, which essentially means "to move out" or "to stir up." Therefore, emotions are connected to purchase decisions by creating a desire for the associated products (or a lack thereof). Unfortunately, this link is hard to measure since, unlike short-term promotions and discounts, its influence is more of a long-term nature as it is developed over time in the privacy of the audience's subconscious mind. However, what we can measure is brand impact.

In sum, it's convenient to view the act of purchasing as a "behavior" and desire as the "mental state" that drives it. In this context, facial coding provides insight into the underlying state of mind, which is crucial for guiding desired behavior, such as making a purchase.

12. How do we benchmark our facial coding KPIs?

When assessing an ad’s performance on an emotional level, providing relevant facial coding benchmarks is a must since there are no universal truths when it comes to gauging a creative's performance. Stimuli of different lengths attain different levels of attention and allow for different emotional engagements to be developed, which is why the stimuli length is the first criterion to be applied. Additionally, it's important to consider the creative approach employed in creating the stimulus, such as whether it adopts a collage or a movielike style (collage or movielike). And although a global benchmark is a good fit in some cases, for some markets it is best to apply a culture-specific benchmark (i. e. for a study conducted in Japan, an “Asian benchmark” would be the best fit, since this market is characterized by more subtle expressions).

Apart from culture-specific benchmarks, many of our clients are interested in testing the animatics version of ads, in order to fine-tune them in the earlier production stage or select the winning ad, and this is where our animatics benchmark is applied. Furthermore, taking into account the industry adds another layer of benchmark specification, as emotional profiles and levels of engagement can vary significantly within different sectors.

In conclusion, it is essential to consider both sample and stimuli specifications when conducting significance testing, and an extensive benchmark base allows EyeSee’s research team to do that, thus reaching reliable conclusions.

13. Is the presence of negative emotions by default a bad thing, and should they be avoided?

Depending on the ad’s narrative, negative emotions like disgust, contempt, sadness, and fear can be a part of creative objectives. While it may seem counterintuitive to intentionally trigger these emotions, they can be powerful drivers of action, especially if your product or service offers a solution to them. This is why valence-based approaches, which limit themselves to positive and negative emotion-oriented KPIs, are to be taken with reserve; they are less explanatory when used alone without emotional profiles since they do not take into account that emotions of the same valence differ in essential ways.

14. What if some parts of an ad elicit no emotions?

When using facial coding to measure emotional engagement in any dynamic content, it's not necessary or realistic to expect that every single moment will elicit an emotional response. The goal is rather to identify key moments or elements that successfully engage the audience and evoke the desired emotions. Neutral respondents can also be engaged (with neutral facial expression), but they can still be fully locked in; this is where facial coding is complemented by retention. 

15. Dislike often appears at the end of an ad, simultaneously with the brand logo - how should this be interpreted?

Dislike at the end of an ad is probably a reaction to the attitude of ‘just another ad’, not necessarily to the brand itself. Still, it might indicate that the creative failed to come across as entertaining/educational content by being too obvious or tactless in revealing the brand. Surveys can be a good add-on by providing another layer of confirmation. 

16. Can facial coding be regarded as confirmation of attention?

Facial expressions do confirm attention, but the absence of facial expressions does not necessarily indicate a lack of attention. 

The relationship between attention and emotion is unidirectional (one-way): if respondents are emotionally engaged with the stimulus, it indeed means that the stimulus has their attention – however, if respondents are not emotionally engaged with the stimulus, it does not preclude them from paying attention. Attention and emotional engagement represent distinct yet complementary responses to stimuli, both of which we assess separately using the Retention and Facial Coding methods. 

However, facial coding as a method brings much more into the picture than merely confirming attention: it delves into second-by-second emotional engagement, capturing the highs and lows of emotional responses and identifying specific emotions over time. This detailed analysis offers valuable insights into how tested stimuli can be adjusted to evoke intended emotional states. In essence, facial coding and retention measurement assess two distinct qualities of stimuli and should be regarded as distinct yet complementary tools for understanding audience response.

17. What is the ultimate benefit of utilizing facial coding?

Besides being both the least intrusive and the most scalable method, facial coding serves as a window to honest feedback on emotional engagement, thus allowing creative teams to answer the following questions: 

1. Does the creative resonate with the target audience on an emotional level? 

2. Is the content creating the intended emotional state? 

3. Which are the most engaging parts of the stimulus? 

4. Where are the emotional peaks? 

5. Based on 1-4, what are the learnings, and how can they be leveraged to design narratives that bring brands closer to their audience? 

Curious to know more about Facial Coding and Social Media testing, read more here.