Test in-context for quality insights: Real vs. simulated feeds

In the world of digital advertising, there are dozens of tools that help marketers measure impact. However, not all measurements are equally useful. How do you reexamine advertising testing for the digital age and its distracted, advertising-avoidant consumers?

Firstly, we need to take a step back from the traditional testing routine: scrutinizing the ad creative through a focus group, displaying the standalone, full-length, full-size ad to a benevolent respondent and following it up with in-depth questions later. Why? Because this process, aside from a delayed and conscious ad assessment, entails forced exposure of the tested material to the consumer. A critical precondition to evaluating ads is checking if they manage to reach the consumer and get noticed in its natural environment, to begin with.

By using natural and unforced exposure of the stimuli, in an environment as close to reality as possible, we get an objective and accurate insight into the actual ad visibility. Studies done this way are the single most useful and authentic measurement nowadays and are becoming a standard go-to approach for brands and companies doing ad pre-testing. If you have a great ad that goes unnoticed when thrown into the highly competitive and crowded social media environment – you are wasting your money. The harsh truth is that the average consumer scrolls through their feed at a speed of around 1-3 posts per second, and the rule of thumb is an actual rule of the almighty scrolling thumb.

What type of context yields optimal results

However, it is not only a matter of testing ads in context, but also about what that context is. More precisely, is it as realistic as it can be? Which is better, testing it the way it would be perceived in real life – in the respondent’s own news feed, or in a natural, albeit simulated context?

Both approaches involve consumers being exposed to your ad in the digital clutter of images, posts, videos, and outside distractions, exactly like they do in real life. Only, depending on the particular situation and the task at hand, you can pre-test ads both by injecting the stimuli in the respondents’ actual social media feed or test them in a simulated feed.

Both approaches have a specific set of KPIs – they measure the visibility of the ad (viewability), time spent on the ad (attention). Additionally, each of them has its own advantages and drawbacks:

Ad insertion – Actual news feed

Ad insertion is a technique by which ads are placed into the real, live feeds or webpages, ensuring that the ad will be seen in a completely natural environment for the respondent. On the other side, a simulated feed is entirely created and controlled by the researcher, also allowing the insertion of the ads into desired places.

• PRO: Enables ad exposure in a completely natural environment – in the kind of setting that it will be seen in by the viewers, which encourages the respondents to act normal, be completely relaxed and thus gives more precise data on ad performance.
• PRO: It is possible to place the tested ad in between any two posts on the timeline, and in this way, measure not only its visibility, but vary the position and measure the influence of its place on visibility and attention.
• CON: It is impossible to control the influence the feeds themselves have on post visibility. Namely, every user’s feed is different, thereby introducing an area that is not regulated by the researchers. The surrounding posts may have a specific type of content (such as more videos, bright colors, engaging content) which can modify post visibility. That is why we can never be entirely sure what the effect of the timeline is and what is an impact of the creative direction of the ad.
• CON: Recruiting is potentially very difficult. This methodology requires an application download and logging into the personal social media profile with their own email and password. Although the researchers do not collect user data in any way, their potential reluctance to share their info is understandable. The respondents might feel uneasy despite the fact that no data is captured while they log in, the content manipulation is taking place inside our app, and is not visible or affecting their profile outside of the app.
• CON: The application of this methodology can have certain technical limitations and requirements since it is not accessed through a browser but as an app.
• CON: The question remains whether people will naturally browse faster after logging in and knowing that they are participating in a study, than with using mock-up websites and content.

CMS (Content management system) – Simulated news feed

• PRO: Enables an entirely experimental approach to research/study. As the ad is inserted into a pre-prepared feed, it ensures that the differences we could measure on KPIs between two or three ads or ad variations can genuinely be attributed to the creative direction of the advertisements, considering that all the other conditions are controlled.
• PRO: Because the researcher has full control of the environment in which the ad will be placed, it opens an exciting field for exploratory research and play. Not only can we measure the visibility of the advertisement in its surrounding, but also measure the effect and impact of the environment on visibility, leaving space for variations. For example, would a post promoting liberal political values have higher visibility in a similar (congruent) feed, or among conservative posts? The creative direction of the feed is vital, and another thing to keep in check is whether the context is natural enough for the respondents (e.g., free scroll or interaction). You should carefully curate the environment, with a randomized combination of posts, a congruence between the theme of the posts and the ad, include various formats (video, image, gif, text).
• PRO: CMS can be developed for any social network – FB, TW, or IG, while ad insertion has so far been tested on Facebook only.
• PRO: Similarly to the first approach, it is possible to place the tested ad on any spot in the feed, and in this way measure not only ad visibility but vary post position and measure the impact of the place and the order in which it shows up on the feed on visibility and attention.
• CON: A simulated feed is not a personal feed, which makes a full replication of the natural situation impossible, considering that in reality, the ad will be placed in many significantly different feeds and profiles.

However, is the respondents’ behavior genuinely different in a simulated vs. actual feed? For the purpose of this blog, we put this to the test and ran a mini-study to uncover whether there are differences, and in which direction they are skewed.

Namely, we tested two ads for mineral water – one that promotes individuality and being in touch with nature (it opens with light colors, with a close up shot of a face), while the other one puts a focus on family spirit and a sense of community (it shows a family in the opening sequence, in a slightly darker palette). Both ads were tested in real and simulated settings. The video ad was the sixth in order in the feed.

The main take away: browsing is slightly different, but the conclusions remain the same

• The results show that the users scroll faster while logged into their personal FB feed. In a simulated feed, it takes them around 22s to reach the tested post, while in the own feed, it took them around 20s. This difference might be due to the fact that they are already familiar with the content of their feed, so they automatically skip/skim through the posts of certain people/groups/concerning some topics.
• Consequently, the visibility of the ad in the personal feed is slightly higher, but the time spent on the ad is lower. In a simulated timeline, respondents spend more time exploring the posts, because they are aware they are part of an experiment, and it feels less natural.
• Nonetheless, both ways of measuring provided the same insight – the ad that promotes individuality and contact with nature performs better in both the personal and simulated feeds – a 10% higher visibility, and 0.5s longer attention.

Conclusion

Having in mind the difficulty of recruiting, and any concerns the potential respondents might have, we can safely say that testing in a CMS provides high-quality insights that have a minimal absolute deviation from the data in personal feeds, but yield the same results. If you want the most accurate assessment of the ad breakthrough, consider testing in the personal news feeds of your target audience. In that case, however, the costs will be multiple times higher, the time to complete the study will increase significantly, legal risks are much higher, and it is questionable whether the behavior will be much more natural given that they were asked to log in.

‍