Journal of Marketing Analytics

https://doi.org/10.1057/s41270-023-00255-1

ORIGINAL ARTICLE

AI technologies in the analysis of visual advertising messages: survey

and application
Larisa Sharakhina1 · Irina Ilyina1 · Dmitrii Kaplun2 · Tatiana Teor1 · Valeria Kulibanova3,4

Revised: 31 January 2023 / Accepted: 26 August 2023

© The Author(s), under exclusive licence to Springer Nature Limited 2023

Abstract
Artificial intelligence technologies are improving the marketing toolkit, making it possible to process large amounts of data
faster and more efficiently than ever before. Machine learning, a subset of AI, uses algorithms that can predict which ads
will be most effective in specific situations, allowing for optimized ad targeting. This research explores the issues of coevo-
lution and distribution of machine and human intelligence in various social practices, including marketing and advertising.
The authors describe the key approaches to studying the visual component of advertising and suggest revising traditional
methods of analyzing advertising messages. The tracking of biometric data combined with AI-based methods that capture
human emotions while viewing video content is proposed as a promising direction for such analysis. This paper presents the
results of a pilot study based on analytical face-tracking technology using AI, where the subject of the experiment was the
analysis of video fragments that may have an impact on the emotional state of the viewer. The AI software platform used
was Amazon Rekognition, and the results show that AI analytics provide the ability to track the level of audience engage-
ment in perceiving video content, which helps to improve communication effectiveness. This allows the use of AI to make
recommendations for the development of more directed and engaging advertising messages.

Keywords AI-based video analysis · Emotions tracking · Advertising effectiveness · AI analytical tools · Video content ·
Emotions

Introduction for marketers is not only to get the attention of the target
audience for an advertised product, but to do so effectively.
Today's consumers are bombarded with marketing messages, One effect as a result of our adaptation to informa-
and information overload is continually increasing over tion overload is the phenomenon known as banner and
time, and as a result, the human mind adapts to minimize ad blindness that has long attracted the attention of social
the overload of information. Therefore, the real challenge scientists. From the end of the twentieth century to the
present, many papers have been published on how to try
to avoid this phenomenon (Benway 1998; Burke, et al.
2005; Owens et al. 2011; Pernice 2018; Ning, et al. 2023).
* Tatiana Teor Although a solution to this problem has been sought for
[email protected] more than 25 years, we still don’t have a clear answer. It's
1
Department of Public Relations, Saint Petersburg clear that the result of this “blindness” causes the effec-
Electrotechnical University “LETI”, Saint Petersburg, Russia tiveness of advertising messages to decline, and advertis-
2
Department of Automation and Control Processes, ers are then faced with a lack of desired outcomes. So far,
Saint Petersburg Electrotechnical University “LETI”, there is no definitive answer to the question of what needs
Saint Petersburg, Russia to be done to reduce ad blindness and improve the results
3
Department of Tourism and Hospitality Management, of the impact on the target audience, but ongoing research
RANEPA St. Petersburg (The Branch of the Presidential is being conducted in several fields. Some researchers have
Academy of National Economy and Public Administration), studied the impact of creativity on ad effectiveness (Yang
Saint Petersburg, Russia
et al. 2021; Abedi and Koslow 2022), while others focus
4
Department of International Business, St. Petersburg State on the impact of an emotional component in reducing
University of Economics, Saint Petersburg, Russia

Vol.:(0123456789)
 L. Sharakhina et al.

banner blindness (Provotorov 2019; Sapronov and Gor- 2012). Visual culture, focused on images, has become part
bunova 2021). Another area of research is trying to find of the everyday environment of our modern society.
the best banner positioning to attract consumer attention Thanks to digital technologies, we can observe that
(Cantoni et al. 2013; Çiçek et al. 2018; Muñoz-Leiva et al. our perception of the world is more visualized, and often
2021). mediated by gadgets with “screens of different sizes with
Without diminishing the importance of the approaches incrementally superior resolutions at every device upgrade”
outlined above, we believe that one of the most promising (Shazna 2013). The volume of data to be analyzed is con-
avenues could be the use of big data, machine learning, and stantly increasing, and according to the IDC report, “the
artificial intelligence algorithms in advertising and market- Global Datasphere will grow from 33 Zettabytes (ZB) in
ing. According to IBM's Global AI Adoption Index 2022 2018 to 175 ZB by 2025” (Reinsel et al. 2018). Taking
research publication, 37% of companies say that they are into account that human brains perceive visual information
using AI to drive more efficient business processes and oper- more effectively than text and combining it with the grow-
ations and 26% are using AI for marketing and sales solu- ing volumes of data, we come to a flourishing world of data
tions (IBM 2022). The use of AI analytics can help define visualization and video content. In other words, emotional
an effective strategy to improve return on investment and processes that arise as a result of video content perception
reduce losses, and AI tools in marketing can help to optimize influence the conscious perception of the surrounding world
a marketing budget, reduce the costs of customer acquisi- and determine the human behavior.
tion, automate routine tasks, and identify other efficiency Therefore, it leads to a specific situation. First, we have
opportunities. the previously mentioned information overload of mod-
The main applications of AI in different areas of market- ern people's lives, and second, there are too many differ-
ing have been described in a significant number of papers. ent goods and services on the market. These factors make
Marketers are increasingly using AI not only to quickly pro- advertisers strive to create advertisements that could be
cess large amounts of data from social media, email, and the noticed in the supersaturated media scene, which requires
web (Rekha et al. 2016; Shah and Shay 2019, Khrais 2020; providing much more relevant, precise, information about
Kumar 2020), but also to understand consumer behavior, the advantages of a certain brand. This situation actualizes
and develop customized offers (Bader and Kaiser 2019; the active use of graphic and video content in advertising as
Paschen et al. 2019; Paschen et al. 2020; Loureiro et al. a quick and effective way to influence, not the rational, but
2021; Enholm et al. 2022; Pedersen and Duin 2022). AI is the emotional perception of the proposed product.
also being actively used in advertising, allowing advertisers This emotional approach to purchases is supported by the
to identify patterns in audience behavior to make the right constant collecting and systemizing of user-data as a result
decision about what content a particular segment of potential of the evolution of computer technologies. Big data analysis
customers prefers to see and when. Additionally, the use of identifies target advertising audiences for establishment of
AI and location data for targeting advertisements greatly effective and “smart” communication channels. Unconscious
facilitates the creation of personalized offers. In general, AI and emotional consumer behavior has stimulated research
can currently be used for a variety of ways to attract the on the role of emotions in decision-making and in assessing
right audience, and through machine learning it will likely possible outcomes for businesses. Some of the most inter-
identify new ways in future (Boz 2018; Kietzmann and Pitt esting studies, dedicated to the analysis of the relationships
2020; Alawaad 2021; Li, et al. 2021; Zhao and Cai 2022). between consumers and their social and physical environ-
Particularly noteworthy is the use of AI for visual impact ment, as well as the interpretation of these relationships, can
studies. Recent scientific achievements in the field of cog- be identified as research by Achar et al. (2016), Kugler et al.
nitive, neuro-, psycholinguistics have made it possible to (2012), Small and Verrochi (2009).
determine the place and role of a visual image in the human We believe that one of the most promising applications of
cognitive process. A human’s brain processes visual infor- AI is in analyzing the communication effectiveness of adver-
mation 60,000 times faster than text, and 90% of information tising messages. Our vision of rapidly developing AI-based
perceived by our brain is visual. Of all sensory receptors in analytical instruments for advertising content in commercial
our body, 70% are in the eyes, and 50% of our entire neural application and the lack of open-source scholarly reflection
system is used for processing visual information. It's also inspired us to set up a pilot experiment to illustrate the mod-
been found that 40% of people react better to visual informa- eling of emotional involvement, through the engagement and
tion than to text (Eisenberg 2014). Recently, the concept of perception of advertising video content. In this study, we aim
“visualization” has become to mean a graphic representation to analyze the application of AI to evaluate the effectiveness
of information rather than a description of the act of creating of visual advertising content. The hypothesis of our study
a mental image and it implies the skill of visual literacy, i.e., is that AI technologies based on big data analysis can pro-
the ability to construct meaning from visual images (Ware vide recommendations for the development of advertising
AI technologies in the analysis of visual advertising messages: survey and application

content, thereby improving the communication effectiveness individuals who either consciously or unconsciously choose
of advertising messages. “behavior consistent with rational and informed maximiza-
The scientific questions that this research aims to answer tion of returns” (Friedman 2007).
were formulated as follows: However, modern behavioral economics argues that eco-
nomic actors tend to act irrationally, giving in to emotions
(1) To review and systemize the key approaches to adver- or moods, not analyzing all possible options, not consider-
tising visual content analysis (traditional and digital) ing risks, and not pursuing self-interested goals. As early
within the frame of advertising visual communication as the 1950s, Katona began to question the very existence
observation; of “rational” economic behavior: “unlike pure theorists, we
(2) To implement empirical pilot study of AI-based biom- shall not assume at the outset that rational behavior exists
etric face-tracking analytical technology. The experi- or that rational behavior constitutes the topic of economic
ment hypothesis: advertising video has a potential to analysis. We shall study economic behavior as we find it. In
be introduced effectively to a broader target audience describing and classifying different reactions, as well as the
by reducing the length of moments with the low-level circumstances that elicit them, we shall raise the question
emotional engagement of the basic target audience; whether and in what sense certain reactions may be called
(3) To raise a question of machine and human intelligence “rational.” After having answered that question and thus
coevolution and allocation in different social practices, defined our terms, we shall study the fundamental problem:
including marketing and advertising. Under what conditions do more and under what conditions
do less rational forms of behavior occur?” (Katona 1951).
The remainder of this paper is organized as follows. In The behavioral approach was further developed in the
“Current approaches to video content analysis” section, we works of Tversky and Kahneman. Reflecting on the pos-
arrange the key approaches to video content analysis within sibility of “rational choice,” the future Nobel laureates cel-
the frame of advertising visual communication observation; ebrated that “deviations of actual behavior from the norma-
customary methods of advertising messages analysis; inno- tive model are too widespread to be ignored, too systematic
vative and prospective methods based on computer tech- to be dismissed as random error, and too fundamental to be
nologies and artificial intelligence (AI). In “Methodology” accommodated by relaxing the normative system” (Tver-
and “Results” sections, we present the results of AI-based sky and Kahneman 1986). Scientists have suggested that a
biometric face-tracking analytical technology that captures person’s decisions are often driven by social, cognitive, and
the mood swings of viewers while watching the, “What does emotional factors rather than rational stimuli.
it mean to be a real man?” advertising video. The experiment From the point of view of the rational paradigm for con-
hypothesis: advertising video has a potential to be intro- sumer behavior, a person first thinks, then acts, and then
duced effectively to a broader target audience by reducing experiences negative or positive reactions and feelings,
the length of moments with the low-level emotional engage- depending on the satisfaction with the action. But Thaler
ment of the basic target audience. In “Discussion” section, rightly points out that “the orthodox economic model of con-
we discuss machine and human intelligence coevolution sumer behavior is, in essence, a model of robot-like experts.
and allocation in different social practices. Finally, in “Con- As such, it does a poor job of predicting the behavior of
clusions” section, we present the conclusions and discuss the average consumer. This is not because the average con-
the experiment results. Future research directions are also sumer is dumb, but rather that he does not spend all of his
highlighted. time thinking about how to make decisions” (Thaler 1980).
According to the behavioral concept, reality is much more
complicated and does not fit the classical model. Thus, while
Current approaches to video content the rational choice of Homo economicus can be analyzed
analysis and predicted, the choice of Homo sapiens is made under
the influence of much more complex factors than what the
One of the basic tenets of traditional economic theory is that mathematical calculation of maximum utility can take into
all people are rational. From the perspective of classical and account (Thaler 2000), and in this process, emotions are one
neoclassical economics, when Homo economicus makes a of the determining factors in the decision-making process.
decision, he is primarily concerned with maximizing utility During an experiment, Libet proved that brain activity
and minimizing loss, while clearly seeing all possible sets significantly anticipates the realization of a decision, and
of alternatives and being able to mathematically calculate it is possible to predict a person's decision long before the
the consequences of his actions. (Smith 2008; Samuelson moment of realization: “the brain evidently 'decides' to initi-
1938; Savage 1954). As a result, the greatest success in the ate, or at the very least, prepare to initiate the act at a time
struggle for survival in the market will accompany those before there is any reportable subjective awareness that such
 L. Sharakhina et al.

a decision has taken place. It is concluded that cerebral ini- video information, and the issue of video advertising per-
tiation even of a spontaneous voluntary act, of the kind stud- ception as a polymodal and heterogeneous construct and its
ied here, can and usually does begin unconsciously” (Libet semantic element perception influence the rising importance
et al. 1983). This situation is quite problematic for marketers of advertising video content research. Roland Barthes’s vis-
trying to understand why and when purchase decisions are ual advertisement studies in the 1960s “has been an enduring
made, because the consumer often cannot accurately under- source of insight in interpreting the visual in advertisements,
stand and explain why the purchase was made. As a result, but until recently little progress has been made in developing
field surveys, especially polls, which have traditionally been new frameworks to study the current diversity of advertising
the focus of consumer choice analysis, are raising more and formats in terms of their multimodality and the interaction
more questions among researchers. Wilson and Joye point of different modes of communication” (Smith 2008) and it
to response bias in surveys, which occurs when participants remains an actual issue. Moreover, the perception of video
systematically answer survey or interview questions from a content on TV and the perception of similar content using
particular perspective (Wilson and Joye 2020). While on the modern technical devices which act as mediators of interac-
other hand, new experimental methods are likely to become tion with the digital environment and artificial intelligence
increasingly important in the study of consumer behavior differ.
(Etzioni 2011). Due to information overload, availability of digital tools
As a result, many previously conducted marketing studies advertisement analysis on Internet platforms is limited to
that were aimed at identifying the perception of a product basic metrics such as impressions number (the number of
or service and tried to develop recommendations to improve times ads were shown), Click-through rate (CTR), etc. It
communication are questionable. These concerns have arisen is not enough to measure the advertising efficiency. These
not because of the quality of the research, but because of a metrics don’t take into account the users’ behavior, their
change in our perspective on the consumer behavior model. attitude to brands and impact on sales. Until 2015, Inter-
In our current research, we draw attention to the need to net video content used TV commercials without adapting
introduce a change to the way we study and assess percep- them to Internet channels. The main goal was mostly image
tual experience, as digital technologies enable new ways to building by fast moving consumer goods or FMCG. So,
assess the impact of video content on human consciousness. there was no demand for deep analytics (Statista 2023). But
since then, this advertising media scene has entered perfor-
Key approaches to video content analysis mance marketing-oriented companies that measure return on
within the frame of advertising visual investment (ROI). They faced difficulties in determining the
communication observation effectiveness of video content, because unlike Google Ads,
Yahoo! Search Marketing, Yandex Direct’s, and Facebook
Studies of video content advertising discourse can be viewed Ads analytics, which take into account short-term results, the
from the perspective of different research paradigms. On one effect of video content can be delayed, but long term. There-
hand, the determining factors for using image visualization fore, the effectiveness of video content should be considered
are pragmatic and correspond to the purposes of advertis- for different time periods.
ing communication. Lee and Lee's research revealed five As a result of the pandemic COVID-19, 91% of market-
basic consumer motivations for watching online video ads— ers think that video is now more important for brands than
“social interaction, relaxation, information, escapism-pass ever. In 2023, ad spending in the TV & Video Advertis-
time, and entertainment” (Lee and Lee 2012). By identifying ing market is projected to reach US$326.20bn in 2023. The
these motivations, advertising professionals tend to under- largest market is Digital Video Advertising with a market
stand the choice of a consumer’s interaction channels and to volume of US$176.60bn in 2023. In global comparison,
assess the effectiveness of those advertising tools (Krishnan most ad spending will be generated in the United States
and Sitaraman 2013; Pashkevich et al. 2012). On the other (US$138.00bn in 2023). The average ad spending per user in
hand, content visualization is one of the most universal ways the Digital Video Advertising market is projected to amount
of transferring messages. Thus, video content represents a to US$33.13 in 2023. The number of TV viewers is expected
complex system of communication units attributed to figu- to amount to 5.69bn users by 2027 (Statista 2023). In other
rative and metaphorical language used to transmit cultural words, video advertising costs are growing as a result of high
models, behavioral, and mental patterns. It makes a study of demand. The process of presenting the consumer's loyalty
methods and the role of social reality construction, by means to advertised products and the process of making decisions
of video content, an issue of high importance. on desired action depend on the repetition of certain desires
The perception of video content is fundamentally differ- at certain moments.
ent from the perception of text or audio, since more than one The issue is directly related to emotion recognition,
sensory modality system is used in processing the observed which a consumer can experience while perceiving video
AI technologies in the analysis of visual advertising messages: survey and application

content advertisement. And for the present time, the prag- advertising messages. Researchers connect the influence
matic approach to mass media audience impact is consid- of emotional components of advertising communication
ered a basic one. But it is also important to consider video with consumer's behavior. A number of researchers note
as an artwork with its means of expression as well as a set the importance of “outlines the relationships between
of promotional metrics (Stepanycheva 2011). In advertising multi-sensory congruent cues (visual, auditory and
visual communication these elements are the unity of signi- olfactory), emotions through valence and arousal as well
fied (designatum, significatum, concept) and significant (sig- purchase behavior” (Helmefalk and Hultén 2017), while
nifier, denotation) and it causes the importance of esthetic “ad-created affect can be transferred to brand attitudes
and ethical approach to video content analysis. Thus, we and purchase intentions, and there are cogent reasons
are able to outline an esthetic approach to advertising video to predict that ad creativity will trigger positive affect”
content analysis. (Yang and Smith 2009).
Pictorial/iconic/visual twist in communications is a trend
which has changed the process and mechanisms of human In general, all approaches to visual advertising content
socialization, caused new phenomena of human existence, analysis can be divided into traditional and innovative. The
and with them the crisis of established forms of social rela- main difference between them is the level of digital technol-
tions (Mitchell 1995). And we can outline mythodesign as ogies, including those AI-based, involved, and a projective
a kind of social and cultural technology when activities are approach to evaluating the empiric material.
carried out on the basis of a specific esthetic method of inte- Some difficulties in assessing the quality of advertising
gral comprehension and mass communication formation in occur before the campaign begins. As a rule, this is the most
the media-field paradigm. important stage of work, because it is not only about the
A review of the main approaches to visual advertising impact of advertising, but also about the effective spend-
content analysis allows us to highlight certain trends: ing of advertising budgets. Table 1 shows the most com-
mon methods for analyzing video content during the test-
(1) research in consumer’s behavior changes, related to an ing and pre-testing stage. It also lists their advantages and
increase of the visual communication impact; disadvantages.
(2) actualization of tools for measuring unconscious human Based on the overview, we can see that the traditional
reactions; methods of advertising impact analysis are characterized
(3) a shift to methods application analysis due to the by the engagement of audience members in the analysis
increasing complexity of advertising discourse. process; with most of the methods, they are characterized
by their focus on quantitative or qualitative results (such as
The reasonably reliable indicator of the effectiveness of consumer attitude to a particular advertisement, personal
advertising during a marketing campaign can be measured opinion about the content of the advertisement, recognition
by the change in demand for the product. But beyond the level of the advertising video message, etc.).
obvious effect of increasing sales through video advertising,
we can distinguish the following components of advertising Customary methods of advertising message
impact: analysis

• The communication component of advertising discourse Focused interviews, focus groups, expert interviews, ques-
as an element of the process of interaction with the audi- tionnaires, etc. are used as traditional methods to determine
ence in the form of attracting attention, understanding the audience's attitude to visual content and help identify
and memorability. In the works of Boerman, Kruike- various aspects related to the emotional and communicative
meier, and Zuiderveen Borgesius noted that the company effectiveness of advertising. A number of researchers have
“willing to spend money on advertisements to increase studied advertising video content through focus groups to
brand awareness, attitude and intention will strive to meet determine the reactions of a specific age group to the percep-
informational goals measured by the number of impres- tion of video advertising (Gurunathan and Lakshmi 2023;
sions” (Boerman et al. 2017), and advertisers often put Ylänne and Williams 2009), the attitudes toward brands
communication goals of advertising above behavioral formed by video content (Baliun et al. 2021), the effect of
(Murillo-Zegarra et al. 2020), since “the more customers video length on content perception (Zhao 2023), and the
learn about a brand—the more information they have— identification of social problems with a view to possible
the more likely they will trust, buy and remain loyal to adjustment of attitudes toward these phenomena (Adnani
that company’s products and services” (Truong 2023). et al. 2021).
• The emotional component of advertising creativity The significant advantage of focus groups is that
depends on the attitude to the brand, its products and when people discuss together, they provide more diverse
Table 1  Characteristics of common video content research methods
Benefits Disadvantages

Traditional Methods Focus groups The method reveals social motives, stereotypes, attitudes, The time-consuming nature of the research;
values and norms in relation to the object of study Questionable final results;
In-depth interviews Aimed at studying the target audience to collect "subjec- Difficulty in creating an objective picture of perceptions;
tive information about how they perceive various prac-
tices or products of those practices" (Morgan, 1996)
Brainstorming sessions consumers and advertisers participate together in a free-
flowing discussion on the perception of video advertising
Formal method In-depth study of verbal material (emotions and dyna- User perception of video is not considered:
Epistolary methods (essays, blogs, diaries, etc.) mism), factors organizing verbal sequence (theme The consumer perception aspect of video Content is not
and emotion), plot, plot and composition of video text taken into account;
without emotional or temporary pressure on the opinion Is evaluated the formation of a set of values, rather than the
of respondents when viewing video materials; actual relationship to the video content
Difficulties in obtaining objective data respondents' lack of
competence in describing emotional reactions to video
clips;
Difficulty in systematizing and categorizing results
Semantic differential method Ability to accurately identify consumers' latent motives in The time-consuming nature of conducting a survey and
choosing goods or services influenced by video content processing the results
Innovative Methods End-to-end analytics method and call tracking The ability to assess the impact of each marketing cam- Time-consuming
paign on the bottom line COSTLY research, especially if participants are in different
geographical locations
Capturing circumstantial evidence (eye tracker, Objective and accurate data on natural human reactions to Labor intensity of the study;
galvanometer (electrical activity of the skin, the content used Complexity of processing;
skin-galvanic reaction, polygraph) Technically complex procedures;
Immediate brain responses (electroencephalog- Require specialized equipment combined with sophisticated
raphy, magnetoencephalography, functional data analysis techniques
MRI, TMS Analysis ToolBox) The need for outsourced expertise and the use of technology
L. Sharakhina et al.
AI technologies in the analysis of visual advertising messages: survey and application

information, gain deeper insight into the topic, and gener- shortcomings in applying to the study of modern media con-
ate ideas that would not come from talking to each other tent. Within the framework of these approaches, emphasis is
individually. The weaknesses of this method are the time placed on the study of verbal material (emotions and dyna-
required for the research and the influence of subjectivity mism), factors that organize the verbal series (theme and
in the interpretation of the results. In addition, informal dis- emotions), history, plot and composition of video text, but
cussion leaders may influence the assessments and views of the aspect of perception and influence on the consumer is
all focus group participants, which can make it difficult to not taken into account. In other words, these methods are
determine the strength of the impact of promotional incen- not aimed to study communication with consumers, but to
tives on consumer behavior in general. study the content itself and the ways of its formation. We
For example, Nevid criticized this method, arguing noted that the fundamental points of these lines of research
that “direct inquiries may be biased by social desirability became the basis for innovative types of research using arti-
responding; i.e., by tendencies to give socially acceptable ficial intelligence to analyze video content for different pur-
responses to direct questions” (Nevid 2010). In other words, poses. For example, research dedicated to studying the use
the individualization of respondents' opinions is minimized, of video content for educational purposes. (Chen and Sun
which makes it difficult to create an objective picture of 2012; Cao et al. 2022).
consumers' perception of advertising content, since “many We can suggest that one of the most accurate traditional
consumers may not be aware of their underlying attitudes methods of video content analysis is the method of con-
or even know their own product preferences” (Nevid 2010). structing semantic spaces. It represents a meta-language
The criticism of the method by some researchers is based for describing the meanings that determine the underlying
on the fact that direct qualitative methods allow to obtain perception of objects in a particular subject area (Petrenko
a range of opinions, assessments or statements existing in 1988; Sikevich 2016). The success of this method lies in the
the society and, presumably, having a near-zero degree of identification of underlying hidden motives of the consumer
distribution, that is, high validity can be achieved only in when choosing products or services under the influence of
highly specialized studies. Even with high pre-test results, video content. The semantic differential method was mainly
it is possible to achieve the effectiveness of communicating used to analyze advertising video content before video
with a potential consumer/buyer and get a high emotional advertising became widespread on the Internet, mainly due
response to the advertising of the addressee, a positive atti- to its ability for accurate targeting, user interaction and trans-
tude to advertising, but it will not mean that the subjects will parent analytics. This method is complex for the develop-
buy the product. ment and interpretation of the data, and its effectiveness is
Brainstorming is used as an alternative to the focus group low. It should be noted that in the last 5 years the research of
method. This approach implies the possibility of participat- video content by the method of semantic differential is not
ing in a free discussion about the perception of video adver- numerous (Baliun et al. 2021).
tising messages and consumers and advertisers. And the Expert analysis of visual advertising content can be quali-
advantages of the method are the possibility of developing tative or quantitative, but qualitative expert results are often
joint solutions interesting for both consumers and advertis- criticized by researchers as not trustworthy methodology.
ers. However, obtaining objective data may be difficult, as in Delphi method, “which is a technique that employs subject
the case of the focus group method. For example, the posi- matter experts to gain a consensus on an issue or phenom-
tive results of the application of this method are shown in a enon” (Branda et al. 2018), is an example of reliable quali-
study where it was decided that “to produce a video profile tative research design. For advertising content analysis, the
that can show the competitive advantage so that it can be expert's evaluation of the advertising product might be rel-
displayed in various digital marketing channels” (Wicaksono evant; open or closed discussions of specific issues by expert
et al. 2022). groups and further voting, or also free and non-formalized
There are some interesting facts about the use of brain- expert observations.
storming with web tools and video communication. The
study conducted by Pituxcoosuvarn et al. found that when Innovative methods based on computer
comparing the use of brainstorming in virtual reality (VR) technologies and artificial intelligence
and in face-to-face classes, “the results reveal that there is
no significant difference in the quality or quantity ideas” IT-development has formed a situation of digital commu-
(2021). Thus, this method may be relevant in adapting to nication processes penetration in our everyday lives and it
information technology and artificial intelligence in market- has greatly influenced the advertising market too. Growths
ing research. in video content in advertising have created a demand for
Dramatic analysis, epistolary methods (essays, blogs, dia- effective and less time-consuming analytical tools. Machine
ries, etc.), despite the depth of analysis, are also not without learning (ML) and Artificial intelligence (AI) technologies
 L. Sharakhina et al.

are able to collect consumer data and help to develop more (2) marketing metrics (what helps to evaluate brand and
effective communication products, because it has become advertising campaign perception);
“an easy task to understand the things like consumer (3) business metrics (what measure video advertising effec-
behavior, preferences, search history tracking, along with tiveness in relation with sales and conversions);
the implementation of Search Engine Optimization (SEO- (4) metrics in pattern recognition (what measures custom-
optimization), content marketing, and speech recognition” ers’ engagement and their emotional response to video
(Ray 2018). ML and AI technologies in advertising are usu- advertising).
ally combined in several basic spheres of use: sophisticated
search engines, recommendation engines, programmatic Media metrics also involve impressions (the number
advertising as “the automated process of buying and selling of times ads were shown), viewability, VTR, and unique
ad inventory through an exchange, connecting advertisers to reach. Marketing metrics are usually demonstrated by
publishers” (Faggella 2019), marketing forecasting, speech/ the results of the BrandLift and Brand Interest analysis.
text recognition (conversational commerce). BrandLift analysis is based on auditorium responses to
Until recent times, the video advertising development some questions with video advertisement demonstration
was restrained by insufficient broad options of analytics: for those who already watched it and those who did not.
widely adopted corporate performance management (CPM), The following indices became evident: recognition ratio,
view-through rate (VTR), that is an indicator that reflects advertising recall, brand/product preferences, and user
the degree of attractiveness of ads on web resources, click- acquisition intentions. Brand interest demonstrates the
through rate (CTR), and other coefficients. They don’t dem- search accession rate (Google + YouTube) among con-
onstrate its impact on consumer behavior. For example, the sumers who have seen and who have not seen the cam-
video was 70% run by a consumer, but it could be caused paign. The use of Yandex. Vzglyad tool is also useful for
by interest or by the absence of a 'skip' button or any other these objectives: this platform helps to develop and to poll
reason. And we cannot really understand the perception of for learning attitudes to video campaigns. Business met-
this video advertising and what impact it will have on our rics can be performed using the Google Ads Conversion
advertising goals and objectives. Tracking tool, promotional codes, assisted conversions,
Ekaterina Odintsova from Yandex company outlines 3 and other.
basic tools to measure real effectiveness of Internet video The 6 rules for successful advertising video were devel-
advertising (Statista 2023): oped based on BrandLift and other IT analytical tools appli-
cation (Statista 2023):
(1) Last-click—the consumer watched the video, clicked it
and made a conversion. (1) Accurate duration: the best advertising video to be
(2) Post-click—the consumer watched the video, clicked watched through is up to 10 s.
it, but made a conversion later. (2) Extensive brand presence: it should be more intensive
(3) Post-view—the consumer watched the video, didn’t in comparison to TV advertising; the logo during the
click it, but came back and made the acquisition later. entire video would be the best.
(3) Clear key message: it should be easy to perceive and
In the case of Yandex as a platform of video seeding, it support by voice and text messages, because lots of
is possible to have postponed post-view conversions up to users switch off the volume of their gadgets.
30–60% of acquisitions (Statista 2023). However, it depends (4) More vivid and eye-catching visuals, than on TV, but it
on the product advertised, because IT-technologies used shouldn't be shocking or obtrusive, because it may lead
for consuming content usually differ: it could be a search- to negative responses.
ing model (a desktop) and a streaming model (a mobile). (5) Call for action buttons should be built in online video
Depending on the product, the model of seeding would advertising to influence conversion.
change and influence the conversion rate. (6) A landing page, linked to a call for action button,
But it is necessary to remember that only a part of the should be well developed in terms of design, logic, and
whole growing video advertising market is targeted to direct usability with a good mobile version, because a cor-
acquisition, thus, the effectiveness of such communication rectly designed landing page is a key element leading
should be measured on the basis of its objectives. Digital to high conversion.
advertising professionals outline 4 basic groups of the effec-
tiveness metrics: Therefore, following these guidelines with a compre-
hensive execution, good creative approach, and compe-
(1) media metrics (coverage, frequency, VTR (view- tent output evaluation, digital video advertising will be a
through rate), clicks, and click-throughs); perfect communication tool. Pattern recognition metrics
AI technologies in the analysis of visual advertising messages: survey and application

use neuromarketing technology to identify the emotional The iMotions platform is a more expensive tool based
response to a visual product and improve its quality. on the coding system of facial expressions (FACS, Facial
As far as a human face is a complex signal system for Action Coding System). In this case, a person fixes a cer-
demonstrating the attitude to external and internal stimuli, tain set of facial muscles movements to the meaning cor-
it functions thanks to 57 autonomous muscles. The con- responding to the emotions displayed. Each unit of action is
traction of the facial muscles may be observed because associated with a single facial muscle or group of muscles.
it depends on the certain mental state of an individual All facial expressions are broken into their constituent units
and it happens reflexively. Studying facial expressions and of action, and these actions are encoded and read by the AI
reaction of facial muscles to information perceived is of system. The system's database then stores all of the facial
interest, because the examination of emotional reactions expressions classifications by anatomical features.
and the subsequent decoding demonstrates the respond- Even at a superficial comparison, differences in analyz-
ent's reaction to visual advertising content. Facial emotion ing the effectiveness of advertising messages by traditional
recognition, by contrast to eye-tracking, in advertising is or innovative methods (Table 2) are evident: a higher effect
mostly applied practical issue rather than advertising or in research, a more accurate evaluation, and a faster rate of
marketing scientific research object: “interactive tablets in processing the large dataset. We tried to confirm this conclu-
rideshare cars that employ facial emotion detection tech- sion in the experiment described in the next section.
nology” (Alfi 2021), “The artificial intelligence-enabled
system can recommend menu items based on a customer’s
estimated age and mood” at Beijing KFC (The Guardian Methodology
2017). The BBC measured “conscious and subconscious
emotional responses to content marketing campaigns on In the case of video content, “banner blindness” is a loss of
BBC.com from clients like HSBC, Dassault and Hainan user attention when viewing video content, mostly due to the
Island Tourism” and offers it to other clients (Davies information overload caused as a result of large-scale gen-
2016). The BBC used “the facial recognition software” eration of content by companies and individuals. In this con-
supported by “webcam tests with each participant activat- text, the video stream perceived by a user's mind is not trans-
ing the tech via their desktop. For the unconscious meas- formed into definite images but forms an unconscious and
urement, facial movements were recorded on a second-by- emotionally dependent behavior to the perceived content.
second basis and then divided into six possible emotions: From the point of view of planning and evaluating an
sadness, puzzlement, happiness, fear, rejection, and sur- advertising campaign using video content, it is important to
prise” (Davies 2016). understand whether advertising causes emotions and what
At the same time, facial emotion recognition is the they are. Note that the emotional experiences themselves and
topic of quite a high interest for researchers in computer the strength of these experiences cannot be considered as
vision, deep learning, and other computer scientists (Kim, criteria for the success of advertising impact. However, the
et al 2021; Ozdemir, et al. 2019; Mehendale 2020; Patlar measure of correspondence of the viewer’s experience when
Akbulut 2022; Canal, et al. 2022). Some professional stress viewing the advertisement to what the advertiser expects is
that “Facial emotion recognition is the process of analyz- important to understand.
ing facial expressions to identify or recognize emotions on Emotional engagement when viewing advertising video
a human face—disgust, joy, anger, surprise, fear, sadness, content is an important component of the stimulating
or other compound emotions—with image processing soft- effect on consumers. According to Wang et al., “avoidance
ware < … > Facial emotion recognition differs from facial
recognition in that it does not attempt to identify the indi-
Table 2  Comparative characteristics of advertising message analysis
vidual; it seeks to identify the emotion” (Alfi 2021). methods
Anyway, advertising video content effectiveness meas-
urement is related to image processing software. The main Charac- Audience The ability Accuracy The resource-
teristics of engage- to reach a in assessing intensive
difference between the analytics platform is the inclusion/ the benefits ment in the wide range the impact nature of the
non-inclusion of a person into the system operations and the of video research of audi- on the audi- study
platform principle itself. For example, Face ID, Microsoft advertising ences ence
Azure Face API, Google Cloud Vision API, and Amazon analysis
approaches
Rekognition Image, Facial recognition technology (FRT) are
the examples of platforms based on the facial recognition Traditional + ± ± +
system and biometric data, where the self-learning neural methods
network itself selects the parameters by which it compares Innovative + + + +
methods
the selected image with the platform database.
 L. Sharakhina et al.

behavior reduces the effect of advertising communication cognitive, and emotional responses to certain stimuli and
and causes advertising waste to enterprises. Therefore, it is how they manifest in facial expressions, movements, ges-
more meaningful to study the changes of audience's emo- tures and body responses. Biometric facial tracking allowed
tional behavior under coercion” (2022). It is possible to ana- us to record all mood swings as the respondents watched the
lyze the video according to the emotions experienced by the proposed video material.
viewers. This analysis will be more objective than a standard The experimental material for this study consists of pro-
focus group because some of the facial muscle reactions can- motional material presented in a video blog on the YouTube
not be controlled, just as it is impossible to remember all the channel of a Russian men’s magazine (https://w ​ ww.y​ outub​ e.​
reactions at any given time. com/​watch?v=​sS3C5-​5j1tg​&t=​3s, Malenkov Cocktail 26:
The evaluation of emotional engagement is being stud- “What does it mean to be a man”). The video fragment is
ied quite actively, and we think it would be useful to high- 6 min and 34 s long and consists of three parts. The first part
light two areas of research. From the first point of view, of the video (42 s long) features the integration of advertis-
the researchers Ekman and Izard talk about the discreteness ing for two brands (Gillette and Porsche), with an explana-
of emotions and their qualitative difference (Ekman 1993). tion provided by the presenter. The second part of the video
Izard proposes to consider 10 basic emotions (joy, sadness, (3 min 52 s long) is presented in a “talking head” format,
anger, surprise, disgust, contempt, fear, shame, guilt, inter- where a monologue is presented by the speaker, the chief
est), which cannot be described by the expression of com- editor, who gives his perspective on the real man. This seg-
mon traits for all emotions, since each basic emotion is ment includes provocative and contentious remarks from
experienced differently and affects physiological processes the speaker. The final Sect. (2′05ʺ) presents the advertising
in different ways (1991). In criticizing this approach, Pos- integration between Gillette and Porsche, comparing their
ner et al. (2005), argue that “affective researchers have also respective brands in order to draw a connection between
proposed that each basic emotion is associated with a char- masculinity, Gillette and Porsche. The promotional content
acteristic facial expression. If this were true, facial expres- of the video was targeted toward men.
sions would then provide overt criteria for classifying the The basic software platform for the experiment was
basic emotions because the basic emotions could be defined Amazon Rekognition (https://​docs.​aws.​amazon.​com/​rekog​
simply by the presence of a characteristic facial expres- nition/). This analyzes the perception of video content
sion.” Researchers have proposed a description of emotions through face-tracking (a biometric tool for neuromarket-
in terms of a set of traits that determine fundamentally dif- ing research) where AI is used to recognize emotions. The
ferent experiences and speak of “a neural basis for positive method used as the basis for the experiment allows the sys-
or negative affective temperaments” (Posner et al. 2005). tem to identify up to 100 faces recorded on video and to
Today, thanks to the advances in AI technology, profes- detail the coordinates of facial elements (eyes, nose, mouth,
sional communication and marketing experts are equipped etc.) and to differentiate additional optional elements, such
with sophisticated technical devices and high-quality soft- as glasses, beard and mustache. This technology is openly
ware to assess the perception of content by users. In the available and is being used by researchers to create data
process of data collection, the activity of various parts of maps that show the magnitude and direction of eye move-
the brain, breathing frequency, pulse, skin moisture, pupil ments, and “to visualize the spatial and temporal changes of
movements, etc. are measured. Electroencephalograms facial expressions associated with happy and sad emotions”
(EEG), magnetic resonance imaging (MRI), magnetic reso- (Saadon et al. 2022) that can be used for the development of
nance tomography (MRT), pupil tracking systems (so-called sensitive, non-invasive, contact-free tools for assessing emo-
eye trackers), etc. are used to record the studied parame- tional states (Saiprasad and Meenakshi 2022). A detailed
ters (Margariti et al. 2023; Białowąs and Szyszka 2019). overview of the specifics of Amazon Rekognition and how
Research on facial emotion recognition for deep neural net- it can be used are described in detail in the paper by Mishra
works and artificial intelligence-based technologies can help (2019).
“in the early detection of facial expression diseases as well Despite the availability of free technology, we created
as in the analysis of consumer behavior” (Ozdemir et al. a custom video capture panel for watching the video. The
2019). panel works in any browser and device except for Apple’s
We conducted a pilot study to test the hypothesis that iOS. The best quality of emotion recognition is achieved
video advertising content has the potential to be effective when recording video on a laptop’s camera. Therefore, it’s
with a wider audience by shortening the most emotionally recommended that respondents use laptops when taking the
engaging moments for key audiences. In our pilot project, test. We wrote the capture panel in Python 3.7, Javascript,
we wanted to find out which components of video adver- and Bootstrap. The video has a windowed view, and the
tising content affect emotional engagement when viewed respondent's facial recording starts automatically when the
by users. We looked at a person's unconscious sensory, video playback starts. That recording is then stored on the
AI technologies in the analysis of visual advertising messages: survey and application

local drive and automatically encoded before being sent between the ages of 19 and 25. The research was conducted
to the Amazon server. Amazon then provides a JSON file remotely, following a similar protocol to that of the origi-
which is processed on the local disk using a SQLite3 data- nal video study, and participant responses to the video con-
base. And then, the final file is created in CSV format to tent were captured using a video capture panel suitable for
enable easy reading and processing. streaming from a personal computer or a gaming console.
The experiment consisted of five stages: Again, the participant was provided with a link to both the
panel and the pre-recorded video clip. While viewing the
(1) Selection of study participants; video, the participant's face was captured by a webcam, and
(2) Demonstration of the video, detection of the emotional acceptance criteria was consistent with those used in the
response and analysis using AI tools; analysis of the original video, including good lighting and
(3) Creation of a second video based on the analysis of the proper centering of the face on the screen. The second video
original video; was analyzed with AI tools similar to the way we analyzed
(4) Demonstration of the mounted advertising video; the original video, and the results of the second and fourth
(5) Analysis of the results. stages were presented in graphs reflecting the group average
value of each indicator obtained using the Amazon analysis
In the first stage, 30 individuals were sought to represent tool at a certain point in time when respondents viewed the
a range of ages, genders, and social backgrounds. We were video.
able to gather 17 male and 13 female participants between
the ages of 23 and 34 to match this goal.
In the second stage, we presented video content to the Results
participants and then analyzed the results. The study was
conducted remotely using a video capture panel for stream- The primary reason for this study consisted of the identifi-
ing from a PC or console, where we sent the respondent cation of emotional engagement in specific types of video
a link to the panel and asked them to view the ad. While content viewed by people who don’t represent the target
viewing the video, we recorded the respondent's face using audience for the advertised products in the video. The study
a webcam. The primary requirements for recording a usable revealed which portions of the video (fragments) elicit nega-
facial capture were good lighting and having the face in the tive emotions (decrease joy, increase sadness or anger) and
center of the screen. further identified the cause of these changes by comparing
The AI tool identified six emotions, namely surprised, the results with minimal changes to the material through
happy, sad, angry, disgusted, and fear, in viewers when they video editing.
were watching the advertisement. It then recorded the prob- The AI algorithm identified several stimuli that prompt
ability of each emotion's manifestation in 0.5 s, and for each an active emotional response from the respondents and dis-
participant, a separate file was created containing tables of cerned which emotions were dominant. The usage of AI
the distribution of emotions, including timestamp columns. tools helped determine the general emotional response to
A timestamp represented a time span of 0.5 s, and the cat- specific events and stimuli in the analyzed advertising video.
egory “Calm” was assigned to timestamps where the algo- These responses were measured as a degree of expression
rithm could not detect any emotions. The analysis resulted of emotion to the video's content, which includes the dialog
in the generation of thirty JSON files, which were also saved and music featured in the video. The AI identified the adver-
in CSV format for ease of analyzing. The collected data were tising video as generally positive, and the total emotional
processed for each participant and an average was calcu- expression was 12%. It was clear that the dialog neither elic-
lated for each of the six emotions. To filter the demographic ited intense negative emotions, such as disgust or rage, nor
report, we created links between the questionnaires and the intense feelings such as sorrow or euphoria. Considering the
Amazon data integrated into the tables. low level of emotionality and rationality present in the video
In the third phase of the experiment, a second video material, where rationality represents how much the content
was created by modifying the original video. In the new “makes sense,” it can be said that the verbal component aims
video, the fragment that elicited the lowest level of emo- to strike an emotional chord with the viewer but lacks a logi-
tional response from the target audience was removed. The cal basis to prompt the desired action, i.e., clicking on the
purpose of deleting this fragment was to test the hypothesis link to profit (Refer to Table 3).
that reducing a portion of a video with limited emotional The AI algorithm highlighted the six primary emotions—
engagement might make the advertisement more attractive happy, surprised, sad, confusion, disgusted, and fear. The
to a wider audience. state of being “calm” is determined as a distinct state, where
The new video, which was uploaded to a test channel, was the algorithm did not identify any emotions. Emotional
viewed by a new group consisting of 30 female participants involvement is measurable by computing the difference
 L. Sharakhina et al.

Table 3  Evaluate the emotional expression to the dialog using AI Conversely, emotions of joy prevail in men (25.36%) com-
algorithm pared to the emotional response among women (10.62%).
Category Expression of emotion Familiar images that are understandable to the audience
in video ­contenta were referenced in the presented video material, which ena-
bled the identification of specific and measurable emotional
Rationality 4%
reactions to the viewed content.
Video content text characteristic Positive
The primary emotional response to the video was sur-
Type of emotion: Expression of ­emotionb
prise among the target audience of 17 males (Fig. 2). This
Happy 7%
is detailed at Point 1 in Fig. 4, where the initial behavioral
Sad 0%
indicators of surprise were observed at times ranging from
Surprised 9%
0′ 18" to 0′ 24" of the video. Also, in Fig. 4, Point 2 and
Fear 3%
8 depict the emergence of the surprise emotions related to
Confusion 0
when the Porsche car is showcased, while Points 3 to 7 cor-
Disgusted 0
respond with the audience's reaction to the contradictory
Total expression of e­ motionc 12%
statements made by the presenter in the second and third
a
The type of expression is to the type of dialog that was heard during portions of the video.
the video The second highest emotional response was to the use of
b
Expression of emotion—min. 0%, max. 100% words associated with happiness (Fig. 5). The target audi-
c
Total expression of emotions—0% = minimum expression of emo- ence reacted positively to the known content and showed the
tions, 100% = maximum intensity of emotions
“happy” emotion to the targeted exaggeration (“girl 56 years
old,” “it’s not too late to change your mind” (be a man),
between the “calm” state and a value of 100%. The emo- “negative balance on all accounts”). And finally, words that
tional expression measured in the second and third blocks of signal fear, and trigger concern (for example, “crisis”) were
the video, as presented in Fig. 1, depict a rise in emotional placed third (Fig. 6). The peaks of emotions and their cor-
engagement that is detailed in the distribution of emotional relation with the timing of the video were analyzed, and a
reactions from the stimulus among 17 men, represented in pattern was found between the events in the frame and the
Fig. 2, and 13 women, represented in Fig. 3. The two vertical audience's reaction to them. The target audience's positive
lines on the graphs separate the video's advertising blocks. emotions decreased on average every 4 s due to the abun-
The reactions of men and women to the same advertis- dance of behaviors of the “real man” (in the second block)
ing video content differ Figs. 2, 3. For instance, the emo- that the editor-in-chief revealed in the video (in the second
tional state of sadness is almost four times higher among block). It was concluded that for one example, the ideal tim-
women (23.46%) compared to the reaction of men (5.47%). ing is no more than 4 s.

Fig. 1  Viewers’ emotional reaction of 30 of different ages, gender and social backgrounds
AI technologies in the analysis of visual advertising messages: survey and application

Fig. 2  The scale of emotional engagement of the male audience

Fig. 3  The scale of emotional engagement of female audience

The opening and closing blocks of the video are identical be effectively delivered to a wider audience by reducing the
in which brands are displayed against a backdrop of active length of moments with less emotional engagement by the
and unsettling music. However, the viewer experienced primary target audience. It is worth mentioning that the col-
confusion toward the end of the video despite registering a lected video lacked the initial video footage, which had less
“happy” emotion in response to the first segment showcas- emotional involvement from the primary target audience.
ing the brands. This could be attributable to the audience's It was found that the second segment of the original video
fatigue and stress level after watching the second segment detracts from the emotions of the non-target audience. With
of the video material, resulting in negative emotions rather minimal video editing changes, the attitude of the non-target
than positive (see Fig. 7). audience remains unaffected. Thus, the emotion of surprise
We conducted further research using an edited excerpt of is expressed quite clearly, as the audience reacts to provoca-
the original video to investigate whether video content could tive statements by the host. In addition, a decrease in the
 L. Sharakhina et al.

Fig. 4  Emotion of surprise (sum/s)

Fig. 5  Emotion of happiness (%/s)

emotion of joy and an increase in the emotions of sadness engagement of those outside the target audience. There-
and confusion were identified (Fig. 8). These parameters fore, we cannot target the video toward this group. How-
were significant for our research because they maximized the ever, to engage a wider audience, it is essential to modify
display of the target audience's emotional response. In the the sequence of video footage accordingly. This is because
video clip, oversaturation of intertextual components with- the target audience (men) responds emotionally to the clear
out any visualization did not cause any significant changes images presented in the video (Fig. 10). Also, advertising
in perception. Hence, these images did not provoke any video content that only shows a single speaker to the general
actual response. Thus, the video requires visual correction audience elicits more negative responses. Although this may
to enhance emotional engagement. increase the number of comments on the advertising video
Additionally, analysis of the responses from the group- content, it does not contribute to a positive brand image.
wide audience participating in the study demonstrated that The experiment did not confirm the hypothesis, but addi-
emotional involvement is at a substantially low level while tional important points were identified:
the level of emotional calmness is significantly high (Fig. 9).
Therefore, it can be concluded that shortening the length (1) Respondents respond to familiar images and perceive
of moments of low levels of emotional engagement, for them, but without emotional involvement. The target
the primary target audience, will not impact the emotional audience showed calm during the viewing.
AI technologies in the analysis of visual advertising messages: survey and application

Fig. 6  Emotion of Fear (%/s)

Fig. 7  Emotion of Happiness & Confused (%/s)

Fig. 8  The scale of emotional of a wide audience (%/s)

 L. Sharakhina et al.

Fig. 9  The scale of emotional of a wide audience (%/s)

Fig. 10  Emotional perception of the video fragment by the target audience

(2) When choosing a video, you need to understand the The potential audience should be shown a short video clip
goals set by the company. This is a targeted impact with different scenes: happiness, sadness, etc., and view-
either on the target audience or on a wider audience. ers should react somehow to the images. If a person has no
clear expression (a “Calm” scale higher than 85–90% during
Additional observations resulting from the experiment are the whole video), they are not allowed to participate in the
as follows. Different people have different reactions to the client’s test. If their expressions have a small coefficient,
same video. Nevertheless, we are able to draw conclusions their result should be multiplied to reach the average. This
about the audience and how the selected video fragments is needed because of the individual characteristics of each
affect people. Some people are quite emotional, and even person. The best audience to make a calibration with are
slight changes on their faces in brief moments of time can be children and teenagers because they are more likely to have
observed. Others do not have detailed expressions, and it can active emotional expressions compared to adults.
be difficult to observe the needed variety of emotions. It does The target with the highest quality for analysis is obtained
not mean that the person who's watching the video doesn't in a full-face view. There is a significant decrease in the
have any emotions at all, but in this case, researchers cannot accuracy of the results if the respondent has an unstable
rely on these people, because the whole experiment could position due to their active movements. Thus, for example,
be inconclusive. That's why it is important to start with an if the subject throws their head back from laughing while
emotional calibration before initiating a research analysis. watching, the AI algorithm would be unable to react quickly
AI technologies in the analysis of visual advertising messages: survey and application

and accurately adapt its calculation parameters for face rec- that using AI to assess visual impact on people is a promis-
ognition. As a result, in this case, the output for all param- ing direction among the applications of AI in different areas
eters is likely 0%. of marketing discussed at the beginning of this paper.
Combined with a short survey, this method can give an Automated scene analysis has attracted a lot of research-
accurate overview of the user's reaction and predict the suc- ers since the beginning of 2010. Intelligent surveillance
cess of advertising campaigns. Thus, it is important in order seems to have a great advantage over traditional manual
for the experiment to be effective to compare facial expres- observation, regardless of the scope of AI. This hypothesis
sion of the definite viewer to their own reactions but not to has been reflected in a number of scientific papers Saleh
the “base” of such reactions, so as to only bring into consid- et al. (2015), Ratre and Pankajakshan (2018), Xu et al.
eration an individual’s personal physiological expression. (2020). It can be argued that computer vision models have
Use of the average indicator on the general comparison of developed rapidly. They are now capable of recognizing
emotions is not accurate enough, as extreme values can be thousands of objects with up to 97% accuracy (Szegedy et al.
very different (again often due to physiology). Therefore, it 2016). Using computer vision, images are automatically ana-
is necessary to calculate indices for the base, neutral face lyzed and classified according to their content (LeCun et al.
of a particular subject, and this index should be subtracted 2015). Because of its automatic nature, computer vision
from all others, and instead of the average index, the median quickly identifies many content features in a large and var-
should be used. ied image set in a short time (Bakhshi and Gilbert 2015;
As a result of the study, it was found that it is necessary to Bianco et al. 2017). However, despite significant advances in
use the technology to not compare one face with thousands this direction, in our study, we found a significant decrease
(for example, the tool Amazon Rekognition, which compares in the accuracy of results when the respondent's position is
emotions and faces observed that watch the video with the unstable, which imposes certain limitations on the conduct
data in its vast database of faces and emotions), and one face of experiments and requires further improvements in com-
to itself (that is, the respondent's face is also compared with puter vision models.
its baseline reference). The collected data may be imperfect, Martínez-Garmendia (2023) has proven that machine
as the tool requires improvement, but it does allow for draw- learning methods are superior to traditional statistical meth-
ing useful conclusions about the audience and the impact of ods in their ability to predict consumer decisions. That's why
selected video fragments. these methods have attracted so much attention from mar-
keters. However, the author points out that machine learn-
ing requires great care in the analysis of marketing research
Discussion results. But all six machine learning alternatives perfor-
mance, analyzed in the article (learning multinomial logit,
There is no denying that visual content has the strong- random forests, neural networks, gradient boosting, support
est emotional impact on the human mind. Moreover, the vector machines, and an ensemble learning algorithm) “can
strength of these effects has been steadily increasing over vary depending on the approach, and presumably the data-
time. One cannot but agree with Ma and Palacios (2021) set.” Thus, it might be a very resource consuming process
who suggested that in recent years, image-based content has to make machine learning techniques in marketing analysis
emerged as the most popular electronic information means, “formidable relative to traditional statistical approaches,”
especially on social media platforms, such as Facebook, Ins- because it includes “data preprocessing, feature engineer-
tagram, Pinterest, Snapchat, TikTok, and Twitter. However, ing, data splitting between training and test sets, cross-
despite its popularity and importance in practice, we can validation, and use of tuning parameters” steps (Kuhn and
note that the issue of researching the impact of visual con- Johnson 2013). We absolutely agree with this statement,
tent on consumers remains understudied. Among the few as the experiment clearly demonstrated the importance of
studies that have been done in this area, we can point to comparing a facial expression of the certain viewer to his or
the research by Xiao and Ding (2014), Liu et al. (2020), her own reactions but not to the “base” of such reactions to
Zhang et al. (2020). Therefore, the study of visual phenom- bring into attention an individual’s physiological make-up.
ena requires an increase in the cognitive potential of research The results of the empiric research, conducted by authors,
methods. are quite alike to the above-mentioned situation: AI-based
We are able to observe various digital technologies, analytical instruments need to be “tuned” very thoroughly
which are used to support and to develop analytical prac- to a specific task for being more precise rather than classical
tices in marketing and advertising, including effectiveness ones. But even though an intelligent video surveillance ana-
measurement: computer vision for automated scene analysis, lytics system provides accurate data processing and effective
machine learning for product choice prediction, eye-tracking integration of information with fewer human operators (Guo
for Mass Media content perception analysis, etc. We believe et al. 2016; Nanne et al. 2020), it cannot be fully relied upon.
 L. Sharakhina et al.

When processing the results of our experiment, a similar group of countries that have similar cultural codes. However,
conclusion can be drawn. despite cross-cultural differences we believe that applying
The data obtained in the experiment are not perfect, as the AI to assess the visual impact on people is a universal direc-
tool itself requires improvement, but it is already possible to tion for its use, regardless of the country of origin of the
draw conclusions about the audience and how the selected advertising content.
video fragment affects. In subsequent empirical studies, it is Thus, we may observe the possibility to “delegate” ana-
necessary to calculate indicators for a neutral person, sub- lytical cognitive tasks from human to artificial intelligence in
tract it from all others, and use the median instead of the different practical oriented fields, including advertising. But
average. We found a similar idea in a study devoted to exam- the reduction in the number of people occupied in the sphere
ining the deferred effects of gaming platforms and gameplay of analytics by simple replacement of human intelligence to
rate of advergames on memory, attitude, and purchase inten- a machine will possibly lead to further reduction of human
tion (Sreejesh et al. 2021). Nevertheless, we didn’t find a analytical skills and, as the result, to the level of general
range of studies focused on AI-based analytical tools for vis- intelligence decreasing (Spearman 1904). Analytical (Logi-
ual advertising content effectiveness application, exploring cal/Mathematical) skills are the most popular to measure
the keys to advertising content success. During our experi- while we assess human intelligence today, nevertheless, the
ment, we found a pattern between the events in the frame development of machine intelligence and social structures
and the reactions to these events: language units to which and practices transformations caused by it will actualize
the viewer reacts most positively to; references to familiar the request for other human intelligence types: Linguistic,
cultural images and violations of logic; moments when the Spatial, Bodily-Kinesthetic, Musical, Interpersonal, Intrap-
contact with the viewer is lost and their causes (verbal word- ersonal, and Naturalist (Gardner 1983, 2011). The Findings
ing, music, editing, or action in the scene, etc.). All of this from the 2020 Artificial intelligence Global Executive study
will make it possible in future to provide recommendations and research project “Expanding AI’s Impact with Organi-
on each specific video for final editing and post-production, zational Learning” stresses the idea to convey some time-
which will certainly allow for more effective development consuming cognitive functions from humans to machines
of advertising video content. by simple replacement unpromising: “This cycle of mutual
The research on mobile in app advertising effectiveness learning makes humans and machines smarter, more rel-
using app publishers-controlled factors (Truong 2023) is evant, and more effective. Mutual learning between human
also interesting, because the author raises the issue of mul- and machine is essential to success with AI” (Ransbotham
tifactor analysis necessity for gaining better results: the dura- et al. 2020).
tion, the size, the position, and the timing of their ad spaces.
The advertisement created with advertiser’s understanding
of balanced consumers’ preferences of all these factors influ- Conclusions
ence greatly on the click-to-impression ratio increase. In our
research, we also noted the differences in perception between Artificial intelligence technology is now in demand across
the target audience and the control group, with the latter's all industries. AI provides the ability to examine huge
emotional engagement at a low level. At the same time, we amounts of information in the ever-growing volumes of
found that a more specific and imaginative visualization of data, to automate repetitive processes, and to identify the
the content is needed to increase the emotional engagement solutions to large-scale problems with data that provide a
of the target audience. competitive advantage for modern organizations. There is
Another promising direction for video content analysis is already a stable correlation between the use of AI and higher
the dictionary-based approach proposed by Humphreys and levels of business performance and revenue (IBM 2020). AI
Wang (2018) for textual material. For example, the authors is guaranteed to help improve economic outcomes, free up
suggest that when studying gestures, one must define a “dic- staff, launch more effective communication campaigns, and
tionary” of gestures that represent a construct, or code all better analyze their results. By 2026, 1/4 of all marketing
gestures and then group them into meaningful categories. tasks will be automated, according to the Marketing Artifi-
For visual displays of emotion that can be captured during cial Intelligence Institute. And 43% of marketers surveyed
observation, a similar technique can be used. However, we believe AI will perform half of all tasks by then (Roetzer
believe that the creation of a universal dictionary for both 2021).
gestures and facial expressions is questionable. It is unlikely In our study, we tried to consider one of the tools to solve
to cover all possible interpretations of gestures and facial the problem of improving communication efficiency while
expressions, taking into account cross-cultural differences saving material resources. One way to do this could be using
in non-verbal communication. Therefore, it makes sense AI. In recent years, the application of AI to visual sentiment
to develop such dictionaries for a single country or for a analysis has been a particularly active area of research. It
AI technologies in the analysis of visual advertising messages: survey and application

should be noted that this is a relatively new way of studying factors, and it is necessary to calculate indicators for a
emotional responses to visual stimuli, which is exactly what baseline, neutral person, subtract it from all others, and
video ads are. While in the past, AI was primarily used to use the median instead of the average.
create visual effects, such as in the film industry, the use (4) The machine and human intelligence coevolution in
of AI now allows advertisers to identify the patterns in the marketing and advertising is a prospective trend. AI-
target audience's response to specific video content. based biometric face-tracking analytical technologies
The experiment we conducted can lead companies pro- are of high potential for these professional practices,
moting video content to develop their own methodology but human intelligence is still needed to “calibrate” the
and improve the tools available, allowing them to expand set of audience reactions to certain advertising video
both video content analytics and reports for their clients. elements. The use of AI helps to tune the most appro-
As a result of the research and the empiric experiment, the priate length of advertising video, letting the advertiser
hypothesis of the empirical pilot study was not confirmed, to save money for placing advertising without losing
but the hypothesis of the study was approved. its effectiveness, to reduce the cost of paid viewers, as
AI technologies based on big data analysis can provide video content is more likely to receive viral reach due
recommendations for the development of advertising con- to user preferences, and as a result, to reduce the cost
tent, thereby to improving the communication effective- of the viewer's target action.
ness of advertising messages, but there are some additional
important points were revealed: Additionally, the video content can be changed according
to AI-based recommendations if it’s possible. In case it is
(1) Respondents react to familiar images and perceive impossible to change the video content, the client may select
them, but without emotional engagement. Those who the target audience that can provide viewings and viral cov-
do not represent the advertising video target group erage of the video. Therefore, AI-based marketing analytical
demonstrate calmness while watching it. When target- technologies can offer more detailed and nuanced analyt-
ing a video, it is necessary to understand the goals set ics that can be used by marketers and advertisers to watch
by the company: the “banner blindness” issue tends to videos through the eyes of the viewer and identify a video's
be related to the situation of processing familiar video weaknesses and to receive more loyal customers.
content without their emotional response. Thus, the tar- We can also consider a growing industry that combines
get audience will not perceive the information about the advertising, digital, and AI technology, where the latter can
provided product, service, or brand in case they are not act as an intermediary between advertisers, video producers,
emotionally involved. and audiences, as a platform. It allows the interests of all
(2) By matching the emotional response to the video with parties involved in the advertising process to be taken into
the client's goals, recommendations can be made for account and enables the distribution of useful, esthetically
the creation of the video. That is, we can understand pleasing visual advertising content.
when and how best to motivate the viewer to perform But there are some limitations that should be taken into
the planned action by using AI-based biometric face- account:
tracking analytical technologies. It is ineffective to try
to replicate the same video content to different audi- • Video advertising multimodal messages should be
ences by 'cutting off' the episodes with the low-level adapted and ‘interpreted’ for different languages and
emotional engagement of other audiences. cultural frames, and vice versa when AI-based analyti-
(3) It is important to make an “emotion calibration” before cal instruments are applied for advertising video content.
analyzing potential advertising video content effective- • The need to ‘calibrate’ the dataset used by AI-based
ness. The use of average indicators and general emo- biometric face-tracking analytics technologies limits the
tional reactions for AI-based biometric face-tracking scalability of the results of such analytics, making them
analytical technologies is not enough to make conclu- relevant for large markets or long-term advertising cam-
sions of potential effectiveness. The introduction of paigns.
neuromarketing techniques to the development of video • The implementation of AI technologies updates ethi-
content for the advertising (for example, the Amazon cal issues such as AI-driven judgments without human
Rekognition platform, which was enhanced with some expertise, cybersecurity, fears of rising unemployment
modifications) must be tuned by a preliminary study of due to automation, etc.
the expression of certain individual emotions. And it is
better if the settings are based on the FACS methodol- As prospective directions of further research on the prob-
ogy (the IMotions platform serves as an example). The lem of AI technologies in the analysis of visual advertising
situation is caused by cultural, psychological, and other messages, we would outline the following:
 L. Sharakhina et al.

• Effective modeling of machine and human intelligence bahaya merokok pada remaja. Health Sciences and Pharmacy
coevolution extent in marketing and advertising, taking Journal 5 (2): 56–62. https://​doi.​org/​10.​32504/​hspj.​v5i2.​520.
Achar, C., et al. 2016. What we feel and why we buy: The influence
into account the market size and homogeneity of the tar- of emotions on consumer decision-making. Current Opinion in
get audience, the planned duration of the marketing and Psychology 10: 166–170. https://​doi.​org/​10.​1016/J.​COPSYC.​
advertising campaign, the project investment costs and 2016.​01.​009.
potential revenues, etc.; Alawaad, D.H.A. 2021. The role of artificial intelligence (AI) in
public relations and product marketing in modern organiza-
• The optimal dataset issue of facial movements, express-
tions. Turkish Journal of Computer and Mathematics Educa-
ing definite emotions of different audiences, customized tion (TURCOMAT) 12 (14): 3180–3187.
by social, demographic, psychographic, cultural, and Alfi. Everything Audience. 2021. Facial Recognition Advertising:
other criteria, which helps to create effective advertising The Future is Here, https://​www.​getal​fi.​com/​adver​tising/​facial-​
recog​nition-​adver​tising-​future-​is-​here/. Accessed 23 July 2023.
visual content; Bader, V., and S. Kaiser. 2019. Algorithmic decision-making? The
• Perspectives of marketing and advertising professional user interface and its role for human involvement in decisions
specialization of machine and human intelligence in the supported by artificial intelligence. Organization 26 (5): 655–
technosociety of our future. 672. https://​doi.​org/​10.​1177/​13505​08419​855714.
Bakhshi, S., and E. Gilbert. 2015. Red, purple and pink: The colors
of diffusion on Pinterest. PLoS ONE 10 (2): 0117148. https://​
Anyway, all the challenges of AI-based analytical tools doi.​org/​10.​1371/​JOURN​AL.​PONE.​01171​48.
for measuring marketing and advertising effectiveness are Baliun, O., et al. 2021. Psychosemantic interpretation of media space
driving advertisers to use them for recurring or large-scale video advertising for brands (Mcdonald’s, Coca-Cola, Nestlé,
Roshen). Revista Amazonia Investiga 10 (46): 259–272. https://​
tasks with tremendous prospects in the AI-pervasive tech- doi.​org/​10.​34069/​AI/​2021.​46.​10.​26.
nosociety of our future. And it brings us to a fundamental Benway, J.P. 1998. Banner blindness: The irony of attention grab-
discussion of autonomous AI personhood and its ethical bing on the World Wide Web. Proceedings of the Human Fac-
nature: “Can they be moral agents? Can they be morally tors and Ergonomics Society Annual Meeting 42 (5): 463–467.
https://​doi.​org/​10.​1177/​15419​31298​04200​504.
responsible for actions and their consequences?” (Maruy- Białowąs, S., and A. Szyszka. 2019. Eye-tracking in market-
ama 2022), and others. These are the subjects of prospective ing research. In Managing economic innovations—meth-
transdisciplinary studies. ods and instruments, ed. R. Romanowski. Poznań: Bogucki
Therefore, we must emphasize the ethical and security Wydawnictwo Naukowe.
Bianco, S., et al. 2017. Deep learning for logo recognition. Neu-
issues of consumer personal data processing in video content rocomputing 245: 23–30. https://​doi.​org/​10.​1016/j.​neucom.​
analysis. However, despite the undoubted relevance of the 2017.​03.​051.
above issues, there is no alternative to driving marketing Boerman, S.C., S. Kruikemeier, and F.J. Zuiderveen Borgesius.
toward the digital future. As Vollrath and Villegas (2022) 2017. Online behavioral advertising: A literature review and
research agenda. Journal of Advertising 46 (3): 363–376.
rightly point out, the field of marketing analytics will be https://​doi.​org/​10.​1080/​00913​367.​2017.​13393​68.
challenged to make sense of more data from more diverse Boz, H. 2018. Emotion extraction from facial expressions by using
sources. artificial intelligence techniques. BRAIN—Broad Research
These issues are still on the human intelligence side. The in Artificial Intelligence and Neuroscience, https://​w ww.​
acade​mia.​edu/​36075​151/​Emoti​on_​Extra​ction_​from_​Facial_​
question of AI agents’ moral responsibility, which is a pre- Expre​s sions_​by_​U sing_​A rtif​i cial_​I ntel​l igen​c e_​Techn​i ques.
condition for juridical responsibility and legal punishment, Accessed 02 May 2023.
should be reviewed in connection to humans’ moral agency Branda, A.F., V. Lala, and P. Gopalakrishna. 2018. The market-
degradation and the governmental role in the process. ing analytics orientation (MAO) of firms: Identifying factors
that create highly analytical marketing practices. Journal of
Marketing Analytics 6 (3): 84–94. https://​d oi.​o rg/​1 0.​1 057/​
s41270-​018-​0036-8.
Declarations Burke, M., et al. 2005. High-cost banner blindness. ACM Transac-
tions on Computer-Human Interaction (TOCHI) 12 (4): 423–445.
Conflict of interest On behalf of all authors, the corresponding author https://​doi.​org/​10.​1145/​11211​12.​11211​16.
states that there is no conflict of interest. Canal, F.Z., et al. 2022. A survey on facial emotion recognition tech-
niques: A state-of-the-art literature review. Information Sciences
582: 593–617. https://​doi.​org/​10.​1016/j.​ins.​2021.​10.​005.
Cantoni, V. et al. 2013. Banner positioning in the masthead area of
online newspapers: an eye tracking study. In Proceedings of
References the 14th International Conference on Computer Systems and
Technologies. New York, NY, USA: Association for Computing
Abedi, F., and S. Koslow. 2022. Can personalization or creativity Machinery (CompSysTech’13): pp. 145–152. doi: https://d​ oi.o​ rg/​
reduce banner blindness? An executive functions approach to 10.​1145/​25167​75.​25167​89.
media and creative strategies. Journal of Advertising Research Cao, W., et al. 2022. Video emotion analysis enhanced by recogniz-
62 (3): 201–218. https://​doi.​org/​10.​2501/​JAR-​2022-​014. ing emotion in video comments. International Journal of Data
Adnani, H., L. Fadila, and S. Royani. 2021. Efektivitas video dan Science and Analytics 14 (2): 175–189. https://​doi.​org/​10.​1007/​
leaflet dalam meningkatkan pengetahuan dan sikap tentang s41060-​022-​00317-0.
AI technologies in the analysis of visual advertising messages: survey and application

Ciçek, M., İ Eren-Erdoğmuş, and İ Daştan. 2018. How to increase the Krishnan, S. S. and Sitaraman, R. K. (2013) Understanding the effec-
awareness of in-app mobile banner ads: Exploring the roles of tiveness of video ads: A measurement study. In Proceedings of
banner location, application type and orientation. International the ACM SIGCOMM Internet Measurement Conference, IMC:
Journal of Mobile Communications 16 (2): 153–166. https://​doi.​ pp. 149–162. doi: https://​doi.​org/​10.​1145/​25047​30.​25047​48.
org/​10.​1504/​IJMC.​2018.​089757. Kugler, T., T. Connolly, and L.D. Ordóñez. 2012. Emotion, decision,
Chen, C.-M., and Y.-C. Sun. 2012. Assessing the effects of different and risk: Betting on gambles versus betting on people. Journal
multimedia materials on emotions and learning performance for of Behavioral Decision Making 25 (2): 123–134. https://​doi.​org/​
visual and verbal style learners. Computers & Education 59 (4): 10.​1002/​BDM.​724.
1273–1285. https://​doi.​org/​10.​1016/j.​compe​du.​2012.​05.​006. Kuhn, M., and K. Johnson. 2013. Applied predictive modeling. Applied
Davies, J. 2016. The BBC uses facial recognition to measure native Predictive Modeling 1: 1–600. https://​doi.​org/​10.​1007/​978-1-​
advertising. https://​digid​ay.​com/​media/​bbc-​facial-​recog​nition-​ 4614-​6849-3/​COVER.
native-​adver​tising. Accessed: 02 May 2023. Kumar, T. 2020. Data mining based marketing decision support system
Ekman, P. 1993. Facial expression and emotion. American Psychologist using hybrid machine learning algorithm. Journal of Artificial
48 (4): 384–392. https://​doi.​org/​10.​1037/​0003-​066X.​48.4.​384. Intelligence and Capsule Networks 2 (3): 185–193. https://​doi.​
Enholm, I.M., et al. 2022. Artificial intelligence and business value: A org/​10.​36548//​JAICN.​2020.3.​006.
literature review. Information Systems Frontiers 24 (5): 1709– Lee, J., and H. Lee. 2012. Canonical correlation analysis of online
1734. https://d​ oi.o​ rg/1​ 0.1​ 007/S
​ 10796-0​ 21-1​ 0186-W/T
​ ABLES/8. video advertising viewing motivations and access characteris-
Eisenberg, H. 2014. Humans process visual data better. Thermopy- tics. New Media & Society 14 (8): 1358–1374. https://​doi.​org/​
lae Sciences + Technology, https://​www.t-​scien​ces.​com/​news/​ 10.​1177/​14614​44812​444708.
humans-​proce​ss-​visual-​data-​better. Accessed 02 May 2023. LeCun, Y., Y. Bengio, and G. Hinton. 2015. Deep learning. Nature 521:
Etzioni, A. 2011. Behavioral economics. American Behavioral Scientist 436–444. https://​doi.​org/​10.​1038/​natur​e14539.
55 (8): 1099–1119. https://​doi.​org/​10.​1177/​00027​64211​412355. Li, R., et al. 2021. Application and development trend of artificial intel-
Faggella, D. 2019. Artificial intelligence in marketing and advertis- ligence in enterprise marketing. Journal of Physics 2: 22032.
ing—5 examples of real traction. EMERJ, 18 February, https://​ https://​doi.​org/​10.​1088/​1742-​6596/​1881/2/​022032.
emerj.​com/​ai-​sector-​overv​iews/​artif​i cial-​intel​ligen​ce-​in-​marke​ Libet, B., et al. 1983. Time of conscious intention to act in relation to
ting-​and-​adver​tising-​5-​examp​les-​of-​real-​tract​ion/. Accessed 02 onset of cerebral activity (readiness-potential). Brain 106 (3):
May 2023. 623–642. https://​doi.​org/​10.​1093/​brain/​106.3.​623.
Friedman, M. 2007. The methodology of positive economics. The Phi- Liu L, Dzyabura D, Mizik N (2020) Visual listening in: extracting
losophy of Economics: An Anthology II: 145–178. https://d​ oi.o​ rg/​ brand image portrayed on social media. Marketing Science 39(4):
10.​1017/​CBO97​80511​819025.​010. 669–686. https://​doi.​org/​10.​1287/​MKSC.​2020.​1226.
Helmefalk, M., and B. Hultén. 2017. Multi-sensory congruent cues in Loureiro, S.M.C., J. Guerreiro, and I. Tussyadiah. 2021. Artificial intel-
designing retail store atmosphere: Effects on shoppers’ emotions ligence in business: State of the art and future research agenda.
and purchase behavior. Journal of Retailing and Consumer Ser- Journal of Business Research 129: 911–926. https://​doi.​org/​10.​
vices 38: 1–11. https://​doi.​org/​10.​1016/j.​jretc​onser.​2017.​04.​007. 1016/j.​jbusr​es.​2020.​11.​001.
Humphreys, A., and R.J.-H. Wang. 2018. Automated text analysis Ma, Z., and S. Palacios. 2021. Image-mining: Exploring the impact
for consumer research. Journal of Consumer Research. 44 (6): of video content on the success of crowdfunding. Journal of
1274–1306. https://​doi.​org/​10.​1093/​jcr/​ucx104. Marketing Analytics 9 (4): 265–285. https://​doi.​org/​10.​1057/​
Gardner, H. 1983. Frames of mind: A theory of multiple intelligences. s41270-​021-​00133-8.
New York: Basic Books. Margariti, K., L. Hatzithomas, and C. Boutsouki. 2023. Implementing
Gardner, H. 2011. The theory of multiple intelligences: As psychology, eye tracking technology in experimental design studies in food
as education, as social science. Madrid: José Cela University. and beverage advertising. In Consumer research methods in food
Gurunathan, A., and K.S. Lakshmi. 2023. Exploring the perceptions of science, ed. C. Gómez-Corona and H. Rodrigues, 293–311. New
generations X, Y and Z about online platforms and digital mar- York: Humana Press. https://​doi.​org/​10.​1007/​978-1-​0716-​3000-
keting activities : A focus-group discussion based study. Inter- 6_​15.
national Journal of Professional Business Review 8 (5): e02122. Martínez-Garmendia, J. 2023. Machine learning for product choice
https://​doi.​org/​10.​26668/​busin​essre​view/​2023.​v8i5.​2122. prediction. Journal of Marketing Analytics 4: 1–12. https://​doi.​
Guo, H., J. Wang, and H. Lu. 2016. Multiple deep features learning org/​10.​1057/​S41270-​023-​00217-7/​FIGUR​ES/1.
for object retrieval in surveillance videos. IET Computer Vision Maruyama, Y. 2022. Moral philosophy of artificial general intelligence:
10 (4): 268–272. https://​doi.​org/​10.​1049/​IET-​CVI.​2015.​0291. agency and responsibility. Lecture Notes in Computer Science
IBM Global AI Adoption Index 2022 | IBM. 2022. https://​www.​ibm.​ 13154: 139–150. https://​doi.​org/​10.​1007/​978-3-​030-​93758-4_​
com/​watson/​resou​rces/​ai-​adopt​ion?​mhsrc=​ibmse​arch_​a&​mhq=​ 15/​COVER.
Global AI Adoption Index 2021. Accessed: 02 May 2023. Mehendale, N. 2020. Facial emotion recognition using convolutional
Izard, C.E. 1991. The psychology of emotions. New York, NY: Plenum neural networks (FERC). SN Applied Sciences 2 (3): 446. https://​
Press. doi.​org/​10.​1007/​s42452-​020-​2234-1.
Katona, G. 1951. Psychological analysis of economic behavior. New Mishra, A. 2019. Amazon Rekognition. In: Machine learning in the
York: McGraw-Hill. AWS cloud. Wiley, pp. 421–444. doi: https://​doi.​org/​10.​1002/​
Khrais, L.T. 2020. (2020) Role of artificial intelligence in shaping 97811​19556​749.​CH18.
consumer demand in e-commerce. Future Internet 12 (12): 226. Mitchell, W.J. 1995. Picture theory: Essays on verbal and visual rep-
https://​doi.​org/​10.​3390/​FI121​20226. resentation. Chicago: University of Chicago Press.
Kietzmann, J., and L.F. Pitt. 2020. Artificial intelligence and machine Muñoz-Leiva, F., et al. 2021. The influence of banner position and
learning: What managers need to know. Business Horizons 63 user experience on recall: The mediating role of visual attention.
(2): 131–133. https://​doi.​org/​10.​1016/J.​BUSHOR.​2019.​11.​005. Spanish Journal of Marketing ESIC 25 (1): 85–114. https://​doi.​
Kim, G., et al. 2021. A CNN-based advertisement recommendation org/​10.​1108/​SJME-​04-​2020-​0050/​FULL/​PDF.
through real-time user face recognition. Applied Sciences 11 Murillo-Zegarra, M., C. Ruiz-Mafe, and S. Sanz-Blas. 2020. The
(20): 9705. https://​doi.​org/​10.​3390/​app11​209705. effects of mobile advertising alerts and perceived value on
 L. Sharakhina et al.

continuance intention for branded mobile apps. Sustainability localisation and detection in surveillance videos. IET Computer
12 (17): 6753. https://​doi.​org/​10.​3390/​su121​76753. Vision 12 (6): 933–940. https://​doi.​org/​10.​1049/​IET-​CVI.​2017.​
Nanne, A.J., et al. 2020. The use of computer vision to analyze brand- 0469.
related user generated image content. Journal of Interactive Ray, T. 2018. Machine learning and artificial intelligence in marketing
Marketing 50: 156–167. https://​doi.​org/​10.​1016/J.​INTMAR.​ and advertising, https://​tanmo​yray.​com/​blog/​marte​ch-​analy​tics/​
2019.​09.​003. machi​ne-​learn​ing-​and-​artif​i cial-​intel​ligen​ce-​in-​marke​ting-​adver​
Nevid, J.S. 2010. Introduction to the special issue: Implicit measures tising/, Accessed: 02 May 2023.
of consumer response-the search for the Holy Grail of marketing Reinsel, D. et al. 2018. The APJxC datasphere: enterprise acceleration
research. Psychology and Marketing 27 (10): 913–920. https://​ to meet demand. The digitization of the world: from edge to
doi.​org/​10.​1002/​mar.​20365. core, IDC #US44413318, (October), https://​www.​seaga​te.​com/​
Ning, B., et al. 2023. Effects of banner ad type, web content type files/​www-​conte​nt/​our-​story/​trends/​files/​data-​age-​apjxc-​idc.​pdf.
and theme consistency on banner blindness: An eye move- Accessed: 02 May 2023.
ment study. Cognitive Processing. https://​d oi.​o rg/​1 0.​1 007/​ Rekha, A.G., M.S. Abdulla, and S. Asharaf. 2016. Artificial intelli-
s10339-​023-​01131-7. gence marketing: An application of a novel lightly trained sup-
Owens, J.W., B.S. Chaparro, and E.M. Palmer. 2011. Text advertising port vector data description. Journal of Information and Optimi-
blindness: The new banner blindness? Journal Usability Studies zation Sciences 37: 681–691.
6 (3): 172–197. Roetzer, P. 2021. 2021 State of marketing AI report, https://w ​ ww.m​ arke​
Ozdemir, M. A. et al. 2019. Real time emotion recognition from facial tingai​ insti​ tute.c​ om/2​ 021-s​ tate-o​ f-m
​ arket​ ing-a​ i-r​ eport. Accessed:
expressions using CNN architecture. In: Proceedings of the 2019 02 May 2023.
Medical Technologies Congress (TIPTEKNO). IEEE: pp. 1–4. Saadon, J.R., et al. 2022. Real-time emotion detection by quantitative
doi: https://​doi.​org/​10.​1109/​TIPTE​KNO.​2019.​88952​15. facial motion analysis. medRxiv. https://​doi.​org/​10.​1101/​2022.​
Paschen, J., J. Kietzmann, and T.C. Kietzmann. 2019. Artificial intel- 10.​28.​22276​059.
ligence (AI) and its implications for market knowledge in B2B Saiprasad, K., and A. Meenakshi. 2022. A study on amazon prime
marketing. Journal of Business and Industrial Marketing 34 (7): video services in Chennai evaluating consumer preferences over
1410–1419. https://d​ oi.o​ rg/1​ 0.1​ 108/J​ BIM-1​ 0-2​ 018-0​ 295/F ​ ULL/​ other. International Journal of Health Sciences. https://​doi.​org/​
XML. 10.​53730/​ijhs.​v6nS5.​8995.
Paschen, J., et al. 2020. Artificial intelligence (AI) and value co-crea- Saleh, S.A.M., S.A. Suandi, and H. Ibrahim. 2015. Recent survey on
tion in B2B sales: Activities, actors and resources. Australasian crowd density estimation and counting for visual surveillance.
Marketing Journal 29 (3): 243–251. https://​doi.​org/​10.​1016/j.​ Engineering Applications of Artificial Intelligence 41: 103–114.
ausmj.​2020.​06.​004. https://​doi.​org/​10.​1016/J.​ENGAP​PAI.​2015.​01.​007.
Pedersen, I. and Duin, A. 2022. AI agents, humans and untangling Samuelson, P.A. 1938. The empirical implications of utility analysis.
the marketing of artificial intelligence in learning environments. Econometrica 6 (4): 344. https://​doi.​org/​10.​2307/​19054​11.
In: Proceedings of the 55th Hawaii International Conference on Sapronov, F., and E. Gorbunova. 2021. The role of emotional informa-
System Sciences. doi:https://​doi.​org/​10.​24251/​HICSS.​2022.​002. tion in banner blindness. Journal of Vision 21 (9): 2533–2533.
Pernice, K. 2018. Banner blindness revisited: users dodge ads on https://​doi.​org/​10.​1167/​JOV.​21.9.​2533.
mobile and desktop. https://​www.​nngro​up.​com/​artic​les/​banner-​ Savage, L.J. 1954. The foundations of statistics. New York: Wiley.
blind​ness-​old-​and-​new-​findi​ngs/. Accessed 02 May 2023. Szegedy, C. et al. 2016. Inception-v4, Inception-ResNet and the impact
Provotorov, G. 2019. “Banner blindness” in relation to motivational of residual connections on learning. In 31st AAAI Conference on
reactivity and emotional expression in banner advertisements. Artificial Intelligence, AAAI 2017: pp. 4278–4284. doi: https://​
Honors College Theses, https://​digit​alcom​mons.​pace.​edu/​honor​ doi.​org/​10.​1609/​aaai.​v31i1.​11231.
scoll​ege_​theses/​253. Accessed 02 May 2023. Shah, D., and E. Shay. 2019. Handbook of advances in marketing in an
Pashkevich, M., et al. 2012. Empowering online advertisements era of disruptions: Essays in honour of Jagdish N. Sheth. Mathura
by empowering viewers with the right to choose. Journal of Road. https://​doi.​org/​10.​4135/​97893​53287​733.
Advertising Research 52 (4): 451–457. https://​doi.​org/​10.​2501/​ Shazna, N. 2013. Visual literacy in an age of data—learning—source:
JAR-​52-4-​451-​457. An OpenNews project June 13, https://s​ ource.o​ penne​ ws.o​ rg/a​ rtic​
Patlar Akbulut, F. 2022. Hybrid deep convolutional model-based emo- les/​visual-​liter​acy-​age-​data/. Accessed 02 May 2023.
tion recognition using multiple physiological signals. Computer Sikevich, Z.V. 2016. Method of semantic differential in sociological
Methods in Biomechanics and Biomedical Engineering 25 (15): research (the experience of application) Vestnik of Saint Peters-
1678–1690. https://​doi.​org/​10.​1080/​10255​842.​2022.​20326​82. burg University. Sociology 9 (3): 118–128. https://​doi.​org/​10.​
Petrenko, V.F. 1988. Psychosemantic of consciousness. Moscow: Mos- 21638/​11701/​spbu12.​2016.​309(inRus​sian).
cow State University. Small, D.A., and N.M. Verrochi. 2009. The face of need: Facial emo-
Pituxcoosuvarn, M. et al. (2021) Effects of virtual reality on students’ tion expression on charity advertisements. Journal of Marketing
creative thinking during a brainstorming session. In Proceedings Research 46 (6): 777–787. https://​doi.​org/​10.​1509/​JMKR.​46.6.​
of the 29th International Conference on Computers in Education 777_​JMR6F.
Conference, ICCE 2021, vol. 1, pp. 387–391. Smith, A. (2007) An inquiry into the nature and causes of the wealth of
Posner, J., J.A. Russell, and B.S. Peterson. 2005. The circumplex model nations. Edited by Sálvio M. Soares. MetaLibri Digital Library.
of affect: An integrative approach to affective neuroscience, Smith, V. 2008. Visual persuasion: Issues in the translation of the vis-
cognitive development, and psychopathology. Development and ual in advertising. Meta 53 (1): 44–61. https://​doi.​org/​10.​7202/​
Psychopathology. https://​doi.​org/​10.​1017/​S0954​57940​50503​40. 01797​3ar.
Ransbotham, S. et al. 2020. Expanding AIs impact with organizational Spearman, C. 1904. “General Intelligence”, Objectively determined
learning. MIT Sloan Management Review, https://​sloan​review.​ and measured. American Journal of Psychology 15: 201–293.
mit.e​ du/p​ rojec​ ts/e​ xpand​ ing-a​ is-i​ mpact-w
​ ith-o​ rgani​ zatio​ nal-l​ earn​ Sreejesh, S., T. Ghosh, and Y.K. Dwivedi. 2021. Moving beyond
ing/. Accessed: 02 May 2023. the content: The role of contextual cues in the effectiveness of
Ratre, A., and V. Pankajakshan. 2018. Tucker tensor decomposi- gamification of advertising. Journal of Business Research 132:
tion-based tracking and Gaussian mixture model for anomaly 88–101. https://​doi.​org/​10.​1016/j.​jbusr​es.​2021.​04.​007.
AI technologies in the analysis of visual advertising messages: survey and application

Statista, Inc. New York, NY, USA. TV & Video Advertising, https://​ and Humanized Computing 13: 105. https://​doi.​org/​10.​1007/​
www.s​ tatis​ ta.c​ om/o​ utloo​ k/2​ 0100/1​ 00/t​ v-v​ ideo-a​ dvert​ ising/w
​ orld​ S12652-​021-​03216-7.
wide. Accessed 10 May 2023. Zhao, Y. 2023. The influence factors of short video marketing on con-
Stepanycheva, E.V. 2011. Mythologisation of consumption as a brand- sumer purchasing behavior and the effective suggestions. In SHS
ing tool, social and economic phenomena and processes. Journal Web of Conferences. Edited by J. Hj Ahmad and J. Guo, vol. 155,
of Consumer Culture 17: 216–222 (in Russian). p. 02002. doi: https://​doi.​org/​10.​1051/​shsco​nf/​20231​55020​02.
Thaler, R.H. 1980. Toward a positive theory of consumer choice. Jour- Zhang, Q., W. Wang, and Y. Chen. 2020. Frontiers: In-consumption
nal of Economic Behavior & Organization 1 (1): 39–60. https://​ social listening with moment-to-moment unstructured data: The
doi.​org/​10.​1016/​0167-​2681(80)​90051-7. case of movie appreciation and live comments. Marketing Sci-
Thaler, R.H. 2000. From homo economicus to homo sapiens. Journal ence 39 (2): 285–295. https://d​ oi.o​ rg/1​ 0.1​ 287/M
​ KSC.2​ 019.1​ 215.
of Economic Perspectives 14 (1): 133–141. https://​doi.​org/​10.​
1257/​jep.​14.1.​133. Publisher's Note Springer Nature remains neutral with regard to
The business value of AI | IBM. 2020. https://​www.​ibm.​com/​thoug​ht-​ jurisdictional claims in published maps and institutional affiliations.
leade​rship/​insti​tute-​busin​ess-​value/​en-​us/​report/​ai-​value-​pande​
mic. Accessed 02 May 2023. Springer Nature or its licensor (e.g. a society or other partner) holds
The Guardian. 2017. KFC China is using facial recognition tech to exclusive rights to this article under a publishing agreement with the
serve customers–but are they buying it?, http://​www.​thegu​ard- author(s) or other rightsholder(s); author self-archiving of the accepted
ian.​com/​techn​ology/​2017/​jan/​11/​china-​beiji​ng-​first-​smart-​resta​ manuscript version of this article is solely governed by the terms of
urant-​kfc-​facial-​recog​nition. Accessed 23 July 2023. such publishing agreement and applicable law.
Truong, V. 2023. Optimizing mobile in-app advertising effectiveness
using app publishers-controlled factors. Journal of Marketing
Analytics. https://​doi.​org/​10.​1057/​s41270-​023-​00230-w. Larisa Sharakhina got a specialst diploma in Public Relations in
Tversky, A., and D. Kahneman. 1986. Rational choice and the framing 2003 (St. Petersburg Elecrotechnical University “LETI”). In 2007 she
of decisions. The Journal of Business 59 (4): S251–S278. defeated her PhD (St. Petersburg University) in philosophy of science.
Vollrath, M.D., and S.G. Villegas. 2022. Avoiding digital marketing In 2017 she became a member of IEEE. She was occupied within
analytics myopia: Revisiting the customer decision journey as a St.Petersburg Government mediarelations for 10 years, was ressponsi-
strategic marketing framework. Journal of Marketing Analytics ble for volunteer popularisarion in the city for 2011-2016, and was giv-
10 (2): 106–113. https://​doi.​org/​10.​1057/​s41270-​020-​00098-0. ing lectures for PR majors since 2005, than in 2016 focused on higher
Ware, C. 2012. Information visualization, third edition: Perception for education in the field of PR and advertising at St.Petersburg Electro-
design. MA: Morgan Kaufmann. technical University. Dr. Sharakhina is a vice-president of the Russian
Wang, Y., Y. Jia, and Q. Xie. 2022. Research on the changes of avoid- Public Relations Academic Association, member of the Global Block-
ance psychology and emotional behavior of consumers in online chain Business Council.
video related advertising. International Journal of Neuropsy-
chopharmacology 25 (1): A49–A50. https://​doi.​org/​10.​1093/​ Irina Ilyina received her PhD from the Academy of Media Industry
ijnp/​pyac0​32.​068. in Moscow in 2009. Since 2011, she has taught at Ulyanovsk State
Wicaksono, S.R., D.P. Nugroho, and R. Setiawan. 2022. pendamp- University and since 2013 at St. Petersburg University of Management
ingan pembuatan video profil untuk konten digital marketing Technologies and Economics. In 2018 she became a member of IEEE.
di Sekolah Islam Al Jannah. Jurnal ABDINUS : Jurnal Peng- Since 2019, she has been an associate professor at the Department
abdian Nusantara 6 (2): 442–447. https://​doi.​org/​10.​29407/​ja.​ of Advertising and Public Relations at Saint Petersburg Electrotechni-
v6i2.​16503. cal University “LETI”. I. Ilyina is the author of more than 50 articles.
Wilson, J. H. and Joye, S. W. 2020. Research designs and variables, Her research interests include processes and applications of PR tech-
in research methods and statistics: an integrated approach. 2455 nologies, public opinion research methods, semiotics of communica-
Teller Road, Thousand Oaks California 91320: SAGE Publica- tions, interaction of artificial intelligence and media.
tions, Inc: pp. 40–72. doi: https://​doi.​org/​10.​4135/​97810​71802​
717.​n3. Dmitrii Kaplun Associate Professor (2015), senior researcher at St.
Yang, Q., et al. 2021. How to overcome online banner blindness? A Petersburg Electrotechnical University “LETI”. In 2009 he defended
study on the effects of creativity. Journal of Research in Interac- PhD thesis in digital signal processing at St Petersburg Electrotech-
tive Marketing 15 (2): 223–242. https://​doi.​org/​10.​1108/​JRIM-​ nical University. The current research and academic work are con-
12-​2019-​0212/​FULL/​XML. cerned with digital signal and image processing, radio monitoring and
Yang, X., and R.E. Smith. 2009. Beyond attention effects: Modeling hydroacoustic monitoring applications, embedded and reconfigurable
the persuasive and emotional effects of advertising creativity. systems, computer vision and machine learning. D. Kaplun regularly
Marketing Science 28 (5): 935–949. https://​doi.​org/​10.​1287/​ takes part in collaborative projects connected with signal processing,
mksc.​1080.​0460. artificial neural networks, and software-hardware implementation of
Ylänne, V., and A. Williams. 2009. Positioning age: focus group dis- digital signal processing algorithms. Author of more than 50 papers in
cussions about older people in TV advertising. International journals and conference proceedings.
Journal of the Sociology of Language. https://​doi.​org/​10.​1515/​
IJSL.​2009.​050. Tatiana Teor got a PhD degree from St. Petersburg State University of
Xiao, L., and M. Ding. 2014. Just the faces: Exploring the effects of Economics and Finance in 1996. She then taught marketing, branding
facial features in print advertising. Marketing Science 33 (3): and corporate social responsibility as an Associate Professor at St.
338–352. https://​doi.​org/​10.​1287/​MKSC.​2013.​0837. Petersburg State Engineering-Economics University, St. Petersburg
Xu, Q., et al. 2020. Deep learning technique based surveillance video State University of Economics and St. Petersburg University of Man-
analysis for the store. Applied Artificial Intelligence 34 (14): agement Technologies and Economics. Since 2020 she works as an
1055–1073. https://​doi.​org/​10.​1080/​08839​514.​2020.​17846​11. Associate Professor of the Department of Public Relations at St. Peters-
Zhao, R., and Y. Cai. 2022. Retracted article: Research on online burg Electrotechnical University “LETI”. T. Teor - author of more than
marketing effects based on multi-model fusion and artifi- 70 publications in the field of marketing, place branding, corporate
cial intelligence algorithms. Journal of Ambient Intelligence branding,corporate social responsibility, and sustainable development.
 L. Sharakhina et al.

Valeria Kulibanova Dr. Sc. (Economics), Professor at the Department Petersburg State University of Economics. Sphere of interests: service
of Tourism and Hospitality Management, RANEPA St Petersburg (the marketing, management of innovations, corporate social responsibil-
branch of the Presidential Academy of National Economy and Pub- ity, sustainable development, regional economy. She has more than 80
lic Administration) and at the Department of International Business, St works, including monographs, textbooks, tutorials.