Asset-driven Threat Modeling for AI-based Systems

Jan von der Assen∗ , Jamo Sharif∗ , Chao Feng∗ , Gérôme Bovet† , Burkhard Stiller∗
∗ Communication
Systems Group CSG, Department of Informatics IfI, University of Zürich UZH
Binzmühlestrasse 14, CH—8050 Zürich, Switzerland
E-mail: [vonderassen, cfeng, stiller]@ifi.uzh.ch, [email protected]
† Cyber-Defence Campus, armasuisse Science & Technology, CH–3602 Thun, Switzerland [email protected]

Abstract—Threat modeling is a popular method to securely One approach that has demonstrated value in the conven-
develop systems by achieving awareness of potential areas of tional application security field is threat modeling, which is
future damage caused by adversaries. The benefit of threat used for secure software development, risk assessment, or to
arXiv:2403.06512v1 [cs.CR] 11 Mar 2024

modeling lies in its ability to indicate areas of concern, paving

the way to consider mitigation during the design stage. However, foster security awareness. Being part of secure development
threat modeling for systems relying on Artificial Intelligence processes (e.g., SSDLC, SAMM), threat modeling is valid
is still not well explored. While conventional threat modeling outside a dedicated cybersecurity context [9], which is critical
methods and tools did not address AI-related threats, research considering that AI system development may be driven by data
on this amalgamation still lacks solutions capable of guiding and scientists and software engineers leveraging services.
automating the process, as well as providing evidence that the
methods hold up in practice. To evaluate that the work at hand is While threat modeling can serve as a key step to identify and
able to guide and automatically identify AI-related threats during mitigate (by means of prevention or response) cybersecurity
the architecture definition stage, several experts were tasked to issues at design time [10], creating suitable threat models is
create a threat model of an AI system designed in the healthcare still a challenge for software engineers and data scientists.
domain. The usability of the solution was well-perceived, and the Multiple reasons can challenge the creation of threat mod-
results indicate that it is effective for threat identification.
Index Terms—AI Security, Cybersecurity, Threat Modeling els for AI systems. In research, wide attention is given to
investigating threats and vulnerabilities from a research per-
spective without proposing practical cybersecurity approaches.
I. I NTRODUCTION
Furthermore, existing threat modeling methodologies and tools
Artificial Intelligence (AI) is considered a disruptive tech- are conceptualized for conventional software systems and,
nology that is being integrated into a myriad of different hence, do not directly support AI threat identification. Recent
domains, ranging from healthcare applications to embedded research [11], [12] addressed how to apply threat modeling
implementations of AI [1], which now serve as a key contribu- for AI. However, this limited body of research has not shown
tor to other technologies such as 6G [2]. Even more impressive how to support or automate the process, especially during the
than the range of domains that see interest in technologies that design phase. Moreover, these approaches were not deployed
can be summarized under the term ”AI” is the speed at which in scenarios involving real users and design problems.
they are adopted. For example, ChatGPT has attracted 100 Thus, the key contribution of this paper is an asset-
million active users per month in just a few weeks [3]. driven threat modeling approach and a guiding tool for said
The fact that AI technologies are now readily available to methodology. The methodology comprises five steps that are
individuals, corporations, and national actors has also given aligned with the design procedures of AI-based systems. To
rise to concern. For example, [4] have analyzed the implica- guide and automate threat identification, existing literature is
tions of Large Language Models (LLMs) in the context of the transformed into a queryable ontology. A stencil library is
Swiss Cybersecurity landscape, summarizing threats such as provided to connect the semantics of the ontology. This allows
spear phishing, vulnerable code injections, and remote code for automated asset and threat identification when AI-based
execution. Furthermore, researchers have demonstrated that system architectures are modeled. Finally, the presented work
these attacks can be executed in a realistic setting [5]. Aside details experimental results demonstrating that (i) the tool can
from LLMs, extensive research has demonstrated weaknesses reproduce a threat model created by cybersecurity experts. For
in related AI technologies, including Machine Learning [6], this, (ii) different types of users were involved in experiments
Federated Learning [7], and Computer Vision [8]. to understand whether non-security personnel can reproduce
It appears that not only the aforementioned hopes but also these results, followed by (iii) a qualitative investigation of
the concerns of these technologies are rightly part of current the tool’s perceived usability. Overall, the tool can guide and
discussions. However, it is vital to consider that the adop- automate threat modeling for AI, effectively reproducing threat
tion is ongoing – organizations are actively integrating these models with acceptable usability when used by data scientists.
technologies into their products and services. This raises the This paper is organized as follows. Section II presents an
question of how organizations should approach these security overview of related literature. While Section III details the
concerns, especially given the scarcity of cybersecurity talent design, evaluations are described in Section IV. Conclusions
and the speed at which AI services are integrated. and directions for future work are outlined in Section V.
 II. BACKGROUND AND R ELATED W ORK on AI, and more specifically, LLMs [32]. A comprehensive
Literature related to this work can be grouped into three report of AI-related threats is presented by the European Union
segments: (i) research identifying adversarial attacks on AI, Agency for Network and Information System [33] (ENISA),
(ii) established threat modeling tools and methods, and (iii) a which are further related to architectural AI assets in [34].
small body of literature looking into the combination of the The third and most closely related literature group reports
former two. Due to the lack of research on AI threat modeling, evidence of integrating the AI paradigm within threat model-
painting a realistic picture of the problem domain requires a ing. The limited number of publications [15] (see TABLE I)
summary of research in all three areas. connect potential risks to the elements generated throughout
A recent survey organizes cyber attacks on AI systems various phases of the life cycle of ML models, ranging from
according to the Machine Learning (ML) pipeline. During data the initial requirements analysis to maintenance.
collection and preprocessing, data poisoning attacks influence [12] applies conventional threat modeling consisting of data
the resulting model by injecting samples. These may be falsi- flow diagramming and STRIDE-based threat identification.
fied in the data source or the database used for collection [16], While the methodology reports the successful mapping of a
[17]. The goal of the attack may vary [18], [19], [17] and threat taxonomy to an illustrative model, the mapping process
multiple poisoning strategies (e.g., random or targeted data is carried out manually by experts. Furthermore, limitations
manipulation and injection) exist [20]. Spanning the feature such as limited results from a singular synthetic case study
selection and model training stages, several strategies are and no investigation on usability are acknowledged.
identified that can replace the model with a poisoned one [17], In [13], a gold standard dataset is used to evaluate the
[21]. During the inference stage, attacks achieving model degradation of a model during the productive stage. A metric
inversion, inference, and failure are described. Model inversion is proposed that quantifies the degradation loss, which could
aims to recover information on the training samples [17], while quantify the impact of a threat. However, focusing on the ex-
extraction attacks attempt to obtain or reconstruct the model isting models might indicate that the method is not applicable
based on limited access [17], [22]. during the design stage, which is critical for threat modeling.
Looking into threat modeling tools and methods, none of A domain-specific threat model is created in [14], focusing
the popular tools such as the Microsoft Threat Modeling Tool, on Open Radio Access Network (O-RAN) architectures. Thus,
CAIRIS, Threatspec, SDElements, or Tutamen focus on threat no generic approach is evaluated. The paper by [15] is the
modeling for AI systems [23], [24], [25], [26]. Although most closely related contribution to threat modeling of AI-
some, such as CAIRIS, present the ability to create custom based systems. It advocates for integrating threat modeling
threat libraries, no taxonomies or support for an AI-related methodologies in AI security analysis and introduces the
method are present. Furthermore, many of the tools are non- STRIDE-AI methodology, a tailored adaptation of the STRIDE
free or closed-source software. It is unclear how well they framework for ML systems. The methodology assigns ML-
are understood outside the security domain. Here, it appears specific interpretations to security properties, facilitating the
that there is a tradeoff between flexibility and guidance. For identification of threats to AI assets. However, it involves a
example, diagrams.net [27] is popular in threat modeling due manual mapping process, lacking automation, which hinders
to its flexibility and widespread familiarity [10]. In such an scalability and adaptability to system changes. The method-
example, STRIDE, a mnemonic-based brainstorming method, ology’s evaluation is based on a single use case without
could be applied to the AI domain at the expense of requiring the involvement of participants, providing insights but not
users to survey and relate threats to the system manually. covering all challenges in diverse ML applications.
In this context, related activities from the industry can be In summary, while one might argue that attacks on AI are
introduced to highlight ongoing efforts in the field. MITRE, not radically different from conventional cyber attacks, it is
is a well-known catalogue of malicious techniques [28]. As not clear how straightforward the creation of a threat model
a complementary knowledge framework, MITRE ATLAS in- for AI is. More specifically, the guiding factors and the degree
cludes tactics that are specific to AI [29]. Another knowledge of automation, especially when creating a threat model by real
base that provides a guideline for the mitigation of AI threats users during the design stage of a system, is unclear, posing
was proposed by Microsoft [30], [31]. Similarly, OWASP has an opportunity for the development of a guiding tool oriented
presented guides to ensure the security of systems relying towards the design process of AI system architectures.

TABLE I
L ITERARY W ORK A PPLYING T HREAT M ODELING TO AI S YSTEMS

Work Contribution Evaluation Domain

[12] [2020] Method, Survey Illustrative Case Study Security Requirements Engineering
[13] [2021] Degradation Quantification Method Demonstration Adversarial Machine Learning
[14] [2022] Threat Model Demonstration Cellular Networks
[15] [2022] Methodology Illustrative Case Study Threat Modeling of AI Systems
ThreatFinderAI [2024] Open-source Tool, Methodology Field Study Threat Modeling of AI Systems
 III. D ESIGN AND I MPLEMENTATION In the second step, the system must be closely analyzed
To design and implement a threat modeling approach for and understood from an architectural perspective. For this,
AI-based systems, it is inevitable to map the architectural the context for the threat modeling process is essential –
semantics of these systems to the threat modeling process. The whether for the design of a completely new architecture or
architecture of the ThreatFinderAI prototype is visualized in a threat model is created for an existing system as part of
Fig. 1. At the top, a high-level overview of the threat modeling a risk assessment. In this work, ThreatFinderAI relies on
procedure is outlined. This process was leveraged in order visual modeling of architectures. Hence, it is crucial to verify
to design the individual components that support the overall whether there are existing system diagrams and models. In
process and, in sum, provide automated threat modeling for AI any case, modeling and drawing the architecture of the AI-
systems. The architectural components are shown in the center, based system can help in comprehension. Here, it is essential
consisting of six key components supporting the approach of to draw a holistic picture of the architecture, for which the
this work. To investigate the feasibility and effectiveness of guiding model of the AI life cycle from [34] can be helpful to
that approach, a prototype was designed and implemented, as elicit all activities and the systems involved. For example, even
presented in the bottom. when using a pre-trained model, it is important to draw the
data collection procedure to capture the whole attack surface,
even though a service provider may perform it transparently.
TABLE II
G ENERIC 5-S TEP T HREAT M ODELING P ROCESS [35], [15]
In the third step, the architecture model serves as a means to
identify relevant assets. Conceptually, these are the functional
Goal Description
and data assets that are subject to the security goals. For
1 Objective Identification Determine system security goals
2 Assessment Identify system assets and interactions
example, if a healthcare project postulates that the confi-
3 Decomposition Select relevant assets dentiality of the data is paramount, then it is vital that the
4 Threat Identification Categorize threats to assets training data (among other assets) is identified. As already
5 Identify Vulnerability Analyze threats and determine vulnerability
mentioned, ThreatFinderAI takes a visual approach since it
is assumed to be well-established for the development of
From the methodological perspective, the generic 5-step software architectures. To solve this problem, stencils can
threat modeling process summarized in TABLE II was lever- support the annotation of software architecture diagrams – if
aged, allowing for a step-by-step approach to address the each element is carefully annotated with metadata to identify
specific problem domain. For the objective identification step, a unique asset from a taxonomy, the diagrams can be analyzed
literary analysis revealed the necessity to adopt the AI- in an automated manner. In the fourth step, the set of assets is
specific proposal of security principles from [15], [34]. There, used as an input to identify threat events that can impact those
the traditional CIA principles (i.e., confidentiality, integrity, assets. For example, the presence of training data which is
availability) are extended to include authorization and non- managed by an untrusted actor, could indicate the vulnerability
repudiation as key concepts. While the definition of important of the system to a data poisoning attack. As will be discussed
security goals may not be fruitful at this stage, it is crucial for the implementation of a prototype, it is critical to consider
to ensure that the business relevance of the system to be the literature for this step since architects may not have the
developed is well understood [36]. resources to develop or research novel threats on their own.

METHODOLOGY
REQUIREMENTS ARCHITECTURE ASSET THREAT THREAT THREAT
ANALYSIS MODELING IDENTIFICATION IDENTIFICATION ANALYSIS MITIGATION

PROPERTY AI ASSET DIAGRAM THREAT

FILTERS STENCILS PARSER REPORTS

DIAGRAM AI THREAT
EDITOR ONTOLOGY
ARCHITECTURE

/home: UPLOAD, diagrams.net (72) ANNOTATED /results: display RESTful API ASSET TAXONOMY PDF EXPORT
FILTER WRAPPER XML STENCILS
localStorage DIAGRAM PARSER THREAT TAXONOMY

REACT FRONTEND FASTAPI BACKEND

PROTOTYPE

Fig. 1. Architecture of the ThreatFinderAI Approach

 In the final steps, the list of threats are analyzed, potentially After the users freely and interactively model the system
revealing threats whose impact cannot be accepted or ignored architecture, the diagram can be transferred to the Python
by the risk sensitivity of the surrounding business context. backend, where it is analyzed to suggest relevant threats
These steps require the adoption of technical, organizational, automatically. To do so, the diagram is exported from the
or strategic mitigation controls. This step could be guided and editor and sent to the backend as an XML file over HTTPS.
automated by specific guidelines. However, the elicitation of There, it is first parsed to retrieve all visual elements from
security controls is out of scope for threat modeling. the diagram. For all elements carrying annotations of the
stencil library, the contextualized assets are extracted using
A. Prototype Implementation a JSON representation of the ontological AI asset knowledge
To implement the components proposed for the ThreatFind- base extracted from the ENISA report. This knowledge base
erAI threat modeling approach, the components outlined at the is semantically connected to another knowledge base, which
bottom of Fig. 1 were implemented and integrated into a web- holds 96 formalized threat descriptions, each contextualized
based solution. Starting from the front end, the user interacts with the connecting AI asset, category, title, description, and
with a web-based graphical user interface implemented as a the potential impact on the security objectives. Thus, based
Single-page Application (SPA) using React.js [37]. First, on on the derived assets, the knowledge base is queried, and a
the /home page, the user uploads a previously developed threat report is generated and returned to the client, detailing
diagram and selects the essential security properties for the for each asset a set of potentially relevant threats.
business case. If no diagram exists, another page hosts the in-
In the front end, the threat report is displayed. Here, the
teractive diagram editor component. This component consists
initial selection of the security goals comes into play by
of an inline HTML frame loading the diagrams.net diagram
focusing the reporting on threats that primarily relate to this
editor [27]. Aside from the functionality inherited by the
objective. Given that threat modeling is a collaborative and
editor, the web application communicates with the frame to
interactive endeavor requiring collaboration with the business
pre-load the stencil libraries as well as automatically load and
and technical stakeholders, the final threat model can be ex-
continuously store the model in the browser’s storage.
ported in PDF form. Moreover, the threat model diagrams can
Furthermore, a bespoke stencil library was crafted to sim-
be exported and extended later on, using the ThreatFinderAI
plify and guide the asset modeling stage. The stencil library
tool or by uploading it to draw.io, hence providing the freedom
provides one stencil for each asset identified from the com-
to reuse the models in other tools. For example, another
prehensive report provided by ENISA [33], which serves to
component of a toolkit could leverage these models to quantify
build an extensive yet extensible ontology that gives detail
the threats or to suggest control mechanisms.
on threat events, actors, and assets. In total, 72 stencils are
formalized into an XML file, allowing the annotation of Since the ThreatFinderAI prototype only relies on a small
metadata to analyze the resulting diagram automatically. As number of software dependencies (i.e., diagrams.net, React,
partially shown in the left modal of Figure 2, the stencils are FastAPI) that are all open-source, the full source code is made
grouped into six categories, encouraging modeling not only publicly available [38], as well as providing a running version
static software elements but also processes and actors. of the tool available online [39].

Fig. 2. Front end of ThreatFinderAI: Architectural Modeling and Asset Annotation

 IV. E VALUATION relevant threats. In total, ten areas of concern were identified
Developing ThreatFinderAI had the goal of investigating and closely investigated. Here, it is important to state that
whether a supporting tool can guide and automate the threat all four experts had to rely on external threat information,
identification stage when modeling threats around AI-based which was surveyed manually in the form of reports and
systems. To assess the effectiveness of ThreatFinderAI, the academic literature. Based on this, potential threats within
problem of a lacking ”perfect” model arises against which the system were then identified and linked to specific threat
the tool’s output can be tested. Furthermore, it must be actors, including malicious platform users, external threats,
acknowledged that threat modeling is, in practice, still highly infrastructure administrators, and automated external entities
centered around humans developing threat models to create like malware. These identified threats span a broad spectrum
value [40]. Thus, the main question guiding the evaluation of security concerns. The experts have pinpointed 44 threats
of ThreatFinderAI was whether experts with only limited throughout the system, with certain threats recurring across
cybersecurity expertise could identify relevant threats in a multiple components. Here, it’s important to highlight that
practical scenario and how they perceive the usability of doing the ThreatFinderAI’s database, containing 96 distinct threats,
so. covers all the threats identified by the experts and extends them
to more specific threats, offering a comprehensive overview,
A. Methodology which is often desired, such as when building attack trees [41].
Thus, a scenario-driven field experiment was conducted. In the second step, seven participants took part in a threat
First, a threat model is created by cybersecurity and data sci- modeling workshop using ThreatFinderAI, receiving a video-
ence experts working together to develop a secure architecture based tutorial on how to use the tool and information about
for an AI system in the medical field. Although it cannot the previously described platform architecture. Although there
be proven that the expertise of the experts leads to modeling certainly is a complexity involved in transferring knowledge
threats that are relevant and sufficiently exhaustive, this model from the scenario to the participants, the experiment was
provides a baseline against which the threat model created by executed to control the architecture used for threat identifi-
non-security experts leveraging the tool in the second step. cation. Furthermore, in a threat modeling workshop, it is not
The practical context of the model that was created in unrealistic to see a transfer of expertise involved, for example,
both steps was the ongoing development of a platform to when a team of data science software architects is tasked to
collect, store, share, and train models from data. The plat- evaluate an existing system. After applying the tool to the
form is a healthcare platform tailored for clinical data anal- scenario, the threat models were collected, and the participants
ysis by engineers, practitioners, and researchers. It aims to were guided through a questionnaire to understand how the
provide a robust data-gathering system with controlled data usage of ThreatFinderAI is perceived.
synthesis to facilitate experimentation and modeling in this
healthcare domain. The platform prioritizes data privacy and TABLE III
security by incorporating advanced anonymization techniques, PARTICIPANTS E DUCATIONAL BACKGROUND AND AI KNOWLEDGE
attribute-based privacy measures, and reliable tracking sys-
# Educational Background AI Knowledge
tems. The main functionalities of the platform are organized
into three primary modules (i.e., Model Training, Model Au- 1 Master of Data Science Practical experience
2 Master of Data Science Practical experience
ditor, and Data Synthesizer), seven supporting modules (i.e., 3 Bachelor of Science Software Systems Theoretical knowledge
Data Anonymization Toolkit, Data Uploader, Cross-Borders 4 Bachelor of Science Software Systems Theoretical knowledge
Database, Dataset Explorer, Dataset Builder, Dataset Evalua- 5 Bachelor of Science Information Systems Little to no understanding
6 Master of Science Pharmacy Little to no understanding
tor, and Federated Learning), and three crosscutting modules 7 Master of Law Little to no understanding
(i.e., Security Control, User Interface, and Orchestrator), ad-
dressing diverse needs and requirements within the healthcare
analytics landscape. As one might expect from the presence At the beginning of the questionnaire, the background and
of anonymization technology, data confidentiality, and privacy expertise of the participants were targeted. As visible from
are of utmost importance to business representatives. Not only TABLE III, participants from different backgrounds were
may data breaches lead to regulatory fines, but the overall trust selected (assessed by the highest completed academic degree).
in the data-sharing platform is a crucial property to stimulate Notably, none of the participants indicated knowledge of
data collection from various parties. cybersecurity and only two out of five participants with a
Computer Science degree majored in Data Science. These
B. Execution participants were also the only ones who stated to have
In the first stage, four experts collaborated to create a practical knowledge of working with AI, while the remaining
threat model of the platform architecture: two with conven- considered themselves to be theoretically knowledgeable about
tional cybersecurity expertise, one with specific AI security AI system architecture.
knowledge, and the fourth one with a data science back- Next, it was investigated whether the design assumption
ground. To do so, they leveraged diagrams.net [27] to draw that computer scientists are already familiar with the diagram
the system architecture, its boundaries, potential actors, and editor diagrams.net was justified. Based on statements shared
on a Likert scale, all participants with a technical background that even a layperson, Participant Six, achieved this, while
considered themselves familiar with the tool. In a similar Participants Four, Five, and Seven missed multiple threats.
question on related tools such as Microsoft Threat Modeling The participants with a background in data science, there-
Tool or OVVL, none of the participants expressed familiarity. fore, discovered all relevant threats (and more granular vari-
To understand the participants’ perceived ability to use ants) from the expert-based model. While the application is
the tool, additional questions investigated this aspect. All effective in that scenario, the efficiency of the application
participants felt successful in navigating the tool. Participant still suffers from the common concern of yielding a signif-
Six faced challenges during the asset identification step, ac- icant number of false positives compared to the 44 threats
knowledging a limited understanding of AI technology from discovered by the experts. Thus, the overall threat modeling
an architectural perspective. When rating the clarity of the procedure still requires a discussion of the threat identification
task instructions, six out of seven considered them at least step, as is the case in the established threat modeling processes.
sufficiently clear. With respect to the concrete scenario and In summary, participants effectively identified all potential
architecture provided, three participants expressed it was easy threats using the architectural model, addressing previously
to understand, while the remaining four expressed it was identified gaps in the literature. However, efficiency could be
sufficiently understandable. improved by further filtering threats, potentially through sub-
categories of security objectives and requirements. Moreover,
TABLE IV exploring participants’ prioritization of threats and inferred
S YSTEM U SABILITY S CORE PER PARTICIPANT vulnerabilities could offer valuable insights. However, this was
beyond the scope of the current study and remains a challenge
# Educational Background Score
for future research. From the experiment, it can be observed
1 Master of Data Science 55
2 Master of Data Science 70
that practitioners with a data science background could use this
3 Bachelor of Science Software Systems 85 tool as guidance for an initial threat identification step. The
4 Bachelor of Science Software Systems 52.5 resulting threat model would require further analysis either by
5 Bachelor of Science Information Systems 52.5
6 Master of Science Pharmacy 75
leveraging additional tools or by involving security experts.
7 Master of Law 45
V. S UMMARY AND F UTURE W ORK
Average 62.14
Due to the necessity of considering the architectures and
Concluding the questionnaire, the perceived usability was security concerns of AI-based systems in current system engi-
assessed by means of the system usability scale (SUS), neering, this paper proposes ThreatFinderAI. The approach
which provides a simple, standardized scoring using ten ques- aligns the AI security domain with the established threat
tions [42]. The resulting scores are shown in TABLE IV. modeling process through a guiding prototype. The prototype
When all participants are included, the average score evaluates includes a front end to collect the most relevant security ob-
to acceptable usability. Looking into participants One to jectives and enables architectural diagramming with a bespoke
Five, who all hold computer science expertise, Participant stencil library of AI assets. In the back end, the diagrams
One’s score stands out negatively. Although SUS in itself is are automatically analyzed against an established AI security
not diagnostic, the answers provided appear conflicting. For report, which is transformed into a computable ontology.
example, the participant indicated that the tool might require To understand the practicability, effectiveness, and usability
assistance from a technical person while also being easy to of the prototype, a user-centric experiment confronted real
learn. Based on additional open feedback collected, Participant users with a real-world AI system architecture. The results
One expressed a dislike for the diagram editor leveraged in show the feasibility of an AI threat modeling approach and
ThreatFinderAI, while otherwise positively acknowledging the prototype, but also the effectiveness of non-security experts
features of ThreatFinderAI. identifying threats. However, the results from the experiment
also demonstrate that threat modeling may not be trivial
C. Analysis for practitioners without cybersecurity expertise. Specifically,
Since the threat modeling process is guided and the key further research is needed to improve the usability and to
steps are automated, artifacts, such as the annotated archi- support the final stages of threat analysis, where threats are pri-
tecture diagrams and the resulting tabular threat models, were oritized and mitigations are designed. Furthermore, although
recorded and analyzed by the experts who conducted the initial the results are obtained from practitioners working on a real
analysis. In the security objective selection, it was observed system architecture, their expressiveness is limited by the small
that both the participants and the expert group mostly agreed number of participants working on a singular case.
on the relevant properties. In the future, the usability and effectiveness will be further
Based on the expert assessment, Participants One, Two, assessed and tested with a broader set of participants and
Three, and Six successfully identified all relevant threats, un- scenarios. Moreover, to improve the threat analysis stages,
derscoring the advantage of AI knowledge in threat modeling the applicability of threat interpretation and quantification
for AI-related systems. While this does not come as a surprise, approaches from established domains such as cybersecurity
given that they are the most likely target group, it is surprising economics or risk management will be investigated.
 ACKNOWLEDGMENTS [18] L. Muñoz-González, B. Biggio, A. Demontis, A. Paudice, V. Wongras-
samee, E. C. Lupu, and F. Roli, “Towards Poisoning of Deep Learning
This work has been partially supported by (a) the Swiss Algorithms with Back-gradient Optimization,” in Proceedings of the
Federal Office for Defense Procurement (armasuisse) with the 10th ACM workshop on artificial intelligence and security, 2017, pp.
CyberMind and RESERVE (CYD-C-2020003) projects and (b) 27–38.
[19] B. Nelson, M. Barreno, F. J. Chi, A. D. Joseph, B. I. Rubinstein, U. Saini,
the University of Zürich UZH. C. Sutton, J. D. Tygar, and K. Xia, “Exploiting Machine Learning to
R EFERENCES Subvert Your Spam Filter,” LEET, vol. 8, no. 1-9, pp. 16–17, 2008.
[20] Ilmoi, “Poisoning attacks on Machine Learn-
[1] L. Perri, Gartner Inc., “What’s New in Artifi- ing,” July 2019, https://towardsdatascience.com/
cial Intelligence from the 2023 Gartner Hype Cy- poisoning-attacks-on-machine-learning-1ff247c254db/, Last Visit
cle,” August 2023, https://www.gartner.com/en/articles/ January 2024.
what-s-new-in-artificial-intelligence-from-the-2023-gartner-hype-cycle, [21] J. Natarajan, “Cyber Secure Man-in-the-Middle Attack Intrusion De-
Last Visit January 2024. tection Using Machine Learning Algorithms,” in AI and Big Data’s
[2] Nokia Corporation, “6G explained,” January 2024, https://www.nokia. Potential for Disruptive Innovation, January 2020, pp. 291–316.
com/about-us/newsroom/articles/6g-explained, Last Visit January 2024. [22] R. N. Reith, T. Schneider, and O. Tkachenko, “Efficiently Stealing your
[3] K. Hu, Reuters, “ChatGPT sets record for fastest-growing user base Machine Learning Models,” in Proceedings of the 18th ACM Workshop
- analyst note,” February 2023, https://www.reuters.com/technology/ on Privacy in the Electronic Society, 2019, pp. 198–210.
chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/, [23] CAIRIS, “Threat Modelling, Documentation and More,” 2022, https:
Last Visit January 2024. //cairis.org/cairis/tmdocsmore/, Last Visit January 2024.
[4] A. Kucharavy, Z. Schillaci, L. Maréchal, M. Würsch, L. Dolamic, [24] Threatspec, “Threatspec,” June 2019, https://threatspec.org/, Last Visit
R. Sabonnadiere, D. P. David, A. Mermoud, and V. Lenders, “Fun- January 2024.
damentals of Generative Large Language Models and Perspectives in [25] SecurityCompass, “SD Elements Datasheet v5.17,” 2023, https://docs.
Cyber-Defense,” arXiv preprint https://arxiv.org/abs/2303.12132, March sdelements.com/release/latest/guide/docs/datasheet.html/, Last Visit Jan-
2023. uary 2024.
[5] P. Dixit, engadget, “A ’silly’ attack made Chat- [26] Tutamantic, “Feauture — Tutamantic,” January 2021, https://www.
GPT reveal real phone numbers and email ad- tutamantic.com/page/features, Last Visit January 2024.
dresses,” November 2023, https://www.engadget.com/ [27] JGraph Ltd, “Diagram Software and Flowchart Maker,” https://www.
a-silly-attack-made-chatgpt-reveal-real-phone-numbers-and-email-addresses-200546649.
diagrams.net/, Last Visit January 2024.
html, Last Visit January 2024. [28] The MITRE Corporation, “MITRE ATT&CK (Adversarial Tactics,
[6] X. Wang, J. Li, X. Kuang, Y. an Tan, and J. Li, “The security of machine Techniques, and Common Knowledge),” https://attack.mitre.org/, Last
learning in an adversarial setting: A survey,” Journal of Parallel and Visit January 2024.
Distributed Computing, vol. 130, pp. 12–23, 2019. [29] ——, “MITRE ATLAS (Adversarial Threat Landscape for Artificial-
[7] L. Lyu, H. Yu, J. Zhao, and Q. Yang, Threats to Federated Learning. Intelligence Systems),” https://atlas.mitre.org/, Last Visit January 2024.
Cham: Springer International Publishing, 2020, pp. 3–16. [30] Microsoft Corporation. Threat Modeling for AI/ML Systems and
[8] N. Akhtar and A. Mian, “Threat of Adversarial Attacks on Deep Dependencies. https://learn.microsoft.com/en-us/security/engineering/
Learning in Computer Vision: A Survey,” IEEE Access, vol. 6, pp. threat-modeling-aiml, Last Visit January 2024.
14 410–14 430, 2018. [31] A. Marshall, J. Parikh, E. Kiciman, and R. Kumar, “Threat Modeling
[9] OWASP, “Software Assurance Maturity Model,” September 2023, https: AI/ML Systems and Dependencies,” Security documentation, 2019.
//owasp.org/www-project-samm/, Last Visit January 2024. [32] OWASP, “AI Security and Privacy Guide,” https:
[10] von der Assen, J. and Franco, M. F. and Killer, C. and Scheid, E. J. and //owasp.org/www-project-ai-security-and-privacy-guide/
Stiller, Burkhard, “CoReTM: An Approach Enabling Cross-Functional #how-to-deal-with-ai-security, Last Visit January 2024.
Collaborative Threat Modeling,” in IEEE International Conference on [33] European Union Agency for Cybersecurity (ENISA), “Securing Machine
Cyber Security and Resilience (CSR 2022), Rhodes, Greece, July 2022, Learning Algorithms,” 2021.
pp. 1–8. [34] ——, “Artificial Intelligence Cybersecurity Challenges, Threat Land-
[11] L. Mauri and E. Damiani, “Stride-ai: An approach to identifying scape for Artificial Intelligence,” 2020.
vulnerabilities of machine learning assets,” in 2021 IEEE International [35] S. Myagmar, A. J. Lee, and W. Yurcik, “Threat Modeling as a Basis for
Conference on Cyber Security and Resilience (CSR). IEEE, 2021, pp. Security Requirements,” in Symposium on Requirements Engineering for
147–154. Information Security (SREIS), August 2005.
[12] C. Wilhjelm and A. A. Younis, “A Threat Analysis Methodology for [36] M. F. Franco, F. Künzler, J. von der Assen, C. Feng, and B. Stiller,
Security Requirements Elicitation in Machine Learning Based Systems,” “RCVaR: an Economic Approach to Estimate Cyberattacks Costs using
in 2020 IEEE 20th International Conference on Software Quality, Data from Industry Reports,” arXiv preprint https://arxiv.org/abs/2307.
Reliability and Security Companion (QRS-C), 2020, pp. 426–433. 11140, July 2023.
[13] L. Mauri and E. Damiani, “Estimating Degradation of Machine Learning [37] Meta Platforms, “React,” https://react.dev/, Last Visit January 2024.
Data Assets,” ACM Journal of Data and Information Quality (JDIQ), [38] Sharif Jamo, “AiThreats,” 2024, https://github.com/JSha91/AiThreats.
vol. 14, no. 2, pp. 1–15, 2021. [39] Sharif Jamo and von der Assen Jan, “ThreatFinder,” 2024, https://www.
[14] E. Habler, R. Bitton, D. Avraham, D. Mimran, E. Klevansky, O. Brodt, csg.uzh.ch/threatfinder/.
H. Lehmann, Y. Elovici, and A. Shabtai, “Adversarial Machine Learning [40] Threat Modeling Manifesto Working Group, “Threat Modeling Man-
Threat Analysis and Remediation in Open Radio Access Network (O- ifesto,” January 2024, https://www.threatmodelingmanifesto.org, Last
RAN),” arXiv preprint https://arxiv.org/abs/2201.06093, March 2023. Visit January 2024.
[15] L. Mauri and E. Damiani, “Modeling threats to AI-ML systems using [41] A. Shostack, Threat Modeling: Designing for Security. John Wiley &
STRIDE,” Sensors, vol. 22, no. 17, 2022. Sons, 2014.
[16] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, [42] GitLab Inc., “System Usability Scale (SUS),” 2023, https:
“Communication-Efficient Learning of Deep Networks from Decentral- //handbook.gitlab.com/handbook/product/ux/performance-indicators/
ized Data,” in Artificial intelligence and statistics, 2017, pp. 1273–1282. system-usability-scale, Last Visit January 2024.
[17] R. S. Sangwan, Y. Badr, and S. M. Srinivasan, “Cybersecurity for AI
Systems: A Survey,” Journal of Cybersecurity and Privacy, vol. 3, no. 2,
pp. 166–190, 2023.