Global Insight and Best

Practices for
Financial Crime Compliance
Model Validation
August 2024
Peter Weitzman, Michael Lovejoy, Sky Hsu, Tony Eddington
Executive Summary

Global regulators unanimously require regulated entities to implement and maintain effective systems and
controls to mitigate financial crime risks. The systems and controls should enable the regulated entities to
identify, assess, monitor, and mitigate money laundering and terrorist financing risks. It is expected that
these systems and controls are comprehensive, as well as proportionate to the nature, scale and complexity
of its business activities.1 2 Additionally, regular assessments should be performed to ensure these systems
and controls remain adequate and fit for purpose.3 4
Within the financial crime systems are various models that can help automate transaction monitoring, sanc-
tions screening, and customer risk rating. Using quantitative and SME driven approaches, Financial Crime
Compliance (FCC) models identify the financial crime risks presented by the customers, counterparties, or
third-party vendors, as well as the risks deriving from the regulated entities’ products and services. 5
Regular validation ensures that the FCC models remain aligned and relevant to the regulatory landscape, the
ever-changing business activities, and evolving technology developments such as the use of Artificial Intelli-
gence and Machine Learning techniques. Key aspects of FCC model validation include the conceptual sound-
ness of the model, ongoing monitoring, outcomes analysis, and comprehensive data testing.
This article explores what you can expect during a model validation and considerations unique to each type
of model and how you can get the most out of a model validation.

What is a model?
The Prudential Regulation Authority (PRA) of the United Kingdom, in the supervisory statement (SS) pub-
lished in May 2023, indicated that a model “is a quantitative method, system, or approach that applies statisti-
cal, economic, financial, or mathematical theories, techniques, and assumptions to process input data into out-
put. The definition of a model includes input data that are quantitative and/or qualitative in nature or expert
judgement-based, and output that are quantitative or qualitative.”6
Traditional credit and market risk models have a significant emphasis on using structured financial metrics
(e.g., loan performance, market prices, and interest rates) to generate model output that predicts risks such
as default probabilities, Value-at-Risk (VaR), and interest rate risks. These models are quantitative in nature,
in that they rely on numerical inputs and mathematical formulas to generate numerical outputs.
On the other hand, FCC models are intended to identify suspicious activities indicative of financial crime.
Some common FCC models include Transaction Monitoring (TM), Sanctions Screening, and Customer Risk
Rating (CRR). TM and CRR models use algorithms and statistical techniques to develop customer clusters and

1
Systems and Controls (SYSC) Handbook 3.2.6A R., https://www.handbook.fca.org.uk/handbook/SYSC/3/2.html#DES69
2
Directive (EU) 2015/849, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:32015L0849
3
Systems and Controls (SYSC) Handbook 3.2.6C R., https://www.handbook.fca.org.uk/handbook/SYSC/3/2.html#DES69
4
EBA Guidelines 2021/02, https://www.eba.europa.eu/sites/default/files/document_library/Publications/Guidelines/2023/EBA-GL-2023-
03/1061654/Guidelines%20ML%20TF%20Risk%20Factors_conslidated.pdf.pdf
5
Financial Crime covers a broad spectrum of risks, including fraud, market misconduct, handling proceeds of crime, and financing of terrorism. This paper
focuses on the most common models that facilitate the identification and mitigation of money laundering and terrorist financing risks, namely, the Trans-
action Monitoring, Sanctions Screening, and Customer Risk Rating models (collectively, the "Financial Crime Compliance models (FCC models)").
6
SS1/23, Model risk management principles for banks, Bank of England PRA, https://www.bankofengland.co.uk/-/media/boe/files/prudential-regula-
tion/supervisory-statement/2023/ss123.pdf

Public. Copyright © 2024 Capgemini. All rights reserved. 2

identify unusual customer activity aligned to known risk typologies. Sanctions Screening models use pattern
matching techniques to identify potential nexus to sanctioned subjects.
FCC models encapsulate quantitative aspects such as thresholds or fuzzy logic settings, as well as Subject
Matter Expert considerations, such as scenario selection and appropriate risk coverage assessments. The
decisions on the weighting of applicable risk factors and relevant red flags are based on both statistical anal-
ysis and expert input. The advantage of this type of hybrid model over a purely statistical or mathematical
calculator is the ability to fine-tune parameters and thresholds, specific to your risk appetite, within a defined
set of rules.

What is model risk management?

Model risk is the possibility that a model is generating inaccurate results that can lead to incorrect decision-
making. Model risk occurs primarily due to invalid input data, incorrect or inappropriate use of the model, or
inaccurate or inefficient output, when viewed against the design and business objectives.
The PRA’s supervisory statement SS1/23 recognised model risk as a risk in its own right and outlined five
model risk management principles that underpinned the regulatory expectations for banks. These principles
encompass the model lifecycle, including the identification and classification of models, model governance,
development, implementation, and model use. The supervisory statement also requires an independent val-
idation process to ensure models continue to be suitable for their intended use, as well as using model risk
mitigants when the models are underperforming. The key to complying with the independence requirement
is to ensure the process is conducted by teams independent of the model development and implementation
processes. This ensures an unbiased and objective assessment of the model's accuracy and performance and
avoids potential conflicts of interest.
It should also be noted that in the policy statement (PS) that PRA issued subsequently, the role of model risk
management was expanded to include “material deterministic quantitative methods such as decision-based
rules or algorithms.” These may not be classified as models under some definitions but should be subject to
similar scrutiny as financial crime and anti-money laundering systems generally involve highly complex quan-
titative calculations and could have a material bearing on a regulated entity’s business decisions. 7
One guiding principle for managing model risk is “effective challenge,” which encompasses critical analysis
by objective, informed parties who can identify model limitations and assumptions. While model risk cannot
be entirely eliminated, it can be more effectively managed through limiting model use, regular tuning over
time, and strong model governance. A sound model validation process is critical to your ability to objectively
assess and mitigate your model risk.

Minimum requirements and best practices in model validation

Model validation is subject to limited, but increasing, regulatory guidance, including from global regulatory
bodies. The Systems and Controls (SYSC) Handbook 8 published by the United Kingdom’s Financial Conduct
Authority (FCA) and guidelines issued by the European Banking Authority (EBA)9 based on the European

7
PS6/23 – Model risk management principles for banks, https://www.bankofengland.co.uk/prudential-regulation/publication/2023/may/model-risk-man-
agement-principles-for-banks?utm_source=Bank+of+England+updates&utm_campaign=f881dbdb03-EMAIL_CAMPAIGN_2023_05_17_08_25&utm_me-
dium=email&utm_term=0_-f881dbdb03-%5BLIST_EMAIL_ID%5D
8
Systems and Controls (SYSC) Handbook 3 Systems and Controls, and 6 Compliance, Internal Audit and Financial Crime, https://www.hand-
book.fca.org.uk/handbook/SYSC/
9
EBA Guidelines 2021/02, https://www.eba.europa.eu/sites/default/files/document_library/Publications/Guidelines/2023/EBA-GL-2023-
03/1061654/Guidelines%20ML%20TF%20Risk%20Factors_conslidated.pdf.pdf

Public. Copyright © 2024 Capgemini. All rights reserved. 3

Union’s Directives 10 outline the minimum regulatory expectations for robust systems and controls to man-
age financial crime risks. Similarly, the United States Federal Reserve and the Office of the Comptroller
of Currency (OCC) issued Supervisory Guidance on Model Risk Management (FRB SR 11-7 and OCC 2011-12)
in 2011 elaborating additional requirements on model risk management and key components of model vali-
dations.11
The industry’s best practices, as echoed by the Joint Money Laundering Steering Group (JMLSG) guidance,
are to ensure the systems and controls put in place are tailored to the level of Money Laundering/Terrorist
Financing risks presented by customers, products and services, delivery channels, as well as the jurisdictions
in which the regulated entities operate. Best practices also include regular assessments of the adequacy of
systems and controls to ensure effective management of money laundering risks. 12
The summary below combines the relevant regulatory guidance with recommendations based on Capgem-
ini’s collective Subject Matter Expert experience and the industry’s best practices.

Regulatory Guidance Capgemini Perspective

Model Model governance is critical for effective We recommend that roles and responsibili-
Governance model risk management, especially for FCC ties for the development, maintenance (i.e.,
models, which typically receive a higher change management), use, and retirement of
model risk rating and require more scrutiny FCC models are documented and reviewed
periodically to ensure the effectiveness and
Regulated entities must have documented
accuracy of the process.
risk management policies and risk profiles,
as well as evidence of the application of Requirements of regular model validation
these policies. Information related to the (e.g., the review frequency, the acceptance
operations and effectiveness of the model criteria, and reporting and escalation proto-
should be regularly, or at a minimum, once cols) should be specified in the Model Gov-
a year, provided to the regulated entities’ ernance documents. Triggers for ad hoc vali-
governing body (e.g., the Board of Direc- dations, such as new or significant changes to
tors) and senior management.13 products and services, should also be thor-
oughly documented.
Key Performance Indicators (KPIs) and Key
Risk Indicators (KRIs) of the FCC models
should be defined, produced, and reported
to relevant stakeholders/oversight commit-
tees that are responsible for or impacted by
the effectiveness of the FCC models.

Conceptual Conceptual soundness is assessment of the We recommend that model methodology,

Soundness quality of the model design and construc- settings, limitations, and data sources should
tion, alongside documentation and evi- be appropriately documented. Transaction
dence supporting the methods and varia- monitoring scenarios and supplemental con-
bles used. This ensures any expert judg- trols should be aligned with and supported
ment used in the design of the model is well by the risk assessment, and sanctions screen-
informed and in line with industry practice. ing parameters should be reasonable and
supported with appropriate rationale. A

10
5AMLD, Directive (EU) 2018/843, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A32018L0843
11
SR 11-7, https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm
12
3.33 Monitoring effectiveness of money laundering controls, Prevention of money laundering/combating terrorist financing guidance (for the UK finan-
cial sector), JMLSG, https://www.jmlsg.org.uk/guidance/current-guidance/
13
Systems and Controls (SYSC) Handbook 3.2.6G G., https://www.handbook.fca.org.uk/handbook/SYSC/3/2.html#DES69

Public. Copyright © 2024 Capgemini. All rights reserved. 4

 Regulatory Guidance Capgemini Perspective
Particularly, the following factors should be sanctions screening model should also be
considered by the regulated entities during supported with appropriate screening lists
model design: that are updated, maintained, and validated.
1. customer, product and activity profiles;
2. distribution channels;
3. the complexity and volume of transac-
tions;
4. processes and systems; and
5. operating environment.14
Ongoing Ongoing monitoring confirms that a model We recommend that testing for effective on-
Monitoring is appropriately implemented and operat- going monitoring requires good model gov-
ing as intended and remains reliable and ap- ernance. For example, our testing of model
propriate. This should be dynamic, i.e., fol- governance includes reviewing the institu-
lowing any change in your risk profile or risk tion’s model risk management framework,
appetite, the model should be evaluated AML and sanctions risk assessment proce-
and adjustments considered to maintain its dures, and documentation of scenar-
effectiveness. ios/rules/thresholds. Key Data Elements
(KDEs) and the associated Extract, Trans-
The ongoing monitoring measures should
form, and Load (ETL) logic should be identi-
enable the regulated entities to identify
fied, defined, and documented.
emerging risks, and should include pro-
cesses to ensure that internal information, Ongoing monitoring should be reviewed, in-
such as information obtained as part of the cluding policies and procedures around vali-
ongoing monitoring of business relation- dating, tuning, and optimising the model. Re-
ships, is reviewed regularly for emerging ports to senior management, change man-
trends and issues in relation to individual agement policies, and issue management
business relationships and the regulated should also be reviewed for completeness
entities’ business. and accuracy. One component of ongoing
monitoring is the Management Information
the financial institution produces and re-
views on a periodic basis to assess model per-
formance, for example, alerts by scenario or
data quality dashboards.

Outcomes Outcomes analysis compares the model We recommend that outcomes analysis in-
Analysis outputs against actual outcomes (i.e., the cludes, for transaction monitoring models,
adjudication results of the alerts, confirm- functional testing by independently replicat-
ing whether the alerts are indeed suspi- ing alerts for a historical period and verifying
cious or a true match to the sanctioned sub- the outputs align. In addition, the parame-
ject). This can involve either independent ters, logic, documentation, and execution
model replication to confirm that results scripts used to implement the scenarios
match historical output and/or an effi- should be reviewed for consistency and accu-
ciency review of false positives against racy. Outcomes analysis should also include a
alert volumes. review of scenario efficiency and false posi-
tive rates.
For sanctions screening, outcomes analysis
includes screening original and altered

14
Systems and Controls (SYSC) Handbook 3.2.6F G., https://www.handbook.fca.org.uk/handbook/SYSC/3/2.html#DES69

Public. Copyright © 2024 Capgemini. All rights reserved. 5

 Regulatory Guidance Capgemini Perspective
names and keywords through the model and
assessing match rates while considering
fuzzy matching performance, message types,
and message fields.

The New York Department of Financial Services (NY DFS) introduced “Superintendent’s Regulation Part
504” (“NY DFS Part 504”) in 2017 that imposes requirements for all regulated institutions in New York State
and specifies several requirements for transaction monitoring and filtering programs (i.e., sanctions screen-
ing). The industry has widely adopted the NY DFS Part 504 requirements as best practices to ensure robust
and rigorous TM and Sanctions Filtering systems that are commensurate to their risk profiles.
The requirements relevant to models include:

Highlighted NY DFS Part 504 Capgemini Perspective

Regulatory Requirements
Transaction Monitor- Transaction monitoring systems We recommend that detection scenarios
ing should: should be in line with the institution’s risk
assessment and be properly documented
• Consist of BSA/AML detection
with underlying assumptions, parameters
scenarios with threshold values
and thresholds. Functional testing of these
that should be designed to de-
scenarios would include replication and
tect potential money launder-
code review to ensure consistency between
ing and other suspicious/illegal
the documentation and production sys-
activity;
tems.
• Include end-to-end pre- and
End-to-end implementation consists of trac-
post-implementation testing
ing data from source to target systems by
including a review of govern-
verifying data mapping and ensuring no
ance, data mapping, transac-
data is truncated or inappropriately modi-
tion coding, detection scenario
fied before reaching the model. This testing
logic, model validation, data in-
should not only be conducted prior to im-
put, and program output; and
plementation, but also on a recurring basis
• Be subject to an ongoing analy- after implementation and with any major
sis to assess the continued rele- changes to the system’s infrastructure or
vancy of the detection scenar- model.
ios, underlying rules, threshold
The model should undergo regular reviews
values, parameters, and as-
to ensure the scenarios, thresholds, param-
sumptions.
eters, and assumptions remain relevant to
the institution’s risk assessment and to any
potential changes to products, customers,
or geographies.

Sanctions Screen- Sanctions screening systems As with transaction monitoring systems, we

ing/Filtering should: recommend that end-to-end implementa-
tion testing consist of tracing data from
• Include end-to-end pre- and
source to target systems by verifying data
post-implementation testing
mapping and ensuring no data is truncated
including a review of data
or inappropriately modified before reach-
matching, whether the OFAC
ing the model. This testing should not only
sanctions list and threshold

Public. Copyright © 2024 Capgemini. All rights reserved. 6

 Highlighted NY DFS Part 504 Capgemini Perspective
Regulatory Requirements
settings map to the institu- be conducted prior to implementation, but
tion’s risk, the logic of matching also on a recurring basis after implementa-
technology, model validation, tion and with any major changes to the sys-
data input, and program out- tem’s infrastructure or model. A unique con-
put; and sideration for sanctions screening is the
changing nature of watchlists – an inde-
• Be subject to an ongoing analy-
pendent model validation should confirm
sis to assess the logic and per-
that all watchlists are relevant, accurate,
formance of the technology for
and up to date.
matching names/accounts and
OFAC sanctions list and thresh- The model should undergo regular reviews
old settings to see if they map to ensure the watchlists, thresholds, param-
to the institution’s risk. eters, and assumptions remain relevant to
the institution’s risk assessment and any po-
tential changes to products, customers, or
geographies.

Both Transaction Mon- Both transaction monitoring and We recommend that KDEs be defined and
itoring and Sanctions sanctions screening systems documented, with data quality and data lin-
Screening/Filtering should: eage testing performed regularly on KDEs
for both TM and sanctions screening mod-
• Identify all data sources with
els.
relevant data;
Reviewing internal governance and over-
• Include validation of the integ-
sight of models is an important element of
rity, accuracy, and quality of
model validations, as ongoing monitoring is
data to ensure accurate and
a key element of model risk management.
complete data flows;
Also keep in mind that while NY DFS Part
• Review data extraction and
504 does not specifically include Customer
loading processes to ensure a
Risk Rating (CRR) models, Capgemini rec-
complete and accurate transfer
ommends that these be in-scope for model
of data from source to transac-
validations. Often, customer risk ratings are
tion monitoring/filtering sys-
used as inputs for transaction monitoring
tems; and
models and CRR models play an important
• Review governance and man- role in mitigating risk for financial institu-
agement oversight to ensure tions.
changes are defined, managed,
controlled, reported, and au-
dited.

Benefits of model validation for financial crime risk management

In addition to compliance with regulatory requirements, there are numerous benefits to financial institutions
for performing model validations.
• Improved efficiency. By identifying and mitigating the source of false positives or inefficient rules, insti-
tutions can more efficiently deploy resources away from clearing false alerts.

Public. Copyright © 2024 Capgemini. All rights reserved. 7

• Senior management reporting and good governance. Model validations include reviewing outputs, es-
calations, and reports to senior management. Following a model validation, leadership can be assured that
they are receiving accurate, timely, and useful data regarding model metrics. Validating this data is crucial
as often this data is used in executive decision-making and risk management.
• Commitment to financial crime risk management.
o Firms who maintain a model validation program can place greater reliance on automated financial
crime controls such as transaction monitoring, sanctions screening, and customer risk rating models.
o Effectively managed and remediated self-identified issues during model validation demonstrate to a
regulator the firm’s proactive commitment to identify and prevent financial crime and evidences a
mature and effective risk management framework.

What to expect
When undergoing a model validation, institutions will want to gather all supporting documentation relating
to the model, management information reporting, the model’s underlying assumptions, and how it supports
the institution’s risk assessment and risk management framework. Walkthroughs of the model’s production
environment and interviews with key staff will also help with conceptual soundness, model governance, and
ongoing monitoring assessments.
From a technical perspective, model input and output data for a specific timeframe will allow for scenario
replication. Source and target data will also permit data quality and data lineage testing. Establishing a test
database environment is a common method to host relevant data and easily facilitate validations and audits.
Most model validations can be completed in eight to 12 weeks, depending on the complexity of the models.
Any observations and findings are provided as the validation proceeds, with a written report summarising
the analysis performed and all findings and recommendations. Findings could range from documentation
(indicating less severe documentation gaps or inconsistencies) to high risk (indicating controls are absent or
require significant enhancement). Workpapers supporting the validation are also included with the report,
with copies of code and results tied to the regulatory framework.

How Capgemini can help you

Capgemini has deep expertise in the design and execution of financial crime model validations and of the
assurance and development of AML and sanctions screening programs. Our methodologies incorporate all
relevant regulatory guidance and requirements to ensure a comprehensive review that will identify any po-
tential or actual control weaknesses and failings, improve your framework’s efficiency, and confirm if your
models are operating as intended and in line with business expectations.
We take a holistic approach that considers all aspects of the model and put observations into meaningful
context. No model is perfect, and any skilled team can identify issues with data, documentation, or other
aspects of a model. We work to ensure that the impact of the issues is well understood and work with our
clients to determine a reasonable approach to issue remediation. Our goal is to provide clients with a clear
view of how their models are performing, and how to improve the performance going forward.
For more information, get in touch to discuss how Capgemini can help you with your model validation needs.

Public. Copyright © 2024 Capgemini. All rights reserved. 8

Authors

Peter Weitzman Michael Lovejoy

Managing Director / Analytics Lead Associate Director
FCC Advisory Solutions FCC Advisory Solutions
[email protected] [email protected]

Sky Hsu Tony Eddington

Managing Consultant Director
FCC Advisory Solutions FCC Advisory Solutions
[email protected] [email protected]

About Capgemini
Capgemini is a global leader in partnering with companies to trans-
form and manage their business by harnessing the power of tech-
nology. The Group is guided everyday by its purpose of unleashing
human energy through technology for an inclusive and sustainable
future. It is a responsible and diverse organization of nearly 350,000
team members in more than 50 countries. With its strong 55-year
heritage and deep industry expertise, Capgemini is trusted by its cli-
ents to address the entire breadth of their business needs, from
strategy and design to operations, fuelled by the fast-evolving and
innovative world of cloud, data, AI, connectivity, software, digital
engineering and platforms. The Group reported in 2022 global rev-
enues of €22 billion.
Get the future you want | www.capgemini.com

Public. Copyright © 2024 Capgemini. All rights reserved. 9