EXAMINERS’ REPORT

CP2 Modelling Practice

Core Practices
Paper One

April 2023
 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

Introduction

The Examiners’ Report is written by the Chief Examiner with the aim of helping
candidates, both those who are sitting the examination for the first time and using past
papers as a revision aid and also those who have previously failed the subject.

The Examiners are charged by Council with examining the published syllabus. The
Examiners have access to the Core Reading, which is designed to interpret the syllabus,
and will generally base questions around it but are not required to examine the content of
Core Reading specifically or exclusively.

For numerical questions the Examiners’ preferred approach to the solution is reproduced
in this report; other valid approaches are given appropriate credit. For essay-style
questions, particularly the open-ended questions in the later subjects, the report may
contain more points than the Examiners will expect from a solution that scores full marks.

For some candidates, this may be their first attempt at answering an examination using
open books and online. The Examiners expect all candidates to have a good level of
knowledge and understanding of the topics and therefore candidates should not be overly
dependent on open book materials. In our experience, candidates that spend too long
researching answers in their materials will not be successful either because of time
management issues or because they do not properly answer the questions.

Many candidates rely on past exam papers and examiner reports. Great caution must be
exercised in doing so because each exam question is unique. As with all professional
examinations, it is insufficient to repeat points of principle, formula or other text book
works. The examinations are designed to test “higher order” thinking including
candidates’ ability to apply their knowledge to the facts presented in detail, synthesise and
analyse their findings, and present conclusions or advice. Successful candidates
concentrate on answering the questions asked rather than repeating their knowledge
without application.

The report is written based on the legislative and regulatory context pertaining to the date
that the examination was set. Candidates should take into account the possibility that
circumstances may have changed if using these reports for revision.

Sarah Hutchinson
Chair of the Board of Examiners
July 2023

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

A. General comments on the aims of this subject and how it is marked

The aim of this subject is to ensure that the successful candidate can analyse data, develop
a model, and document the work (including maintaining an audit trail for a fellow student
and senior actuary). They should be able to analyse the methods used and outputs
generated and communicate to a senior actuary the approach, results and conclusions.

The subject is split into two papers. The first, dealt with in this report, covers the
objectives:

· analysis of data.
· development of a model with clear documentation.

As the focus of the subject is on communication, the majority of the marks are for the
documentation and outputs generated rather than for technical modelling skills. For
example, a technical mistake is only penalised once, and candidates can still earn marks
for accurate and clear communication of what was done.

Candidates who give well-reasoned points not in the marking schedule are awarded marks
for doing so.

B. Comments on candidate performance in this diet of the examination.

The overall difficulty of the paper was in line with the average CP2 diet.

Modelling
The more straightforward elements of the modelling were well handled, with the data
analysis particularly well covered. Averaging costs over the year was only managed by
the stronger candidates, while later elements of the modelling were only completed
correctly by the stronger candidates.

We were pleased to see candidates attempt the modelling even where challenges were
faced. By taking this approach candidates had a good model off which to base their audit
trail. The descriptions in the audit trail are awarded credit where they reflect the
modelling completed therefore even where the modelling did not achieve all of the
technical marks, the audit trail would gain credit for the description.

Prospective candidates should be reminded that there is an expectation that any element
of the Core Principles can be drawn on in the CP2 exam. It is therefore recommended that
CP2 is best approached after being familiar with the content of these earlier subjects.

Audit Trail
Most candidates get the main elements of an audit trail. The quality of methodology
descriptions continues to improve over time, as more candidates cover both the what and
how of what the model is doing. Reasonableness checks continue to be a weak area for
most candidates.

C. Pass Mark

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

The Pass Mark for this exam was 60

1107 presented themselves and 708 passed.

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

Solutions for Subject CP2-1 - April 2023

Q1
Validation of data:
(i)
The status at the start of the year is in one of the four states (H, S, I, D) [½]
The status at the end of the year is in one of the four states [½]
No resurrection [1]
Adequate automated checks to identify the incorrect entries (one resurrection and
two unknown statuses) [1]
Reasonable correction of the three entries (e.g. deletion of the incorrect entries) [2]
Any other valid data checks (e.g. check no missing data, check counts etc) [1]

(ii)
Calculation of the number of movements in each of these 16 states [3]

(iii)
Conversion of the movements into transition probabilities [2]

(iv)
For each year in the projection period:
Calculation of the number of people in each of the four states [2]

Calculation of the present value of the total costs:

Allowance for inflation [1]
Allowance for the timing of the costs [2½]
Allowance for discounting [1]
Calculation of present value of cost per life alive (calculation of number of lives
alive and correct division using the number of lives alive at t-1) [1½]

(v)
For the alternative scenario:
Adjustment to the transition probabilities [1]
Adjustment to the drug costs [1]
Correct calculation [1]

(vi)
For the drug cost in the alternative scenario:
Adjustment to the set-up of the drug costs in the S and I states [1]
Correct set up of a goal seek [1]
Calculation of the drug costs [1]

(vii)
Production of a suitable chart for the number of lives alive under the base and
alternative scenario [2]

(viii)
Production of a suitable chart to illustrate the real cost per individual alive
under each of these three scenarios [2]
[Sub-total 29]

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

The technical modelling for this paper was reasonably simple throughout, but had
enough complexity to allow better candidates the opportunity to display a more
thorough understanding of the assignment.

The initial data analysis was very well handled by nearly all candidates. The errors in
the data were found, and a variety of corrections were made, which were all
acceptable provided a rationale was given.

The majority of candidates managed the simpler elements of the calculations well,
with most managing to project the number of people in each state, and calculating the
costs each year. However, only the stronger candidates managed to apply some
approach to averaging the costs over the course of a year. This was despite many
candidates including assumptions about transitions between states taking place half-
way through the year. There was scope within the assignment to allow for a number of
approaches to this issue, but it is expected that candidates present a coherent
solution, where the assumptions made tie in to the modelling performed.

Better candidates managed to complete the modelling, and managed to adjust the
transition probabilities correctly, and perform the goal seek for the altered drug costs.
For weaker candidates, common errors were to not fully account for the transitions
from the three live states into all four states, or to not apply the additional transitions
correctly.

When it comes to the final results and the creation of graphs, this was less well
handled. Many candidates misunderstood the concept of the present value of costs per
life alive, and there were a large variety of approaches here. A notable proportion
didn't understand that this result needed to be calculated for each year of the
projection, and thus only produced a single figure based on the total discounted cost.
Candidates are expected to consider the rationale behind what is being asked for and
use this understanding of the assignment to inform their modelling, and the
reasonableness checks on their work. In this case, the impact of the different drug was
cumulative, with the transition probabilities to healthy improving and to sick and very
sick reducing over time. Stronger candidates were able to pick this up and present the
results appropriately as graphs over time showing the difference between scenarios.
They were also able to provide reasonableness checks which displayed their grasp of
the assignment. It should be kept in mind that the ability to produce reasonableness
checks that show an understanding of what the model is supposed to do is one of the
best indicators of a pass.

Q2
(i)
Auto checks on the modelling completed in Q1:
Check the system is closed (i.e. all transitions are accounted for/all statuses

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

always add up to 1,000,000) [1]

Check the probabilities of each of the four transitions sum to 1 [1]
By setting the parameters in Scenario 2 to the same as scenario 1, the model
gives the NPV results of scenario 1 [1]
Check that the sum of those dead and alive at the end of the 30-year projection
period equals to 1,000,000 i.e. no. of lives we started out with [1]
By setting goal seeked drug costs in Alt_scenario 2 same as those in Alt_
Scenario 1, the NPV results are the same and model should show that goal seek needs
to be rerun. [1]
Any other reasonable auto checks
[Marks available 5, maximum 2]

(ii)
Good spreadsheet practice:
No hard coding (use of parameters and no copy and paste values) [1]
Flagging rows / columns that don’t copy down [1]
Easy to follow (inputs, checks and outputs easy to find) [1]
Logical order (left to right, top to bottom, within and between sheets) [1]
Clear and accurate labelling within the spreadsheet - rows, columns, worksheets [1]
Use of simple techniques (but not oversimplified) - formulae not overly
complex/steps split out and calcs built up [2]
[Sub-Total 9]
[Total for spreadsheet model 38]

Most candidates obtained marks for auto checks on the transition probabilities.
Spreadsheet practice continues to be well handled, with only the weaker candidates
scoring poorly here.

Q3

Audit trail
(i)
Communication skills:
How the steps have been executed is clear, rather than just What has been done
being stated [2]
There is sufficient technical detail and does not include excessive use of Excel
formulae to describe steps [1]
Sufficient detail is provided in the audit trail as a stand-alone document [1]

(ii)
Fellow student can review & check the methods used in model:
For a newcomer, the audit trail is easy to follow i.e. the marker does not have to
look at the model directly to understand what has been done [2]
All the steps are correctly and clearly described [1]
The workbook is well labelled and is easy to navigate through [1]
Where there are, or could be errors, the audit trail would enable the student to
identify and correct errors [2]

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

Danger areas in the spreadsheet are appropriately flagged (e.g. goal seek) [1]

(iii)
Senior actuary can scrutinise & understand what has been done:
A reasonable overview of the model is included [1]
There are clear statements of the assumptions made (i.e. concise list of value added)
assumptions, not long list with many not adding value [1]
Data sources are clearly described [1]
It is easy for a senior actuary to pick up the high-level detail of the modelling - can
pick up the high level without having to read all the detail [2]
The level of detail is appropriate for a senior actuary - explanations are clear
and concise [1]
Reasonableness checks are clearly stated and their results explained [1]

(iv)
Written in clear English:
The audit trail is written in clear, crisp and flowing English [2]
Accurate spelling [1]
The audit trail is laid out well, with good formatting to aid clarity [1]

(v)
Logical order:
Data is introduced before referring to it [1]
Assumptions are stated before using them [1]
The methodology is described in a logical order (i.e. nothing is introduced which
would require that the reader has read ahead) [1]

(vi)
Audit Content:
All steps clearly explained
The level of detail in the audit trail is appropriate for a newcomer to understand
what has been done [1]
All the methodology steps are set out clearly [2]
Data provided and any necessary adjustments made are described and justified
clearly [1]
All reasonableness checks applied are adequately documented [1]
Areas where manual intervention or caution is required are well flagged
(e.g. goalseeks or non-standard model areas) [1]
The marker does not need to look directly at the model to understand what has been
performed [2]

(vii)
Reasonableness checks:
The Dead (D) state is an absorbing state (i.e. no transitions out of this state)
(This mark can only be awarded once - if awarded as an auto check, cannot
get further credit here) [1]

Comment on the reasonableness of the transition probabilities, with specific example: [1]
Expect most healthy people stay healthy [1]
Expect most of the individuals with minor illnesses (S state) to recover [1]

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

Increasing probability of dying the more ill an individual becomes [1]

Probabilities of each transition state sum to 1 (This mark can only be awarded
once - if awarded as an auto check, cannot get further credit here) [1]

The change to the transition probabilities sum to zero in the alternative scenario [1]
There are no probabilities greater than 1 or less than 0 at any time t in the alternative
scenario [1]
There are more lives alive in the alternative scenario [1]
Because the transition probabilities have improved [1]
The present value of the cost of the programme came to about $8.8bn. This seems to
be consistent with the NHA budget of $9bn. [1]
For Alt_Scenario_1 - cost has gone up to $10bn which is over NHA’s budget. [1]
This is not unreasonable given the new drugs cost significantly more than those in
the base scenario. [1]
The drug costs using goal seek is less than that in the alternative scenario 1, which is
reasonable, given the total real costs in the alternative scenario 1 exceeds that of the
original scenario. [1]
Any other sensible reasonableness check
[Marks available 14, maximum 6]

(viii)
Signposting / labelling clear:
The audit trail allows the user to follow the model through [1]
The audit trail allows the user to understand each calculation easily [1]
There is adequate signposting in the audit trail to describe the purpose of each
tab [1]
Model labelling is consistent with the audit trail (data, parameters, scenarios, outputs,
charts) [1]

(ix)
Up to 3 marks for including assumptions (1 for each distinct, reasonable “added value”
one listed) [3]

(x)
Steps correctly described:
Purpose [1]
Data used, including source [1]
Data checks [2]
Data adjustments (correcting errors identified and removed) [1]
Calculation of the number of movements amongst health statuses [1]
Conversion of movements into transition probabilities [1]
Calculation of the number of individuals in each of the four states in each year [1½]
Calculation of the present value of the total cost in treating these people [1½]
Calculation of the real cost per live alive [1]
Calculation of the alternative scenario [2]
Calculation of the drug costs for the counter proposal [2]
Construction of the two charts [1]
Any other distinct and valid step
[Total marks for audit trail 62]

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

In general, audit trails were fairly well written, and show a heartening continuation
of an improving trend over the past few years. Better candidates showed evidence of
time planning and were able to focus their attentions on the key areas of the
modelling and audit trail.

Once again, there was a wide range in the quality of audit trails produced, from
those that clearly ran out of time, to the stronger candidates who are able to clearly
explain both what was done and how it was done in the model.

Many candidates struggled with providing assumptions that add value to the audit
trail. Many simply re-stated information provided in the background information, or
resorted to listing items which are assumed not to change. The better candidates
were able to grasp the ambiguity in the question and make assumptions that were
relevant to that situation. As mentioned above, many candidates made common
assumptions but then didn't perform the modelling in line with them.

Most candidates included only very basic reasonableness checks, with little attempt
to check that the final outputs or graphs made sense or that the scenario
calculations were working properly.

[Paper Total 100]

Issue with the exam paper

There were two sheets, which were inadvertently left 'hidden' in the ‘data for candidate’
spreadsheet. The names of these work sheets were ‘Alt scenario_1 (alt)’ and ‘Alt
scenario_2 (alt)’.

It could be deduced that the first of these relates to the alternative scenario examiners
outlined in the exam paper, and the second relatives to the goal seek scenario. (The (alt)
in brackets is a nod to this being an alternative approach - we include 3 possible
approaches in the sample solution.)

Within these worksheets is a table, with column labels, that could be used to derive a
high-level structure of a potential approach to solving the alternative scenario. Much of
this structure, however, flows organically from the exam paper.

All formulae contain #REF!, meaning all cells within the table show #REF! as the
output. No numerical elements or answers are shown. Candidates would have needed to
derive a parameters sheet by taking the information from the exam paper and then use
these parameters to correct the #REF!'s.

An extensive detection exercise was undertaken, combining electronic and manual

searches to determine which candidates may have identified these hidden sheets during
the course of the exam. It was concluded up to c.1.5% of candidates may have used these
hidden sheets.

CP21 A2023 © Institute and Faculty of Actuaries

 CP2-1 - Modelling Practice - Core Practices April 2023 Examiners’ report

The results of these candidates were analysed extensively, both independently of and
alongside the marks achieved under paper 2. The results varied, and the pass rate of these
was broadly consistent with the average for the diet overall.

It has been concluded that the presence of these sheets was only discovered by a small
cohort and has not had an impact on the overall performance of candidates in this sitting.

The pass mark was determined excluding these candidates, considering the skills being
examined by the paper. It was therefore not biased by this issue.

The Chief examiner and their team apologise for the stress this may have caused
candidates sitting this session. Steps are being taken to sanitise materials provided to
candidates in future sittings.

END OF EXAMINERS’ REPORT

CP21 A2023 © Institute and Faculty of Actuaries

Beijing
14F China World Office 1 · 1 Jianwai Avenue · Beijing · China 100004
Tel: +86 (10) 6535 0248

Edinburgh
Level 2 · Exchange Crescent · 7 Conference Square · Edinburgh · EH3 8RA
Tel: +44 (0) 131 240 1300

Hong Kong
1803 Tower One · Lippo Centre · 89 Queensway · Hong Kong
Tel: +852 2147 9418

London (registered office)

7th Floor · Holborn Gate · 326-330 High Holborn · London · WC1V 7PP
Tel: +44 (0) 20 7632 2100

Oxford
1st Floor · Belsyre Court · 57 Woodstock Road · Oxford · OX2 6HJ
Tel: +44 (0) 1865 268 200

Singapore
5 Shenton Way · UIC Building · #10-01 · Singapore 068808
Tel: +65 8778 1784

www.actuaries.org.uk
© 2021 Institute and Faculty of Actuaries