Welcome to this Assessment Practitioner Workshop series:

Computer Adaptive Testing for beginners

Please take a moment to read the information below so that you can fully participate in today’s session.

• Please check that you have selected the ‘microphone’ option when 1 2 3
joining the session. If you have joined as ‘listen only’, you will
not be able to unmute yourself and fully participate in the
interactive activities in the workshop.
• If you have joined as ‘listen only’, please click on ‘leave audio’
and then ‘join audio’. You can then select ‘microphone’, and
will be able to mute and unmute yourself as needed.

• For the session to run smoothly we recommend you:

• check that the audio on your device is active, and that the correct mic/speaker has been selected
• close background applications (Microsoft Teams, etc)
• pause computer software updates
• use Google Chrome as your browser.

If you are having problems after trying these options, please let us know in the chat box on the left of your screen and our
moderator will be able to provide additional assistance.
Question writing for effective assessment

Session 1: Principles and design of assessment

Nancy Sneddon and Brooke Wyatt | 14 November 2024

© Cambridge University Press & Assessment 2024

 3

Question writing for effective assessment

A series of linked webinars:

Title Date
Session 1 Principles and design of assessment 14th November 2024
Session 2 The process of question writing 21st November 2024
Session 3 Writing and reviewing a range of question types 28th November 2024
Session 4 Designing reliable mark schemes 5th December 2024
 4

Session 1: Principles and design of assessment

In this session, we will cover:

• the principles of good assessment

• developing a ‘blueprint’ / specification / syllabus
• developing and producing test papers.
 5

The principles of good assessment

 6

Task 1: What makes a good test or exam?

In the chat box:

• identify one feature of a good test.

 7

The principles that underpin good assessment

• purpose
• validity
• reliability
• manageability
• impact
 8

Clarity about purpose

• What is this assessment (and its results) to be used for?

• Who are the target candidates?

• Identifying the purpose(s) is important because:

 we need to know the purpose(s) to evaluate an assessment
 we need to know if the different kinds of inferences made from test
results are appropriate.
 9

What is validity?

Traditional definition:
‘whether a test really measures what it purports to measure’
(Kelley, 1927)

Contemporary definition:
‘the extent to which certain inferences can be made from an
assessment’s outcomes’
(Isaacs et al, 2013)
 10

Construct – what an assessment ‘purports to measure’

The construct is what are we measuring.

The ‘construct is … the concept or characteristic that a test is designed to

measure’.
(AERA Standards, 2014)

Another way to think about a particular construct is what it means to be good

at this subject / topic.

knowledge, understanding and skills

 11

Validity at the level of a question

An exam question can only contribute to valid assessment if:

the students’ minds are doing the things we want them to show us they
can do;

and if we give credit for, and only for,

evidence that shows us they can do it.

(Pollitt & Ahmed, 2008)

 12

Questions for reflection

Think about an assessment you know well.

What is the construct being assessed?

Do all stakeholders share an understanding of the construct?
Is there a clear statement of the construct?
What claims are made about someone who has passed this
assessment?
How far are these claims valid?
How far is this assessment fit for purpose?
 13

Two threats to the validity of assessments

Construct under-representation Construct irrelevant variance

 14

Task 2: What are these examples of?

CU (construct under-representation)
or
CIV (construct irrelevant variance)?

1. A reading test that includes a question about how long it would

take to get from one place to another on the basis of train times
given in the text.

Type CU or CIV in the chat box.

 15

Task 2: What are these examples of?

CU (construct under-representation)
or
CIV (construct irrelevant variance)?

2. A science exam with two papers of equal length. The first paper
assesses 75% of Section A of the syllabus. Paper 2 assesses
40% of Section B and 25% of Section C.

Type CU or CIV in the chat box.

 16

Task 2: What are these examples of?

CU (construct under-representation)
or
CIV (construct irrelevant variance)?

3. An online multiple choice test about the science of hair and

different hair styles, which claims to qualify people to be
hairdressers.

Type CU or CIV in the chat box.

 17

Task 2: What are these examples of?

CU (construct under-representation)
or
CIV (construct irrelevant variance)?

4. An image that gives away the answer to a question in a

geography exam.

Type CU or CIV in the chat box.

 18

Fairness

‘If an assessment is to be fair and equitable, it must give fair access to

those individuals engaging with the assessment, as well as be unbiased
towards different sub-groups within the cohort.’
(Isaacs et al, 2013)
 19

Fair Access By Design

diversity, inclusivity and accessibility

• be free from gender, ethnic, political, cultural or other discrimination and

stereotyping
• use content, resources and assessment materials that recognise the
achievements and contributions of different groups
• where appropriate, provide a balance of assessment methods and permit
alternative approaches

https://qualificationswalesˌorg/media/avuda1dk/fair-access-by-designˌpdf
 20

Task 3: Are these questions fair?

Use the chat box to identify aspects of these two questions that might make them
unfair:

1) ‘Lack of European identity is the Achilles heel of the European

Union.’

Discuss.
 21

Task 3: Are these questions fair?

2) Ali is going to cover a 3m x 3.6m patio with paving slabs. Each

paving slab is a square of side 60cm. Ali buys 32 of the paving
slabs.

Does Ali buy enough paving slabs to cover the patio?

 22

Reliability

• Reliability is part of validity.

• ‘If the same person had taken the test for the first time on another
occasion, then they would have got the same results.’ (NFER, 2001)

• Consistency and repeatability

 inter-rater reliability / marking consistency
 year-on-year consistency
 23

Task 4: What are possible threats to reliability?

Use the chat box to suggest possible threats to the reliability of

assessments.
 24

Task 4: What are possible threats to reliability?

Use the chat box to suggest possible threats to the reliability of

assessments.

What do you think is the biggest threat?

 25

Threats to reliability

• the occasion – the weather might be very hot, or the invigilator might give unclear
instructions
• the questions – the choice of topics might be less familiar, the wording of a question
might be ambiguous
• the mark scheme – the mark scheme might be very rigid / vague / not reward atypical but
correct answers
• the marker – is a human being who might be generous / severe / inconsistent / subjective
• the level-setting panel – the decisions might be swayed by one or two people with a
particular view / a different group of people might have made different decisions
• the candidate – is a human being / dynamic / subject to change from day to day
 26

Manageability and impact

Manageability: Impact:

costs – development, washback

administration, marking, etc
motivation
staffing
space / equipment curriculum
time course books
 27

Practicalities of assessment

Assessors need to balance the requirements of ‘fitness for purpose’ with

the manageability of their assessments.
 28

Questions for reflection

Think about an assessment you work on.

Are there any issues with the validity, reliability, manageability or

impact of this assessment?
If so, what are they?
Are ‘validity’ and ‘reliability’ terms you use regularly in your context?
If not, would it be useful to introduce these concepts to colleagues?
 29

Developing a ‘blueprint’ for an assessment

 30

Task 5: The challenges of developing good assessments

In your breakout group:

• make a list of the biggest

challenges you face in trying
to develop valid and reliable
assessments.
 31

Developing assessments – what you need to think about

purpose and construct

test specification / syllabus / ‘blueprint’
specimen material
test / item development process
expert / trained writers and reviewers
documentation / quality assurance
feedback loop
validation
 32

What is in a ‘blueprint’ / specification / syllabus?

Details of:

the target group

the aims
the assessment objectives
the content / options
the scheme of assessment
 33

Assessment objectives (AOs)

What are they?

A set of statements in a syllabus that describe the focus of assessment.

AOs determine what students must do to prove that they have fulfilled
the requirements of the syllabus.
 34

Aims and assessment objectives

Aim: ‘develop and extend their knowledge and understanding of

specified key events, periods and societies in local, British, and wider
world history; and of the wide diversity of human experience’

AO1: ‘demonstrate knowledge and understanding of the key features

and characteristics of the period studied’

https://www.aqa.org.uk/subjects/history/gcse/history-8145/specification/scheme-of-
assessment
 35

Scheme of assessment

number of components / papers / range of evidence

assessment objectives and weighting
tasks / activities
question formats
number of marks
timings
 36

Developing and producing tests

 37

Test production process

 writer produces first draft (of paper or items)

 review
 writer produces second draft
 editing
 trialling
 final review / scrutiny / fact checking
 proofreading
 material is banked / prepared for delivery
 38

Creating a test paper: layout

 front cover / information onscreen – clear and in a logical order

 presentation – consistent use of headings, bullets, bold type, etc
 rubrics / instructions – clear and consistent
 onscreen navigation – clear and intuitive
 39

Creating a test paper: content

 sequence of questions – coherent

 questions ordered with increasing difficulty
 some variety of question / task type
 avoid too much unpredictability / over-predictability
 avoid overlap
 avoid giving the answer to one question in another
 avoid follow-on effect
 options (if used) should be clear and comparable
 questions should be accessible (which is not the same as easy)
 ‘universal design’
 40

What do setters/writers need?

Setters / writers need:

 a clear specification
 copies of specimen material / past papers
 specific guidance relevant to the subject / context, including what to
avoid
 a house style guide
 checklists
 a setting grid
 training and feedback!
 41

Guidance for setters/writers

• general information about the qualification

• supporting documents
• overview of the setting/writing process
• overview of the question paper
• guidance on writing questions (and texts)
• guidance on selecting/adapting authentic material
• checklists
 42

Assessment / setting grids (1)

Question A01 A02 A03 Total

• ensure coverage of the
curriculum over time / 1a 3 3
balanced papers 1b 2 2 1 5

• questions and marks allocated

to assessment objectives 1(a) Explain why hot deserts are classified as
extreme environments.
______________________________ [3]
(b) Name and describe ONE physical process and
resulting landform that may occur in a hot desert
environment.
You may draw a diagram to help you.
______________________________ [5]
 43

Assessment / setting grids (2)

Content
AOs Demand Assessed
Total Mark for paper AO1 AO2 AO3 Challenging Regular Basic G10 only
40 40 Specified Marks 30 10 0 8 16 16 40

Setter to complete green area

Question Strand Sub-strand / AOs Demand
and key ref description AO1 AO2 AO3 Challenging Regular Basic
1C 2.1.1 Structure of DNA 1 1 1
2A 2.1.2 transcription and translation 1 1 1
3D 2.1.4 Effect of mutations 1 1 1
4B 2.2.1 Genetic engineering 1 1 1
5D 2.2.3 GM crops 1 1 1
6B 3.1.1 Diffusion 1 1 1
7A 3.1.2/3 Water potential 1 1 1
8C 3.1.6 Diff, Act. Trans and Osmosis 1 1 1
9B 3.2.1 Use a microscope 1 1 1
10 B 3.2.2 Leaf cross section 1 1 1
11 A 3.2.2 Leaf tissue function 1 1 1
12 C 3.3.2 Transpiration rate factors 1 1 1
 44

Questions for reflection

Think about how you develop and produce assessments in your

context.

What process do you use? Could it be improved?

Do you have a clear ‘blueprint’ / specification?
Can you see a use for assessment grids in your context?
Do you have sound QA processes in place?
 45

Summary of the session

We have looked at:

• the principles of good assessment, especially validity and reliability

• what to think about when developing an assessment
• what is included in a specification and why it is important to all
stakeholders
• the process of developing and producing question papers.
 46

Any questions?

Use the chat box to ask any questions you have at this point.

We won’t be able to answer them all, but we’ll do our best.

 47

Task 6: Review and reflect

Please reflect on today's session:

• Identify THREE key points you’ve learned during this session.

• Identify TWO ways in which you will apply this knowledge in your context.
• Identify ONE point that you would like to explore further.

Please write your reflections in your Course Journal on the VLE.

If you have any questions following the session, please email:

[email protected]
 48

Takeaway resources

• Handout 1: Sample setters’ checklist

• Reference list (same for all four sessions)

 49

Question writing for effective assessment

A reminder:
Title Date
Session 2 The process of question writing 21st November 2024
Session 3 Writing and reviewing a range of question types 28th November 2024
Session 4 Designing reliable mark schemes 5th December 2024
cambridgeassessment.org.uk/the-network © Cambridge University Press & Assessment 2024
Join The Assessment Network

Cambridge Assessment Network

@AssessNetwork

Sign-up for our weekly news round-up and regular email updates about training courses
and events.

cambridgeassessment.org.uk/the-network

[email protected]
 52

Enhance your status as a

recognised assessment expert

Increase your knowledge,

develop best practice, and gain
professional recognition by
joining a worldwide community of
assessment experts.

Find out more:

cambridgeassessment.org.uk/membership