From Measurement to Assessment:

Exploring Predictability in Education

YAN Zi
Department of Curriculum and Instruction
The Education University of Hong Kong

24 June 2024

© 2024 YAN Zi EdUHK 1

 EdUHK ranks
20th IN THE WORLD IN
We stand at the forefront of research into Education and Psychology.
E D U C AT I O N QS World University Rankings by Subject 2024

© 2024 YAN Zi EdUHK 2

晏子教授
香港教育大学，课程与教学系署理系主任
主要研究兴趣：评估与测量
四项优配研究基金、两项优质教育基金
主持教师专业发展讲座超过 100 场，培训超过 3000 来
自香港、澳门、菲律宾的教师

3
© 2024 YAN Zi EdUHK
Outline
1 Education systems in Mainland China and Hong Kong

2 Searching for predictability in measurement ---- Rasch measurement

3 Searching for predictability in assessment ---- Formative assessment

4
© 2024 YAN Zi EdUHK
 The education systems in Mainland China and Hong Kong
Hong Kong
Mainland China

© 2024 YAN Zi EdUHK 5

From: https://edugatorsense.wordpress.com/2013/05/17/shanghai-chinas-system-of-education/ From: https://edugatorsense.wordpress.com/2013/05/17/hong-kongs-educational-system/
 Competence desired Be bi-
literate &
tri-lingual

Develop Acquire a
positive broad
attitudes knowledg
e base

Be an
Lead a informed
healthy &

Competence desired life-style

Be a
responsibl
e citizen
Acquire IT 21stcritical,
Century
& other reflective
Learner
life-long &
learning independ
skills ent
Mainland China thinker

Wang, T. (2019). Competence for Students’ Future: Curriculum Change and Policy Redesign in
China. ECNU Review of Education, 2(2), 234–245. https://doi.org/10.1177/2096531119850905
Hong Kong
https://www.edb.gov.hk/en/curriculum-development/7-learning-goals/secondary/index.html 6
© 2024 YAN Zi EdUHK
 Dominated by high-stakes public examinations
High academic ability
• Both Chinese mainland students and Hong Kong students face rigorous educational systems that emphasize academic excellence.
• In mainland China, the gaokao (National College Entrance Examination) is a high-stakes exam that determines university admissions. Students
prepare intensively for this exam, aiming for top scores to secure spots in prestigious institutions.
• Similarly, Hong Kong students take the DSE (Hong Kong Diploma of Secondary Education) exams, which significantly impact their university
prospects.
• The emphasis on academic ability reflects the competitive nature of higher education admissions in both regions.

“ 工程师红利”？ (Engineer Dividend )

• This term refers to the demand for engineers and technical professionals in the job market.
• In mainland China, rapid industrialisation, urbanisation, and technological advancements have created a surge in engineering-related jobs.
• Many students pursue engineering degrees due to the perceived job stability and financial rewards associated with engineering careers.
• Hong Kong also recognizes the importance of STEM (Science, Technology, Engineering, and Mathematics) fields, leading to a similar trend in
engineering education.

Searching for predictability in education and training?

• Both mainland China and Hong Kong value predictability in education.
• High-stakes exams provide a clear pathway for students: perform well, gain admission to top universities, and secure better job prospects.
• Predictability extends beyond exams. Students seek structured curricula, well-defined career paths, and clear expectations.

7
© 2024 YAN Zi EdUHK
 The Hempel-Oppenheim scheme
• A framework proposed by philosophers Carl Hempel and Paul
Oppenheim in the philosophy of science.
• It aims to explain scientific explanation by formulating it as a
deductive argument.
• The deductive-nomological model suggests that a scientific
explanation involves deriving the explanandum as a logical
consequence of the covering laws and the initial conditions. In other
words, if the covering laws and initial conditions are true, then the
explanandum must also be true.

8
© 2024 YAN Zi EdUHK
 A simplified example to illustrate the Hempel-Oppenheim
scheme:
• Explanandum: The apple fell from the tree.
• Explanans:
• Covering Law: All unsupported objects near the Earth's surface
will fall due to gravity.
• Initial Condition: The apple was unsupported and near the
Earth's surface.
• Using the covering law and the initial condition, we can deduce
the explanandum: The apple fell from the tree.

9
© 2024 YAN Zi EdUHK
Searching for predictability in measurement

10
© 2024 YAN Zi EdUHK
Check the predictability using the Hempel-Oppenheim scheme:
• Explanandum: A math test score of 80 represents a higher math ability compared
to a score of 70.
• Explanans:
• Covering Law: In this educational system, math test scores are a reliable
measure of math ability, such that higher scores correspond to higher levels of
math ability.
• Initial Condition: The specific conditions or facts relevant to the explanandum,
which may include the math test scores of individual students, the grading
criteria of the math test, the established relationship between test scores and
math ability within the given educational system, etc...
• Using the covering law and the initial condition, can we deduce the
explanandum?
11
© 2024 YAN Zi EdUHK
Measurement
In physical sciences In social sciences

What’s the difference?

12
© 2024 YAN Zi EdUHK
Objective Measurement

• Objective measurement is the repetition of a unit amount that

maintains its size, within an allowable range of error, no matter
which instrument is used and no matter who or what relevant
person or thing is measured.

Institute for Objective Measurement, 2000

4 13
© 2024 YAN Zi EdUHK
Objective Measurement Requires …
• Interval scale
• Person measures independent of items &
item calibrations independent of persons

14
© 2024 YAN Zi EdUHK
 Levels of Measurement
Stevens (1946, 1951, 1958, 1960)
Nominal scale (e.g., gender)
- Labels
- No meaningful order
Ordinal scale (e.g., rankings)
- reflect the order on the amount of the property
Interval scale (e.g., temperature)
- distances between numbers have meanings;
- no fixed origin (or the origin is arbitrary)
Ratio scale (e.g., weight, height, time)
- distances between numbers have meanings
- zero means no amount of the property 19
© 2024 YAN Zi EdUHK
 Are raw scores in tests interval measures?
E.g., 5 students’ scores on a test
45 52 59 91 98

+7 +7

Does earning a few extra marks near the midpoint of the test results, say from 52
to 59, reflect the same ability leap required for a move from 91 to 98?

No proof, but generally used as if they are interval.

22
© 2024 YAN Zi EdUHK
 Are raw scores independent of test items?
0 5
Dif = 5 – 0 = 5

5 5
Dif = 5 – 5 = 0

2 4 Dif = 4 – 2 = 2

3 3
Dif = 3 – 3 = 0

The relative scores of the same two persons change according to item locations.
23
© 2024 YAN Zi EdUHK
 Is conventional measurement in education
objective?

24
© 2024 YAN Zi EdUHK
 The Rasch Model
• The Rasch model is a latent trait model.
• A person’s response to an item is
determined by the difference between
person ability and item difficulty,
measured on the same continuous
scale.

25
© 2024 YAN Zi EdUHK
Pair-wise Comparison
between Individuals and Items

• Ability = difficulty, probability

of correct = 0.5
• Ability > difficulty, probability
of correct > 0.5
• Ability < difficulty, probability
of correct < 0.5

26
© 2024 YAN Zi EdUHK
 For the same person, the possibility of correct answer
varies according to various difficulties of items

㏑ 5 00 =
?

2 1 x 63
=?

1 + 1= ?

27
© 2024 YAN Zi EdUHK
 The Rasch model could be express as:

• The probability (P) for person m to elicit a correct

response to item i (in the dichotomous case of ‘0’ -
incorrect; ‘1’- correct) is a function of the difference
between the measure of a person’s ‘ability’ and an
item’s ‘difficulty’ (θm -δi).
28
© 2024 YAN Zi EdUHK
 Rasch-Model Properties (1) The Common
Scale
More capable More difficult
individuals items

Less capable Less difficult

individuals items
30
© 2024 YAN Zi EdUHK
Rasch-Model Properties (2): Interval Scale
• The Rasch scale (in logit, i.e., log odds unit) can be made
independent of the particular group of items that happen to be
included in a test this time, or the particular samplings of persons
that happen to have been used to calibrate these items.
• Example: when the data fit the dichotomous Rasch model, a
person with ability one logit higher than an item's difficulty
(whatever it is) always has a probability of 0.73 of succeeding on
the item.

31
© 2024 YAN Zi EdUHK
Rasch-Model Properties (2): Interval Scale
θ-δ Pr(X=1)
4.0 0.98
3.5 0.97 When the person is 3 logits above the
3.0 0.95 item, there is 95% chance of success.
2.5 0.92
2.0 0.88
1.5 0.82 When the person is 1 logit above the
1.0 0.73 item, there is 73% chance of success.
0.5 0.62
0.0 0.50
-0.5 0.38 When the person is 1 logit below the
-1.0 0.27 item, there is 27% chance of success.
-1.5 0.18
-2.0 0.12
-2.5 0.08 When the person is 3 logits below the
-3.0 0.05 item, there is 5% chance of success.
-3.5 0.03
-4.0 0.02
32
© 2024 YAN Zi EdUHK
What has been shown is…
Measurement is
objective; Relative The Rasch model
abilities of two persons
are constant irrespective
holds
of which items have been
used for measuring the
persons

Measurement is The Rasch model

not objective does not hold

36
© 2024 YAN Zi EdUHK
In Sum
• Rasch measurement transforms ordinal raw scores to interval measures
• If you are only interested in ONE test, and you are only interested in ranking
students, then Rasch measures do not tell you much more than raw scores do.
• If you want to
• generalize your test results,
• compare students who took different tests,
• perform statistical analyses on your data,
then Rasch measurement is what you need.

25 46
© 2024 YAN Zi EdUHK
Applications of Rasch measurement
• Developing new instruments
• Developing vertical scales
• Rasch + Path Analysis
• Solve practical measurement problems, e.g., classroom testing

48
© 2024 YAN Zi EdUHK
 Some research examples
• Examining instrument psychometric properties
• Yan, Z., Brubacher, S., Boud, D., & Powell, M. (2020). Psychometric properties of the Self-assessment Practice Scale for professional training contexts: Evidence from confirmatory factor analysis and Rasch
analysis. International Journal of Training and Development, 24(4), 357-373. https://doi.org/10.1111/ijtd.12201
• Yan, Z. (2020). Developing a short form of the Self-assessment Practices Scale: Psychometric Evidence. Frontiers in Education, 4, 153. https://doi.org/10.3389/feduc.2019.00153
• Yan, Z. (2018). The Self-assessment Practice Scale (SaPS) for students: Development and psychometric studies. The Asia-Pacific Education Researcher, 27(2), 123-135.
https://doi.org/10.1007/s40299-018-0371-8
• Yan, Z. (2016). The self-assessment practices of Hong Kong secondary students: Findings with a new instrument. Journal of Applied Measurement, 17(3), 335-353.
https://www.researchgate.net/publication/310443914_The_Self-assessment_Practices_of_Hong_Kong_Secondary_Students_Findings_with_a_New_Instrument
• Yan, Z., Lum, R. C. W., Lui, R. T. L., Chu, S. S. W., & Lui, A. M. (2015). Measuring teaching assistants’ efficacy using Rasch model. Journal of Applied Measurement, 16(1), 60-75.
• Yan, Z., & Mok, M. M. C. (2012). Validating the Coping Scale for Chinese Athletes using multidimensional Rasch analysis. Psychology of Sport and Exercise, 13, 271-279.
https://doi.org/10.1016/j.psychsport.2011.11.013
• Yan, Z. & Bond, T. G. (2011). Developing a Rasch measurement physical fitness scale for Hong Kong primary school-aged students. Measurement in Physical Education and Exercise Science, 15(3), 182-203.
https://doi.org/10.1080/1091367X.2011.590772
• Rasch plus Path analysis
• Yan, Z. (2020). Self-assessment in the process of self-regulated learning and its relationship with academic achievement. Assessment & Evaluation in Higher Education, 45(2), 224-238.
https://doi.org/10.1080/02602938.2019.1629390
• Yan, Z., & Cheng, E. C. K. (2015). Primary teachers’ attitudes, intentions and practices regarding formative assessment. Teaching and Teacher Education, 45, 128-136.
https://doi.org/10.1016/j.tate.2014.10.002
• Yan, Z., & Sin, K. F. (2015). Exploring the intentions and practices of principals regarding inclusive education: An application of the Theory of Planned Behaviour. Cambridge Journal of Education, 45(2), 205-
221. https://doi.org/10.1080/0305764X.2014.934203
• Rasch plus Regression
• Yan, Z. (2018). How teachers’ beliefs and demographic variables impact on self-regulated learning instruction. Educational Studies, 44(5), 564-577. https://doi.org/10.1080/03055698.2017.1382331
• Yan, Z. (2014). Predicting teachers’ intentions to implement school-based assessment using the Theory of Planned Behaviour. Educational Research and Evaluation, 20(2), 83-97.
https://doi.org/10.1080/13803611.2013.877394
• Rasch plus other conventional analyses (e.g., t-test, ANOVA, regression)
• Coniam, D., & Yan, Z. (2016). A comparative picture of the ease of use and acceptance of onscreen marking by markers across subject areas. British Journal of Educational Technology, 47(6), 1151-1167.
https://doi.org/10.1111/bjet.12294
• Cooper, P., & Yan, Z. (2015). Some possible effects of behaviour management training on teacher confidence and competence: evidence from a study of primary school teachers in Hong Kong. Educational
Studies, 41(1-2), 156-170. https://doi.org/10.1080/03055698.2014.955739
• Ho, F. C., & Yan, Z. (2014). Identification of the patterns of Chinese character recognition in students with learning disabilities requiring Tier-2 support: a Rasch analysis. Educational Psychology, 34(3), 305-
322. https://doi.org/10.1080/01443410.2013.785060
• Yan, Z., & Coniam, D. (2013). Assessing the ease of use in the environment and markers’ acceptance of onscreen marking: A Rasch measurement perspective. Educational Research and Evaluation, 19(5),
461-483. https://doi.org/10.1080/13803611.2013.793604
78
© 2024 YAN Zi EdUHK
Books
However,
• Rasch model does NOT …

82
© 2024 YAN Zi EdUHK
 Searching for predictability in assessment

83
© 2024 YAN Zi EdUHK
Check the predictability using the Hempel-Oppenheim scheme:
• Explanandum: Feedback on students’ math test performance can increase their
performance on the next math test.
• Explanans:
• Covering Law: In educational settings, providing feedback to students tends to
enhance their learning performance.
• Initial Condition: The specific conditions or facts relevant to the explanandum,
which may include the type and quality of feedback provided, the characteristics of
the students (such as their prior knowledge or motivation), the learning context,
etc...
• Using the covering law and the initial condition, can we deduce the explanandum?

84
© 2024 YAN Zi EdUHK
Impact of self-assessment on academic performance
• A meta-analysis including 86 independent studies focusing on mainstream
K-12 to higher education students
• Self-assessment interventions had meaningful effects on academic
performance (Hedges’ g = 0.585).
• However, the effect size varies from study to study; 22.79% of self-
assessment interventions had nil or negative impact; So, predictability is
not straightforward; design/implementation and context matter!

85
© 2024 YAN Zi EdUHK
Formative assessment reform in Hong Kong
课程发展议会 (2001) 。学会学习─课程发展路向。
课程发展议会 (2002) 。基础教育课程指引─各尽所能•发挥所长（小一至中
三）。
课程发展议会 (2009) 。高中课程指引：立足现在、创建未来（中四至中六）。
课程发展议会 (2014) 。基础教育课程指引─聚焦•深化•持续﹙小一至小
六﹚。
课程发展议会 (2017) 。中学教育课程指引。
课程发展议会 (2022) 。小学教育课程指引 ( 试行版 ) 。

88
© 2024 YAN Zi EdUHK
Formative assessment reform in Hong Kong
 Curriculum Development Council (2001). Learning to Learn - The Way Forward in
Curriculum Development
 Curriculum Development Council (2002). Basic Education Curriculum Guide - Building
on Strengths (Primary 1 - Secondary 3)
 Curriculum Development Council (2009). Senior Secondary Curriculum Guide: The
Future is Now: from Vision to Realisation (Secondary 4-6).
 Curriculum Development Council (2014). Basic Education Curriculum Guide - To
Sustain, Deepen and Focus on Learning to Learn (Primary 1 – 6).
 Curriculum Development Council (2017). Secondary Education Curriculum Guide.
 Curriculum Development Council (2022). Primary Education Curriculum Guide (Pilot
Version).

© 2024 YAN Zi EdUHK

 系统层面评估改革
（公开考试）
的困境
社会价值导向
教育资源的竞争 • 社会价值导向
考试主导的文化 • 教育资源的竞争
• 考试主导的文化

92
© 2024 YAN Zi EdUHK
课堂评估改革的目标
促进学习的评估 作为学习的评估
(Assessment for Learning) (Assessment as Learning)
“教师会透过回馈为学生在整个学 “要求学生在学习过程中学习
习进程中提供协助和改善建议，亦会按 检视自己或同侪的学习效能，反思
学生学习表现调整学与教策略，促进学 和调整学习策略，以发展他们自主
生有效学习。” 学习的能力。”
连系「促进学习的评估」与「作为学习的评估」
“「两种的评估模式有不少互相扣连和配合的地方，学校可有机结合两
种评估模式，设计整全的学习活动，一方面提供机会让学生主动参与评估任
务，另一方面，教师就学生评估后的反思 提供适切的回馈，促进学习。”
课程发展议会 (2022) 。《小学教育课程指引》 ( 试行版 ) 。

94
© 2024 YAN Zi EdUHK
 Cultural/contextual hurdles

Cultural/contextual hurdles

Teacher-student ratio Exam dominance 95

© 2024 YAN Zi EdUHK
The neglect of student agency
• Discourage self-regulation
• Disengage in assessment processes
• Suboptimal use of feedback

97
© 2024 YAN Zi EdUHK
 The research agenda
• Develop a holistic framework to research School
assessment
• with various methodologies (e.g., theoretical
argument, quantitative, qualitive, review)

learning
Assessment-as-
learning
Assessment-for-
• An effective and sustainable assessment
ecology
• Synergise teacher-directed formative assessment Teacher Student
(Assessment-for-learning) and student-initiated
formative assessment (Assessment-as-learning)
• Reshape the roles of students, teachers, and
parents
• Enhance all stakeholders’ assessment literacy
• Fulfil both short-term and long-term learning
targets Parent

98
© 2024 YAN Zi EdUHK
 Assessment-as-learning
5
6 (Student-initiated)
2 S1. Theory: A cyclical self-
assessment process model
5 3 S2. Instrument: The Self-
3 6
1 4 assessment Practices Scale
4 (SaPS)
2
S3. Impact of self-assessment on
Teacher- Student- learning
directed 1 initiated S4. Self-assessmen, SRL,&
Formative Formative

Meaningful Formative Assessment

Academic achievement
Assessment Assessment
S5. Predictors of self-assessmen
Assessment-for-learning (Teacher-directed) S6. Self-assessmen intervention
T1. Teacher formative assessment
literacy/practice scale
T2. Assessment-for-learning policy
T3. Impact of teachers’ FA on academic
performance
T4. Teachers’ FA & Growth Mindset
T5. Predictors of teachers’ FA Practices
T6. Teacher professional development © 2024 YAN Zi EdUHK
99
 Assessment-as-learning
5
6 (Student-initiated)
2 S1. Theory: A cyclical self-
assessment process model
5 3 S2. Instrument: The Self-
3 6
1 4 assessment Practices Scale
4 (SaPS)
2
S3. Impact of self-assessment on
Teacher- Student- learning
directed 1 initiated S4. Self-assessmen, SRL,&
Formative Formative

Meaningful Formative Assessment

Academic achievement
Assessment Assessment
S5. Predictors of self-assessmen
S6. Self-assessmen intervention
Assessment-for-learning (Teacher-directed)
T1. Teacher formative assessment
literacy/practice scale
T2. Assessment-for-learning policy
T3. Impact of teachers’ FA on academic
performance
T4. Teachers’ FA & Growth Mindset
T5. Predictors of teachers’ FA Practices
© 2024 YAN Zi EdUHK 100
T6. Teacher professional development
 Predictors of
SA & SRL
SA
Impact of SA SA
Intervention

Instrument: The Self-

assessment Practices Scale
(SaPS) (Yan, 2018a)
Theory: Assessment-as-Learning
concept (Yan & Boud, 2022); Self-
assessment process model (Yan &
Brown, 2017)
101
© 2024 YAN Zi EdUHK
 S1- Theory: Conceptualising Assessment-as-learning

• Assessment-as-learning is
“Assessment that necessarily
generates learning opportunities
for students through their active
engagement in seeking,
interrelating, and using evidence”

103
© 2024 YAN Zi EdUHK
 課程發展議會 (2022) 。《小學教育課程指引》 ( 試行版 ) 。
© 2024 YAN Zi EdUHK 104
 S1- Theory: A cyclical self-assessment process model

107
© 2024 YAN Zi EdUHK
S1- Theory: An updated self-assessment model

109
© 2024 YAN Zi EdUHK
 e21
Item 1 e1
.77

SEFM
.78 Item 2 e2 S2- Instrument: The Self-assessment
• 2906 Hong Kong students
.73

.93
.75 Item 3 e3 Practices Scale
ranging from Primary 4 to
.65
Item 4 e4
SEF

Secondary 3 e22 Item 5 e5

.87
.74 Item 6 e6

• Yan & Brown’s (2017) cyclical

SEFI .75
Item 7 e7
.76

self-assessment process
.59
.77 Item 8 e8

model
Item 9 e9

Item 10 e10

• Factor analysis + Rasch

.65
.83
Item 11 e11
.62

analysis
SIF .81
Item 12 e12
.71

Item 13 e13

Item 14 e14
.55

.67 Item 15 e15

.72
Item 16 e16
.79

SR .80
Item 17 e17
.81

.70
Item 18 e18
.74

Item 19 E19
© 2024 YAN Zi EdUHK 114
Item 20 e20 114
 S3- Impact of SA on learning
• Outcome: academic performance
• A meta-analysis including 86 independent studies focusing on mainstream
K-12 to higher education students
• SA interventions had meaningful effects on academic performance
(Hedges’ g = 0.585).
• 22.79% of SA interventions had nil or negative impact; So, design and
implementation matter!

116
© 2024 YAN Zi EdUHK
 S3- Impact of SA on learning
• Outcome: self-regulated learning strategies and affective outcomes
• A meta-analysis including 45 independent studies focusing on
mainstream K-12 to higher education students
• SA and PA interventions had positive effects on desirable learning
strategies (Hedges’ g ranging from 0.220 to 0.683) and affective
outcomes (Hedges’ g ranging from 0.205 to 0.615).
• Significant moderators: assessment method (quantitative vs. qualitative);
teacher feedback

117
© 2024 YAN Zi EdUHK
 S4 - SA, SRL, & Academic achievement

• 63 students enrolled in a 13-week

Master course
• A longitudinal design; 3 waves of
surveys
• Findings
• SA is a fundamental skill for SRL
and occurs at each SRL phase
with different patterns

119
© 2024 YAN Zi EdUHK
S4 - Relationships between of SA, SRL, & Academic achievement
• Findings
• Autoregressive relationships
were found for all self-
assessment actions between
different SRL phases
• Self-directed feedback seeking
through monitoring at
Performance phase was the
strongest and positive predictor
of academic achievement
• Achievement had negative
impact on all self-assessment
actions at Appraisal phase
120
© 2024 YAN Zi EdUHK
 S5 - Predictors of SA intention and practice
• 1425 Hong Kong students (Primary 4 to Secondary 3)
• An extended framework of the Theory of Planned Behaviour
• A cross-sectional survey
• Findings
• attitude, subjective
norms, self-efficacy, and
psychological safety were
significant predictors on
intention to self-assess
• self-efficacy, intention,
and psychological safety
had significant influence
on self-assessment
practice
© 2024 YAN Zi EdUHK
122
 S6 - SA intervention
• The intervention: self-assessment diaries; self-
assessment mind-maps; no feedback/marking
required
• Contexts
• 3 secondary schools; 3 primary schools
• 5 HK schools; 1 Philippines school
• Subjects
• Chinese/English/Math/Integrated Humanities

• Intervention length
• 5-10 minutes for each session
• Twice to three times per week
• 5 to 8 weeks

124
© 2024 YAN Zi EdUHK
S6 - SA intervention
• Enhanced academic performance, self-efficacy, self-assessment behavior,
self-regulated learning strategies, learning motivation
• Practical advantages
• Easy design and implementation
• No extra workload for teachers

125
© 2024 YAN Zi EdUHK
 Interaction AI-enabled
between SA Neural
SA/PA/TA mechanism
of SA

Predictors of
SA & SRL
SA
Impact of SA SA
Intervention

Instrument: The Self-

assessment Practices Scale
(SaPS) (Yan, 2018a)
Theory: Assessment-as-Learning
concept (Yan & Boud, 2022); Self-
assessment process model (Yan &
Brown, 2017)
127
© 2024 YAN Zi EdUHK
 Assessment-as-learning
5
6 (Student-initiated)
2 S1. Theory: A cyclical self-
assessment process model
5 3 S2. Instrument: The Self-
3 6
1 4 assessment Practices Scale
4 (SaPS)
2
S3. Impact of self-assessment on
Teacher- Student- learning
directed 1 initiated S4. Self-assessmen, SRL,&
Formative Formative

Meaningful Formative Assessment

Academic achievement
Assessment Assessment
S5. Predictors of self-assessmen
S6. Self-assessmen intervention
Assessment-for-learning (Teacher-directed)
T1. Teacher formative assessment
literacy/practice scale
T2. Assessment-for-learning policy
T3. Impact of teachers’ FA on academic
performance
T4. Teachers’ FA & Growth Mindset
T5. Predictors of teachers’ FA Practices
128
T6. Teacher professional development © 2024 YAN Zi EdUHK
 Predictors
Impact of FA
of FA
FA FA, Mindset
Policy & Academic
achievement

Instrument: Teacher Formative

Assessment Literacy/Practice Scale (Yan
& Pastore, 2022a, 2022b)
Theory: Teacher formative assessment
literacy and practice (Black & Wiliam, 2009;
Pastore & Andrade, 2017)

129
© 2024 YAN Zi EdUHK
T1 - Instrument:
Teacher Formative
Assessment Literacy Scale
• 585 teachers in Hong Kong
primary and secondary schools.
• A three-dimension model of
teacher assessment literacy
(Pastore & Andrade, 2019)
• Factor analysis + Rasch analysis

130
© 2024 YAN Zi EdUHK
T1 - Instrument:
Teacher Formative
Assessment Practice Scale
• Two samples of primary and
secondary school teachers: Hong
Kong (N = 449) and Italy (N = 309)
• The 5-category framework of
formative assessment strategies
(Wiliam & Thompson, 2008)
• Factor analysis + Rasch analysis

131
© 2024 YAN Zi EdUHK
 T2 - Assessment for learning policy in Hong Kong

• A narrative review and analysis of official documents and

empirical studies, guided by a social positivist critique of the
assessment for learning policy in the Hong Kong context.
• Findings
• Implementation of AfL in HK is unsatisfactory despite teacher
preference
• Continued use of public examinations has undermined AfL
• The successful implementation of AfL requires involving students in
assessment and enhancing assessment literacy of all stakeholders.

© 2024 YAN Zi EdUHK 132

 T3 - Impact of formative assessment on
reading performance (PISA 2018)
• This study examines the link between teachers’ formative assessment
strategies (i.e. clarifying goals and monitoring progress, providing
feedback, and instructional adjustments) and students’ reading
achievement, based on data from 151,969 fifteen-year-olds in 5,225
schools in 19 countries/regions in PISA 2018
• Findings
• Teacher feedback alone is not linked to student reading test scores
• Schools with more teachers sharing learning progressions show higher reading
scores
• Instructional adjustments link to higher reading scores

© 2024 YAN Zi EdUHK 133

 T4 - Formative Assessment & Growth
Mindset/Reading Performance (PISA 2018)

• 109,204 fifteen-year-olds from both West and

the East in PISA 2018.
• Findings
• Formative assessment strategies were associated
with having a growth mindset in both the West and
the East.
English-speaking countries/economies
• Growth mindset was positively associated with
reading achievement only in the West, but not in the
East.
• The impacts of different formative assessment
strategies on reading achievement varied. In
particular, teacher feedback was negatively
associated with reading achievement in both the
West and the East.

Confucian countries/economies 134

© 2024 YAN Zi EdUHK
 T5 - Predictors of teachers’ formative
assessment: A Hong Kong sample (study 1)

• 450 teachers from 10 Hong

Kong primary schools
• Based on Ajzen’s (1985,1991)
Theory of Planned Behaviour
(TPB)
• A cross-sectional design

© 2024 YAN Zi EdUHK 135

 T5 - Predictors of teachers’ formative assessment:
a Hong Kong sample (study 2)

• 296 teachers from 12

Hong Kong schools
(seven primary and
five secondary
schools)
• A longitudinal design;
two waves of surveys
one year apart

© 2024 YAN Zi EdUHK 136

 T5 - Predictors of teachers’ formative
assessment: a systematic review (Study 3)

• 52 eligible studies
• Personal and contextual
factors that that influence
teachers’ intentions and
implementations regarding
formative assessment

© 2024 YAN Zi EdUHK 137

 T6 - Teacher professional development in
improving formative assessment literacy

• A small-scale study (N = 27) with a 7

mixed method 6
5.36

• A five-day course for in-service 5

4.51 4.53

teachers. 4

• Findings 2 1.89

• The program developed teachers’ 1.18 1.23

1
formative assessment literacy
• Professional learning community
0
Conceptual dimension Praxeological dimension Social-emotional dimension

(PLC)
Pre-survey Post-survey

© 2024 YAN Zi EdUHK 139

 Adaptive
AI-enabled FA &
learning
formative feedback
feedback literacy

Predictors
Impact of FA
of FA
FA FA, Mindset
Policy & Academic
achievement

Instrument: Teacher Formative

Assessment Literacy/Practice Scale (Yan
& Pastore, 2022a, 2022b)
Theory: Teacher formative assessment
literacy and practice (Black & Wiliam, 2009;
Pastore & Andrade, 2017)

140
© 2024 YAN Zi EdUHK
 What can GenAI potentially bring?
GenAI has the potential to reshape the way students conduct self-assessment:
• GenAI can address the persistent obstacle of self-assessment (i.e., lack of
external feedback) by providing instant feedback to students that can help with
generating internal feedback;
• GenAI allows students to learn and progress at their own pace, without having
to wait for a teacher's availability, or reveal their deficiencies to another human,
and is thus likely to increase student engagement and make self-assessment
more efficient.

141
 Be cautious!
• GenAI may hinder learning if it gives incorrect, unclear, or confusing
information;
• GenAI may hinder learning if students:
(1) don't know how to use appropriate prompts to obtain useful
feedback;
(2) lack the ability to evaluate the validity of responses they get from
GenAI; or
(3) rely on GenAI to get an answer without any reflection on their
part.

142
 ChatGPT may be a game-changer, but it still depends on the player

• ChatGPT technology acceptance positively predicted self-reflection.

• However, feedback sense-making moderated these relationships—the benefits
of ChatGPT acceptance on self-reflection only occurred at moderate-to-high
levels of sense-making.

Mendoza, N. B., Xiong, Y., & Yan, Z. (under review). “ChatGPT is a game-changer, but it still depends on the player”: How feedback sense-making moderates the link between students’
acceptance of ChatGPT and self-regulated learning.

143
In summary
• Predictability is desirable
• Predictability in education is not straightforward
• Understanding learning processes is crucial
• Unpack the “black boxes”!

148
© 2024 YAN Zi EdUHK
 Key Research Area --- Assessment Research into Enhancing Learning
and Teaching

• https://www.eduhk.hk/artist/

© 2024 YAN Zi EdUHK 149

 Books

150
© 2024 YAN Zi EdUHK
 Journal Publications on student-initiated assessment-as-learning
• Dawson, P., Yan, Z., Lipnevich, A., Tai, J., Boud, D., & Mahoney, P. (2023). Measuring what learners do in feedback: The Feedback Literacy Behaviour Scale. Assessment & Evaluation in
Higher Education. https://doi.org/10.1080/02602938.2023.2240983
• Yan Z., Panadero E., Wang, X., & Zhan, Y. (2023). A systematic review on students’ perceptions of self-assessment: Usefulness and factors influencing implementation. Educational
Psychology Review, 35, 81. https://doi.org/10.1007/s10648-023-09799-1
• Yang, L., Yan, Z., Zhang, D., Boud, D., & Datu, J. A. (2023). Exploring the roles of academic self-concept and perseverance of effort in self-assessment practices. Assessment in
Education: Principles, Policy & Practice. https://doi.org/10.1080/0969594X.2023.2191161
• Zhan, Y., Yan, Z., Wan, Z. H., Wang, X. Zeng, Y., Yang, M., & Yang, L. (2023). Effects of online peer assessment on higher-order thinking: A meta-analysis. British Journal of Educational
Technology, 54(4), 817-835. https://doi.org/10.1111/bjet.13310.
• Mendoza, N., & Yan, Z. (2023). Exploring the moderating role of well-being on the adaptive link between self-assessment practices and learning achievement. Studies in Educational
Evaluation, 77, 101249. https://doi.org/10.1016/j.stueduc.2023.101249
• Yan, Z., Brubacher, S. P.*, Boud, D., & Powell, M. (2023). The influence of self-assessment practices on online training for investigative interviewing skills. Studies in Educational
Evaluation, 76, 101236. https://doi.org/10.1016/j.stueduc.2023.101236
• Mendoza, N.B., Yan, Z., & King R.B. (2023). Supporting students’ intrinsic motivation for online learning tasks: The effect of need-supportive task instructions on motivation, self-
assessment, and task performance. Computers & Education, 193 ， 104663. https://doi.org/10.1016/j.compedu.2022.104663.
• Mendoza, N., Yan, Z., & King, R. (2023). Domain-specific motivation and self-assessment practice as mechanisms linking perceived need-supportive teaching to student
achievement. European Journal of Psychology of Education, 38, 607–630. https://doi.org/10.1007/s10212-022-00620-1
• Yan, Z., Wang, X., Boud, D., & Lao, H. (2023). The effect of self-assessment on academic performance and the role of the explicitness: A meta-analysis. Assessment & Evaluation in
Higher Education, 48(1), 1-15. https://doi.org/10.1080/02602938.2021.2012644
• Yan, Z., Lao, H., Panadero, E., Fernández-Castilla, B., Yang, L., & Yang, M. (2022). Effects of self-assessment and peer-assessment interventions on academic performance: A meta-
analysis. Educational Research Review, 37, 100484. https://doi.org/10.1016/j.edurev.2022.100484

151
© 2024 YAN Zi EdUHK
 Journal Publications on student-initiated assessment-as-learning
• Wang, X., Yan, Z.*, Huang, Y., Tang, A., & Chen, J. (2022). Re-developing the Adversity Response Profile for Chinese university students. International Journal of Environmental
Research and Public Health, 19(11), 6389. https://doi.org/10.3390/ijerph19116389
• Yan, Z., Lee, C. K. J., Hui, K. F. S., & Lao, H. L. (2022). Enhancing students’ self-efficacy in creativity and learning performance in the context of English learning: The use of self-
assessment mind-maps. Frontiers in Psychology, 13, 871781. https://doi.org/10.3389/fpsyg.2022.871781
• Mendoza, N., Cheng, E. C. K., & Yan, Z. (2022). Assessing teachers' collaborative lesson planning practices: Instrument development and validation using the SECI knowledge-
creation model. Studies in Educational Evaluation, 73, 101139. https://doi.org/10.1016/j.stueduc.2022.101139
• Yan, Z., & Carless, D. (2022). Self-assessment is about more than self: The enabling role of feedback literacy. Assessment & Evaluation in Higher Education, 47(7), 1116-1128.
https://doi.org/10.1080/02602938.2021.2001431
• Yan, Z. (2021). Assessment-as-learning in classrooms: The challenges and professional development. Journal of Education for Teaching, 47(2), 293-295.
https://doi.org/10.1080/02607476.2021.1885972
• Mendoza, N. B., & Yan, Z. (2021). Validation of a subject-specific student Self-assessment Practices Scale among secondary school students in the Philippines. Journal of
Psychoeducational Assessment, 39(4), 481-493. https://doi.org/10.1177/0734282921994374
• Yan, Z., Chiu, M. M., & Ko, P. Y. (2020). Effects of self-assessment diaries on academic achievement, self-regulation, and motivation. Assessment in Education: Principles, Policy &
Practice, 27(5), 562-583. https://doi.org/10.1080/0969594X.2020.1827221
• Yan, Z., Brubacher, S., Boud, D., & Powell, M. (2020). Psychometric properties of the Self-assessment Practice Scale for professional training contexts: Evidence from confirmatory
factor analysis and Rasch analysis. International Journal of Training and Development, 24(4), 357-373. https://doi.org/10.1111/ijtd.12201
• Yan, Z. (2020). Self-assessment in the process of self-regulated learning and its relationship with academic achievement. Assessment & Evaluation in Higher Education, 45(2), 224-
238. https://doi.org/10.1080/02602938.2019.1629390
• Yan, Z., Brown, G. T. L., Lee, C. K. J., & Qiu, X. L. (2020). Student self-assessment: Why do they do it? Educational Psychology, 40(4), 509-532.
https://doi.org/10.1080/01443410.2019.1672038

152
© 2024 YAN Zi EdUHK
 Journal Publications on student-initiated assessment-as-learning
• Yan, Z. (2020). Developing a short form of the Self-assessment Practices Scale: Psychometric Evidence. Frontiers in Education, 4, 153.
https://doi.org/10.3389/feduc.2019.00153
• Yan, Z. (2018a). The Self-assessment Practice Scale (SaPS) for students: Development and psychometric studies. The Asia-Pacific Education Researcher, 27(2), 123-135.
https://doi.org/10.1007/s40299-018-0371-8
• Yan, Z. (2018b). Student self-assessment practices: the role of gender, year level, and goal orientation. Assessment in Education: Principles, Policy & Practice, 25(2), 183-199.
https://doi.org/10.1080/0969594X.2016.1218324
• Yan, Z., & Brown, G. T. L. (2017). A cyclical self-assessment process: Towards a model of how students engage in self-assessment. Assessment & Evaluation in Higher
Education, 42(8), 1247-1262. https://doi.org/10.1080/02602938.2016.1260091
• Yan, Z. (2016). The self-assessment practices of Hong Kong secondary students: Findings with a new instrument. Journal of Applied Measurement, 17(3), 335-353.
https://www.researchgate.net/publication/310443914_The_Self-assessment_Practices_of_Hong_Kong_Secondary_Students_Findings_with_a_New_Instrument

153
© 2024 YAN Zi EdUHK
 Journal Publications on teacher-directed assessment-for-learning
• Li, Z., Yan, Z.*, Chan, K. K. Y., Zhan, Y., & Guo, W. Y. (2023). The role of professional development program in improving primary teachers’ formative assessment literacy. Teacher
Development, 27(4), 477-467. https://doi.org/10.1080/13664530.2023.2223595
• Yan, Z., & King, R. B. (2023). Assessment is contagious: The social contagion of formative assessment practices and self-efficacy among teachers. Assessment in Education: Principles,
Policy & Practice, 30(2), 130-150. https://doi.org/10.1080/0969594X.2023.2198676
• Yan, Z., & Chiu, M. M. (2023). The relationship between formative assessment and reading achievement: A multilevel analysis of students in 19 countries/regions. British Educational
Research Journal, 49, 186-208. https://doi.org/10.1002/berj.3837
• Yan, Z., & Pastore, S. (2022). Are teachers literate in formative assessment? The development and validation of the Teacher Formative Assessment Literacy Scale. Studies in
Educational Evaluation, 74, 101183. https://doi.org/10.1016/j.stueduc.2022.101183
• Yan, Z., & Pastore, S. (2022). Assessing teachers’ strategies in formative assessment: The Teacher Formative Assessment Practice Scale. Journal of Psychoeducational Assessment,
40(5), 592-604. https://doi.org/10.1177/07342829221075121
• Yan, Z., Chiu, M. M., & Cheng, E. C. K. (2022). Predicting teachers’ formative assessment practices: Teacher personal and contextual factors. Teaching and Teacher Education, 114,
103718. https://doi.org/10.1016/j.tate.2022.103718
• Yan, Z., King, R. B., & Haw, J. Y. (2021). Formative assessment, growth mindset, and achievement: Examining their relations in the East and the West. Assessment in Education:
Principles, Policy & Practice, 28(5-6), 676-702. https://doi.org/10.1080/0969594X.2021.1988510
• Yang, L., Chiu, M. M., & Yan, Z. (2021). The power of teacher feedback in affecting student learning and achievement: insights from students’ perspective. Educational Psychology,
41(7), 821-824. https://doi.org/10.1080/01443410.2021.1964855
• Yan, Z., Xiao, Y., Sin, K-F., Yang, L., & Guo, W-Y. (2021). Formative assessment practices in special school classrooms with the support of e-books: A case study. Frontiers in Education,
6. https://doi.org/10.3389/feduc.2021.674869
• Yan Z., Li, Z., Panadero, E., Yang, M., Yang, L., & Lao, H. (2021). A systematic review on factors influencing teachers’ intentions and implementations regarding formative assessment.
Assessment in Education: Principles, Policy & Practice, 28(3), 228-260. https://doi.org/10.1080/0969594X.2021.1884042

154
© 2024 YAN Zi EdUHK
 Journal Publications on teacher-directed assessment-for-learning

• Yan, Z., & Brown, G. T. L. (2021). Assessment for learning in the Hong Kong assessment reform: A case of policy borrowing. Studies in
Educational Evaluation, 68, 100985. https://doi.org/10.1016/j.stueduc.2021.100985
• Guo, W. Y., & Yan, Z. (2019). Formative and summative assessment in Hong Kong primary schools: Students’ attitudes matter. Assessment in
Education: Principles, Policy & Practice, 26(6), 675-699. https://doi.org/10.1080/0969594X.2019.1571993
• Yan, Z., & Cheng, E. C. K. (2015). Primary teachers’ attitudes, intentions and practices regarding formative assessment. Teaching and Teacher
Education, 45, 128-136. https://doi.org/10.1016/j.tate.2014.10.002
• Yan, Z. (2014). Predicting teachers’ intentions to implement school-based assessment using the Theory of Planned Behaviour. Educational
Research and Evaluation, 20(2), 83-97. https://doi.org/10.1080/13803611.2013.877394

155
© 2024 YAN Zi EdUHK
 Thank you ！

[email protected]
29486367

© 2024 YAN Zi EdUHK 156