Scribd
Upload
20 views

Applying Reliability Information

This document discusses applying reliability information from tests, including the standard error of measurement and how it relates to test reliability. It also covers how to use the standard error of measurement to calculate confidence intervals around scores and evaluate differences between scores.

Uploaded by

Gasai Yuno
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
20 views

Applying Reliability Information

This document discusses applying reliability information from tests, including the standard error of measurement and how it relates to test reliability. It also covers how to use the standard error of measurement to calculate confidence intervals around scores and evaluate differences between scores.

Uploaded by

Gasai Yuno
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 34

APPLYING RELIABILITY

INFORMATION
APPLYING RELIABILITY INFORMATION
○ To determine the extent of measurement error in a test
○ The presence of error leads to two conclusions about test
scores
■ Should always be viewed as estimates of an individual’s
knowledge or characteristics
■ Decisions based on test scores should always take into
consideration the possibility of such chance variation
STANDARD OF ERROR
OF MEASUREMENT
(SEm)
STANDARD OF ERROR OF MEASUREMENT
● Standard deviation of errors of measurement that are
associated with test scores
● Allows us to quantify the extent to which a test provides
accurate scores
● For example, a student gets an IQ score of 80.
● How confident are we that the person’s true IQ score is 80?
STANDARD OF ERROR OF MEASUREMENT

SEM = SD x √(1-r)
● Standard deviation of the sample scores multiplied by the
square root of 1 minus the reliability (precision) of the scores
STANDARD OF ERROR OF MEASUREMENT
● Directly related to test reliability
● Uses the reliability coefficient to determine the average
number of points by which test scores and true scores differ
○ The larger the SEM, the lower the test’s reliability.
■ If test reliability = 0, the SEM will equal the standard
deviation of the observed test scores.
■ If test reliability = 1.00, the SEM is zero .
STANDARD OF ERROR OF MEASUREMENT
● Consider an test with an SD of 10
● Let’s quantify the SEM for a score of 100 under four different
conditions
○ Reliability = 0.9
○ Reliability = 0.8
○ Reliability = 0.5
○ Reliability = 0.2
r = 0.9 SD = 10 good measure

Equation SEM = SD x √(1-r)

Substitute SEM = 10 x √(1-0.9)

Calculate the SEM SEM = 3.16


good measure

r = 0.8 SD = 10 good measure

Equation SEM = SD x √(1-r)

Substitute SEM = 10 x √(1-0.8)

Calculate the SEM SEM = 4.47


r = 0.5 SD = 10 poor measure

Equation SEM = SD x √(1-r)

Substitute SEM = 10 x √(1-0.5)

Calculate the SEM SEM = 7.07


r = 0.2 SD = 10 really poor measure

Equation SEM = SD x √(1-r)

Substitute SEM = 10 x √(1-0.2)

Calculate the SEM SEM = 8.94


STANDARD OF ERROR OF MEASUREMENT

SCORE

55 65 75 85 95

10 10
CONFIDENCE INTERVALS
CONFIDENCE INTERVALS
● We use the SEM to calculate confidence intervals around
obtained scores
● Because all test scores include measurement error and only
estimate true scores, it is useful to convert individual test
scores to ranges within which the true score is likely to fall
● We use the person’s test score and SEM to infer what would
happen if the person were tested repeatedly
CONFIDENCE INTERVALS
● Common SEM confidence intervals and their formulas:

68%CI = Score ±SEM


95%CI = Score ±(1.96*SEM)
99%CI = Score ±(2.58*SEM)
CONFIDENCE INTERVALS
SEM = 15 x √(1-.70) 95%CI = 1.96 x 8.22
SEM = 8.22 95%CL = 16.22

100 95%CI = 100 + 16.11 = 116.11

100 95%CI = 100 - 16.11 = 83.89

68%CI = 8.22 99%CI = 2.58 x 8.22

100 68%CI = 100 + 8.22 = 108.22 99%CI = 21.21

100 68%CI = 100 - 8.22 = 91.78 100 68%CI = 100 +21.21 = 121.21

100 68%CI = 100 - 21.21= 78.79


CONFIDENCE INTERVALS
SEM = 15 x √(1-.90) 95%CI = 1.96 x 4.74
SEM = 4.74 95%CL = 9.29

100 95%CI = 100 + 9.29 = 109.29

100 95%CI = 100 -9.29 = 90.71

68%CI = 4.74 99%CI = 2.58 x 4.74

100 68%CI = 100 + 4.74 = 104.74 99%CI = 12.22

100 68%CI = 100 - 4.74 = 95.26 100 68%CI = 100 + 12.22 = 112.22

100 68%CI = 100 - 12.22 = 112.22


CONFIDENCE INTERVALS
SEM = 15 x √(1-.70) 95%CI = 1.96 x 8.22
SEM = 8.22 95%CL = 16.11

100 95%CI = 100 + 16.11 = 116.11

100 95%CI = 100 - 16.11 = 83.89

68%CI = 8.22 99%CI = 2.58 x 8.22

100 68%CI = 100 + 8.22 = 108.22 99%CI = 21.21

100 68%CI = 100 - 8.22 = 91.78 100 68%CI = 100 +21.21 = 121.21

100 68%CI = 100 - 21.21= 78.79


CONFIDENCE INTERVALS
● Just because an instrument produces scores with a high level of
accuracy does not mean those scores are valid
● The size of the confidence interval increases as the extent of
error in test scores increases
EVALUATING THE
DIFFERENCES BETWEEN
TWO SCORES
EVALUATING THE DIFFERENCE BETWEEN TWO SCORES

● Since all test scores reflect some measurement error, we must


be careful when we compare test scores
○ For example, a person took an SAT and received a total
score (verbal + math) of 980. The person then enrolls in a
cram course designed to increase SAT scores, retakes SAT,
and receives a total score of 1100.
○ The test score has improved but why?
SEM OF A DIFFERENCE

SEMdiff = √(SEM test1)2 + (SEM test2)2


● The SEM of a difference is the standard deviation of the set of

possible difference scores that could occur on a set of tests

● Indicates the average amount by which test scores can be

expected to differ on the basis of chance

● How likely it is that a particular difference score will occur by

chance
SEM OF A DIFFERENCE
A student receives a score of 63 on a unit exam and a score of 72
on a retake using an alternative form of the exam

Observed score difference = 72 - 63 = 9

SEM form 1 = 4 points SEM form 2 = 5 points

SEMdiff = √42 + 52 = √16+25 = √41 = 6.4 points

68%CI = Score of a difference of ±6.4

95%CI = Score of a difference ±12.544


SEM OF A DIFFERENCE
● EXAMPLE
○ Mary Jones has T scores of 45 on one test and 60 on
another. The first test A has a reliability of 0.80 and the
second B of 0.90. Was her score on test B statistically
significantly better than her score on test A? The scale of
SD is 10 on both tests because both are T score scales. To
answer this question we must use four steps.
SEM OF A DIFFERENCE
● First step: calculating the SEM

SEM = 15 x √(1-.70)
Test A SEM = 10 x √(1-0.80) = 4.47
Test B SEM = 10 x √(1-0.90) = 3.16
SEM OF A DIFFERENCE
● Second step: Calculate the SEdiff

SEMdiff = √(SEM test1)2 + (SEM test2)2


SEMdiff = √(4.47)2 + (3.16)2
SEMdiff = 5.47
SEM OF A DIFFERENCE
● Third step: we need to evaluate the difference (or distance)
between scores and see if this is greater or less the two
SEdiffs.
● One score is 45 and the other is 60. So the difference between
the two scores is 60 - 45. This is a distance of 15 (SEMdiffAB)
points which is greater than two SEdiffs (twice 5.47) another
method is to divide the distance by the SEMdiffAB to see if
the answer comes out bigger than 2

15 ፥ 5.47 = 2.74 SEdiffs


SEM OF A DIFFERENCE
● Fourth step: we have to decide on the confidence level
● If two scores differ by one SEdiff we can only be 68%
confident that the true scores are different. If they differ by
two or more SEdiffs we can be 95% confident the true scores
really differ. In this case they have a difference of 2.74 Sediffs
which is greater than the 95% confidence interval. So if the
difference is 2.74 SEdiffs we can be sure that the two scores
are statistically significantly different with 95% confidence
Therefore, Mary really is better on test B than on test A
EVALUATING COMPOSITE
OR AVERAGE SCORES
EVALUATING COMPOSITE OR AVERAGE SCORES

● In certain conditions, a person is evaluated not by a single test


score, but on the basis of a series of tests.
● It is a good idea to calculate a confidence interval around a
student’s total or average score before deciding on a final
grade.
EVALUATING COMPOSITE OR AVERAGE SCORES

● SEM of a total or average score

SEMean = SD/√N
EVALUATING COMPOSITE OR AVERAGE SCORES

A student receives the following five grades: 76. 74, 78, 83, 84
X = 79
SD = 6.8987
N=5

SEMean = 3.899/√5 = 1.744

95%CI = 79 ± (1.96)(1.744) = 79 ± 3.418 = 75.582 to 82.418


EVALUATING COMPOSITE OR AVERAGE SCORES

● Based on these test scores, our best estimate (95% confidence


interval) is that this student’s true average lies between 75.58
and 82.42. Now the grade can be compared with other
students who have averages of 79 or 80 to see what grade
seems most appropriate.
ref
http://www.fldoe.org/core/fileparse.php/7567/urlt/y1996-7.pdf

https://home.apu.edu/~bsimmerok/WebTMIPs/Session6/TSes6.html

https://www.statisticssolutions.com/composite-scoring-and-reliability/

https://www.statisticshowto.datasciencecentral.com/standard-error-of-measurement/

You might also like