National Examinations Development Executive

(NEDE)
Constructing Tables of
Specifications
Werabe, CERS
April 08, 2024
Fikremariam Regassa
CONTENTS

Introduction
I. Defining “Test Item”
II. Educational Tests
III. The Concept of Validity in Educational Tests
IV. Test Development

Presented by: Fikremariam Regassa, EAES April, 2024

Integration of Curriculum, Instruction and Assessment

Curriculum

Instruction

Assessment/Examination

The gears process depiction of the interlocking of the three basic element in an
Educational System.
A system is defined as a set of related components,
3 activities, processes, and human
beings interacting together so as to accomplish some Objective.
What is a ‘Test Item’?
 I. Defining “Test Item”
* Test items are the units that make up a test and the means
through which samples of test takers’ behavior are
gathered.

* It follows that the overall quality of a test depends

primarily on the quality of the items that make it up,
although the number of items in a test, and their
sequencing or position within the test, are also matters of
fundamental importance.

* individual items must be evaluated based on the extent to

which they meet the purposes of the test as a whole.
I. Defining “Test Item” (conti.)
The basic test item is fundamentally a stimulus-response
(S→R) pair.
We present a stimulus in the form of a question or task.
The examinee provides the response upon which we
determine the proper score, interpretation, and decision
for action.
Based on the responses from examinees, test items can be
roughly divided into two major types: The select-
response type and the generate-(constructed) response
type items.
 I. Defining “Test Item” (conti.)

Osterlind (1990a) offered this definition of a test item:

* A test item in an examination of mental attributes
is a unit of measurement with a stimulus and a
prescriptive form for answering; and, it is intended
to yield a response from an examinee from which
performance in some psychological construct (such
as knowledge, ability, predisposition, or trait) may
be inferred. (p. 3)
(in this definition bolding, the terms above is added.)
I. Defining “Test Item” (conti.)

Unconditional response Conditional response

I. Defining “Test Item” (conti.)
Criteria For Good Test Items
• The first criterion for constructing good test items, and
the most important one, is that there must be a high
degree of congruence between (alignment with) a
particular item and the key objective of the total test.
• Simply stated, the primary question is, “How well does a
particular test item match its intended objective?” (Here,
the term objective is being used as a synonym for a
competency (in our case) OR construct.)
• This congruence (alignment) criterion is the item writer’s
primary consideration because it is at the heart of validity,
the most important consideration in test construction.
I. Defining “Test Item” (conti.)
The second criterion for constructing good test items is
closely allied with the first.
This criterion is that the key objectives must be clearly defined.
If a test item is to meet the congruence criterion, it must be
matched to a defined entity.

To state loosely that an item measures a very broad concept, like

critical thinking, without further defining what is meant by
critical thinking, diminishes its potential for yielding validity
evidence.

The third criterion of constructing good test items is that each

item’s contribution to measurement error in a test’s scores should
be minimized to the extent possible.
I. Defining “Test Item” (conti.)
The third criterion means that there should be a
systematic consideration of the degree to which test
items may contribute to errors of measurement.
The fourth criterion for good test items is that the format be
suitable to the goals of the test.
Straightforward, uncomplicated goals generally require simpler
item formats than those necessary for assessing complex goals.

The fifth criterion for constructing good test items is that each
item meet specific technical assumptions.
These assumptions are psychometric concerns of measurement
theory.
I. Defining “Test Item” (conti.)
The sixth criterion for constructing good test items is that they
should be well written, following uniform style or editorial
standards.
These standards cover grammar, diction, usage, spelling,
punctuation, and syntax.

The seventh and final criterion for constructing good test

items is that they satisfy legal and ethical questions.
 Sometimes, test item writers may be tempted to use another’s
set of well-constructed test items to save time or effort. Often,
little or no consideration is given to matching the purposes of
their own test.
 Regretfully, this form of plagiarism happens too often. Not
only is this unethical, it can be illegal.
 I. Defining “Test Item” (conti.)
Assumptions For Test Items
If a test item is to be used properly for educational measurement, one
must understand the relevant theoretical assumptions and give them
ample consideration throughout the process of developing the item.
The Assumption of Unidimensionality for Items
The first assumption underlying a mathematical model for test items is
that the items are unidimensional.

Unidimensionality of a test item means that an examinee’s

response to a test item can be (by inference) attributed to a single
trait or ability.
In other words, a test item is designed to measure one, and not
more than one, psychological construct.
I. Defining “Test Item” (conti.)
The Assumption of Local Independence for Items
The second assumption for test items is local independence.
*In practice, local independence means that an examinee’s
response on any particular test item is unaffected and
statistically independent from a response to any other test
item.
*In other words, local independence presumes that an
examinee approaches each test item as a fresh, new
problem without hints or added knowledge garnered from
responding to any other test item.
II. Educational Tests
 II. Educational Tests
What Are Educational Tests?
According to many measurement experts (like Susana Urbina, 2004)
An educational test is a systematic procedure for obtaining
samples of behavior, relevant to cognitive OR affective
functioning, and for scoring and evaluating those samples
according to standards.
Educational tests are often described as standardized for two
reasons, both of which address the need for objectivity in the
testing process.
1- uniformity of procedure in all important aspects
2- the use of standards (derived from normative group) for
evaluating test results.
 II. Educational Tests (conti.)
Standardized Testing in Educational Settings
Achievement is usually thought of as planned changes in
cognitive behavior that result from instruction or training,
although certainly achievement is possible because of factors
outside of instruction or training.
All achievement can be defined in terms of content. This content
can be represented as knowledge, skills, or cognitive abilities.
An Achievement Test is a systematic procedure for determining
the amount a student has learned. (Norman E. Gronlund, 1977)
They are also used in other situations, including personnel
selection, that require the assessment of mastery of a given
field of knowledge.
III. The Concept of Validity
in Educational Tests
 III. The Concept of Validity in Educational Test
• Validity
Let us consider, however, a fuller description of validity. Citing
the AERA/APA/NCME Standards (1985):

• Validity is the most important consideration in test development.

The concept refers to the appropriateness, meaningfulness, and
usefulness of the specific inferences made from test scores.

• Test validation is a process of accumulating evidence to support

such inferences.

• A variety of inferences may be made from scores produced by a

given test, and there are many ways of accumulating evidence to
support any particular inference.
III. The Concept of Validity in Educational Tests (Con.)
Validity, however, is a unitary concept. Although evidence may be
accumulated in many ways, validity always refers to the degree to
which that evidence supports the inferences that are made from the
scores. The inferences regarding specific uses of a test are validated,
NOT the test itself. (p. 9).

A Complete Definition of Validity

According to the most recent edition of the Standards for
Educational and Psychological Testing (AERA, APA, NCME, 1999),
* “Validity refers to the degree to which evidence and theory support
the interpretations of test scores entailed by proposed uses of test
scores” (1999, p. 9).
III. The Concept of Validity in Educational Tests (Con.)

*The sponsor of the testing program (in our case EAES)

creates a logical argument and assembles validity evidence
supporting that argument.

*The investigative process of creating this argument and

collecting validity evidence in testing this argument is
validation.

*The quality of test items depends on two complementary

activities, the statistical analysis of item responses featured
and the item review procedures.
III. The Concept of Validity in Educational Tests (Con.)
We can study item response patterns using a
statistical theory that explains item responses
variation (Psychometrics), or we can study item
responses in a theory-free context, which is more
intuitive and less complex.

Psychometrics is a scientific discipline concerned

with the construction of assessment tools,
measurement instruments, and formalized models
that may serve to connect observable phenomena
(e.g., responses to items in an IQ-test) to theoretical
attributes (e.g., intelligence).
III. The Concept of Validity in Educational Tests (Con.)

*Most of educational tests are designed to sample behavior

that can be linked more or less directly to the inferences
we wish to make based on their scores.

*By and large, these instruments fall in the category of

Content or Criterion-Referenced Tests (CRTs ).

*Content is viewed as an important source of validity

evidence.

*Each item has an important content identity that conforms

to the test specification.
III. The Concept of Validity in Educational Tests (Con.)

*The primary requirement for developing

tests of this sort is a careful specification of
the content domains, cognitive processes,
skills, or types of performance to be
sampled by the test and of their relative
importance or weight.

*Expert judgment is needed to ensure that

every item is correctly classified by
content.
IV. Test Development
 IV. Test Development
 For an achievement test, a content analysis is vital.

 The resulting analysis report shows that the content has

been defined and revealed its meaning in a satisfactory way
so that the basis for the test is well grounded in the
curriculum (Syllabus).

The Standards (AERA, APA, & NCME, 1999, p.45) are very clear about this
need.

 Standard 3.11 states that “test developers should

document the extent to which the content domain of a
test represents the defined domain and test
specifications.”
 IV. Test Development (conti.)
What is Table of Curriculum Specifications?
• The development of Curriculum Specification necessarily
precedes any test writing process because it helps to
develop the respective “Table of Test Content
Specifications“ and “Table of Test Item Specifications" that
guide the test construction activity.

• Curriculum Specifications identifies the competencies

(knowledge and skills) which are to be tested and the
relative weight on the test given to each.
IV. Test Development (conti.)
 IV. Test Development (conti.)
• The development of this specifications is the crucial first
step in the test development process.

• Curriculum Specification is a list or table of Competencies

and their classification according to Bloom’s Taxonomy
or Revised Bloom’s Taxonomy

• The Original and the Revised Bloom’s Taxonomy

depicted as a pyramid, with the lower levels at the
bottom of the figure, indicating a larger body of
knowledge available in foundational material.
 IV. Test Development (conti.)
Bloom’s Taxonomy [1956 ] &
Bloom’s Cognitive Process Dimensions [2001]
Knowledge -- Define, duplicate, label, list, name, Remember Retrieve knowledge from long-term
order, recognize, relate, recall memory, recognize, recall, locate, identify

Comprehension -- Classify, describe, discuss, Understand -- Construct meaning, clarify,

explain, express, identify, indicate, locate, paraphrase, represent, translate, illustrate, give
recognize, report, review, select, translate examples, classify, categorize, summarize,
generalize, predict…
Application -- Apply, choose, demonstrate, Apply -- Carry out or use a procedure in a given
dramatize, employ, illustrate, interpret, practice, situation; carry out or use /apply to an unfamiliar
write task
Analysis -- Analyze, appraise, explain calculate, Analyze -- Break into constituent parts, determine
categorize, compare, criticize, discriminate, how parts relate
examine
Synthesis -- Rearrange, assemble, collect, Evaluate -- Make judgments based on criteria,
compose, create, design, develop, formulate, check, detect inconsistencies/fallacies, critique
manage, write
Evaluation -- Appraise, argue, assess, choose, Create -- Put elements together to form a coherent
compare, defend, estimate, explain, judge, predict, whole, reorganize elements into new patterns/
rate, core, select, support, value structures
ORIG. & REVI. BLOOM’S TAXONOMY for TEST BLUEPRINT DEVELOPMENT

Original Revised

Evaluation Creating

Synthesis Evaluating

Analysis Analyzing

Application Applying

Comprehension Understanding

Knowledge Remembering
Noun Verb
 IV. Test Development (conti.)
What is Table of Test Content Specifications ?
• The test specifications development is done by using a
two way grid table known as Table of Test Content
Specification having
 Content (in a vertical column)
 Cognitive dimensions (in a horizontal row)
 Relative weight assign to each cell
• Cognitive dimension classified according to either in
Bloom’s or Revised Bloom’s taxonomy (listed from
simplest mental operation to the most complex one)
• Each test item created falls into one cell.
• By creating this Table of Test Content Specification, it
can be seen the relative weight assign to each cell.
 IV. Test Development (conti.)
Constructing the Table of Test Content Specifications
Step: 1. Determine the Topics to be Included in the Test
•Course contents to be Covered by the Test,
•Competencies (Knowledge and Skills) in accordance with
the Testing Modalities,
•Cognitive Dimensions (Behavioral Dimension) across content
•Analyze results by level and content area

Step 2. Determine the Competencies to be assessed by the Test

Step 3: Determine the number of total number of test Items

For example we want to develop Math Test for Gr-11 with 50 Items
 IV. Test Development (conti.)
Step 4: Determine the Number of Items per Topic or Content Area

Formula : Total Number of Items = Result (Const.)

Total Number of Periods
Number of Items from a Topic = Result × Allotted Periods for the Topic

Example
Total number of Items : 50
Total number of Periods/Time/ : 156
Allotted period for a pertinent topic : 18 (Unit-1)
Calculation
50/156 = 0.32051 ≈ 0.32 (Const.)
0.32 × 18 ≈ 5.77 = 6 Items (Question) from this topic (Unit-1)
 IV. Test Development (conti.)
Example of Table of Test Content Specification for GRr-11 Math
GRADE 11 CONTENTS No. of No. of No. of Items from Each Unit
K C A An Sy Ev
(UNITS) Periods Compe. (If the Test Contains 50 Items)

11.1 Relations and Functions 18 7 8 8 10 1 – 34 5.77 6

11.2: Rational Expressions and

18 2 4 4 3 2 – 15 5.77 6
Rational Functions
11.3: Matrices 25 1 3 12 4 2 – 23 8.01 8
11.4: Determinants and their
28 – 1 6 4 – – 11 8.97 9
properties
11.5: Vectors and Transformation
16 4 5 12 2 – – 22 5.13 5
of the Plane
11.6: Transformation on the Plan 10 – – 6 1 – – 7 3.21 3
11.7: Statistics 20 3 3 13 – – – 19 6.41 6
11.8: Probability 21 7 12 18 10 3 – 50 6.73 7
156 24 36 79 34 8 0 181 50 50
60 79 42
33% 44% 23%
 IV. Test Development (conti.)
Step 5: Determine the level of Difficulty and their
percentages, in the Examination
 IV. Test Development (conti.)
Step 6: Determine the number of Items for Each Pertinent Topic per
Level of Difficulty

Step 7: Make Sure The Vertical and The Horizontal Sum of the Items
are Correct

Step 8: Write the Item Placement for Each Number of Items

Step 9: Complete the Table of Test Content Specifications, Making

Sure Everything is in Place
IV. Test Development (conti.)
Example of Table of Test Content Specification for GRr-11 Math
GRADE 11 CONTENTS No. of No. of No. of Items from Each Unit
K C A An Sy Ev
(UNITS) Periods Compe. (If the Test Contains 50 Items)

11.1 Relations and Functions 18 1 2 1 2 0 0 34 5.77 6

11.2: Rational Expressions and
18 1 2 1 1 1 0 15 5.77 6
Rational Functions
11.3: Matrices 25 0 1 4 2 1 0 23 8.01 8
11.4: Determinants and their
28 0 1 5 3 0 0 11 8.97 9
properties
11.5: Vectors and Transformation of
16 1 1 3 0 0 0 22 5.13 5
the Plane
11.6: Transformation on the Plane 10 0 0 3 0 0 0 7 3.21 3
11.7: Statistics 20 1 1 4 0 0 0 19 6.41 6
11.8: Probability 21 1 2 3 1 0 0 50 6.73 7
156 5 10 24 9 2 0 181 50 50
15 24 11
30% 48% 22%
 IV. Test Development (conti.)
TEST ITEM SPECIFICATIONS
•Test Item Specifications are a specialized kind of
technical writing used in developing a set of items.
•Just as Test Content Specifications describe the
content and intellectual processes for an entire test,
item specifications give directions for preparing
particular items.
•However, the similarity between Test Content
Specifications and Item Specifications is only
general.
 IV. Test Development (conti.)
TEST ITEM SPECIFICATIONS
•Test Item Specifications are NOT merely a more
specific version of Test Content Specifications. They
differ in purpose, scope, and function.
•Test Item Specifications are formal, systematized
directions from a test developer to the item writer that
seek to put the test content specifications into action.
•They may include such information as eligible item
formats, kinds of directions, limits for the stem,
characteristics of the response alternatives, as well as
features for the correct response and distractors.
 Behvioural
CONTENTS and COMPETENCIES of GRADE 11 Dimension
(Bhev. Dimen )
UNIT 1 Relations and Functions (18 periods ) (Bhev. Dimen )
1.1 Relation and Inverse of Relation
1. • write domain and range of a relation in interval notation, inequalities and set notation. Comprehension
(2 period)
1.3 Types of Functions
5. • define power functions. Knowledge
(14 periods)
1.3.1 Power Functions with their graph 7. • describe the properties of powers functions in relation to their exponents. Analysis
1.3.2 Modulus Functions (Absolute Value
13. • determine the domain and the range of modulus function. Comprehension
Function)
1.4 Composition of functions 24. • determine the composite function given the component functions. Application
1.5 Inverse Functions and their Graphs 31. • relate the graph of the function and the graph of the inverse function. Analysis
UNIT 2: Rational Expressions and Rational Functions (18 periods ) (Bhev. Dimen )
2.1 Simplification of Rational
2. • identify the universal set of a given rational expression. Comprehension
Expressions
2.1.1 Operations with rational
4. • perform the four fundamental operations on rational expression. Application
expressions (2 periods)
2.1.2 Decomposition of rational
expression into partial fractions. 5. • decompose rational expressions into sums of partial fractions. Synthesis
(2 periods)
2.2 Rational Functions and their
6. • define rational function. Knowledge
graphs (10 periods)
2.2.2 Graphs of rational
10. • determine the intercepts and symmetry of the graph of a given rational function. Comprehension
functions
2.3 Applications of rational
expressions and functions 14. • formulate real life problems by rational functions and solve it. Analysis
(4 periods)
UNIT 3: Matrices (25 periods ) (Bhev. Dimen )
3.2 Operations in Matrices
(6 periods)
3. • identify order of a matrix. Knowledge
3.2.1 The transpose of a matrix
4. • determine the transpose of a matrix. Application
and its properties
3.2.4 Multiplication of a matrix by
7. • multiply a matrix by a scalar. Application
a scalar.
3.2.5 Matrix Multiplication 10. • describe the properties of the product of two matrices. Analysis

3.4 Elementary Operations of Matrices 14. • use row operations to solve system of equations. Analysis

3.6 Solutions of systems of Linear

17. • solve systems of equations in two or three variables using the elementary operations. Application
equations (4 periods)
3.6 Solutions of systems of Linear
18. • reduce the augmented matrix of the system into triangular matrix. Synthesis
equations
3.7 Inverse of a square matrix
19. • find inverse of a square matrix. Application
(3 periods)
 Behvioural
CONTENTS and COMPETENCIES of GRADE 11 Dimension
(Bhev. Dimen )
UNIT 4: Determinants and their properties (28 periods ) (Bhev. Dimen )
4.1 Determinants of order 2.
1. • determine the determinant of a square matrix of order 2. Application
(2 periods)
4.2 Minors and cofactors of the
2. • determine the Minor and Cofactor of a given element of a matrix. Analysis
elements of matrices. (2 periods)
4.3 Properties of determinants
4. • apply properties of determinants to compute determinants. Application
(4 periods)
4.3 Properties of determinants 5. • describe the properties of determinants. Analysis
4.5 Inverse of a square matrix
6. • use determinants to determine whether a square matrix is invertible. Analysis
(6 periods)
4.5 Inverse of a square matrix 7. • determine inverse of a square matrix using formula. Application

4.6 Systems of equations with two and

8. • find associated augmented matrix of system of equations. Comprehension
three variables (4 periods)

4.7 Solutions of linear system of

equations by using Cramer's Rule 10. • apply Cramer's rule to solve systems of linear equations. Application
(4 periods)
4.8 Applications of Determinants and
11. • interpret matrix concept in real life scenario. Application
their properties (3 periods)
UNIT 5: Vectors (16 Periods ) (Bhev. Dimen )
5.1 Revision on vectors and scalars
1. • define a scalar quantity. Knowledge
(2 periods)
5.2.2 Addition and subtraction of
7. • determine the sum of two or more vectors. Application
vectors.
5.2.3 Multiplication of vectors by
9. • multiply a vector by a scalar. Application
scalars
5..2.4 Unit vectors 10. • resolve a given vector in to two components. Comprehension
5..3.3 Applications of scalar
16. • use cross product of a vector to find area of parallelogram. Application
and cross product
UNIT 6: Transformation on the Plan (10 periods ) (Bhev. Dimen )
6.1 Transformations of the plane
Application
(8 periods) 1. • explain what is meant by transformation of the plane.
6.1.2 Reflection 4. • reflect points, lines, circle and some other plane figures. Application
6.2 Applications of Transformation of
6. • use the concept of vectors to solve different real life problems. Application
the Plane (2 periods)
 Behvioural
CONTENTS and COMPETENCIES of GRADE 11 Dimension
(Bhev. Dimen)

UNIT 7: Statistics (20 periods ) (Bhev. Dimen )

7.1 Types of data (2 periods)

7.1.1 Qualitative and 1. • identify qualitative and quantitative data. Knowledge
Quantitative data
4. • determine class interval (class size) as required to form grouped data from a given ungrouped data. Comprehension
7.2.1 Grouped discrete data 5. • make cumulative frequency table for grouped data (for both discrete and continuous). Application
7.4 Measures of Central Tendency
and their interpretation (7 periods) 10. • find the Mean of a given grouped data. Application
7.4.1 Mean for grouped data
7.4.3 Quartiles, Deciles and
Percentiles for Grouped 13. • determine the quartiles for a given grouped data. Application
Data.
7.5 Applications of statistics and
19. • apply statistical method to solve real life problems. Application
probability (4 periods)
UNIT 8: Probability (21 periods ) (Bhev. Dimen )
8.2.2 Combination 7. • describe what is meant by "combination of objects". Comprehension
8.2.2 Combination 8. • determine the number of different combination of n objects taken r at a time. Application
8.4 Random Experiments and its
20. • explain what is meant by an outcome of a random experiment.
∪
Knowledge
Outcomes
8.5.1 Types of Events 35. • identify independent events. Comprehension
8.6.4 The rules of Addition of
43. • find the probability of E1 E2 where E1 and E2 are events in a random experiment. Application
Probabilities
8.6.4 The rules of Addition of
44. • determine the probability of mutually exclusive events. Analysis
Probabilities
8.7 Applications of probability
50. • apply statistical method to solve real life problems. Application
(2 periods)
 IV. Test Development (conti.)
TMAKING AN ITEM CONSISTENT WITH ITS SPECIFICATION
The Importance of Item-Competency Alignment

Achieving the maximum degree of alignment between

the knowledge, skill, or ability actually assessed by an
item and the intention for that item (as articulated by its
specification) is of paramount concern to the item writer.

It directly affects the interpretation of an item for

content-related evidence of validity, and it influences the
error of measurement, or reliability.
 IV. Test Development (conti.)
The Importance of Item-Competency Alignment
Hence, the item-objective match is highly important.
Unfortunately, determining the strength of that match is
difficult.

It involves an awareness of the full implications of the

objective or skill to be assessed.

Several examples will illustrate this point. Let us see

the following two examples.
 IV. Test Development (conti.)
The Importance of Item-Competency Alignment
Example 1
To begin, suppose that following item is designed to
gauge a student’s skill in “solving simple problems
that involve proportions”.
Which is equal to 5 pounds?
A. 80 grams
B. 16 ounces
*C. 80 ounces
D. 7,000 grains
 IV. Test Development (conti.)
The Importance of Item-Competency Alignment
As can be seen from a casual inspection, this item
involves recognizing units of measure and is
unrelated to proportions.

Hence, the item-objective congruence is lacking.

This item is inappropriate for assessing this

objective because of this mismatch.
In contrast, consider the next item in Example 2, which
is meant to assess the same skill of using proportions.
 IV. Test Development (conti.)
The Importance of Item-Competency Alignment
Example 2
Student X spends 45 minutes each day exercising,
Student Y exercises for 15 minutes a day, and Student Z
exercises 60 minutes daily. What is the ratio of how
long X exercises to how long Z exercises?

A.
45 C.
60
120 45

B.
45 D.
45
60 15
 IV. Test Development (conti.)
The Importance of Item-Competency Alignment

This item in Example-2 does require the examinee to

use proportions, and thus exhibits a high degree of
alignment with the competency.

Of course, in this simple comparison of Illustrative Items

in Example 1 to Example 2, one should not concerned
with whether the level of item difficulty is appropriate to
the examinees, as well as other considerations for good
items; rather, it is used only to make the point of item-
objective congruence.