RESEARCH REVIEW:

Data-driven decision making in education agencies

Data Driven, not

Data Drowning 1

Being driven by data requires more than

the existence of a data infrastructure,
accessible data, and a culture of data use.
It also requires careful attention that data
are both relevant and diagnostic for each
decision maker and decision. Meaningful use of data begins with who will access,
analyze, or review the data and for what purpose. The figure below displays several
examples of purposes that education decision makers have for using data.

Who will access, analyze,

For what purposes
or review the data
Classroom teachers • Assessing the needs, strengths, progress, and performance of students
• Developing and revising classroom instruction
• Understanding professional strengths and weaknesses

School administrators • Assessing the needs, strengths, progress, and performance of staff
and students
• Developing and revising school plans, targets, and goals
• Monitoring the implementation and impact of school practices, programs,
and policies
Superintendents, school • Assessing the needs, strengths, progress, and performance of schools,
boards, district staff, staff, and students
charter management • Developing and revising district curricula, standards, plans, targets,
organization leaders, and goals
charter authorizers • Monitoring the implementation and impact of district practices, programs,
and policies
State education agency • Monitoring statewide achievement and attainment levels, overall and
officials for subgroups, statewide and by school/district
• Monitoring and reporting measures of school performance
• Measuring teacher effectiveness
• Monitoring human capital pipeline
• Evaluating program implementation and impacts
• Developing and revising state standards, curricula, and goals
1
The information presented in this infographic is presented in more detail at https://www.mathematica-mpr.com/our-publications-and-findings/publica-
tions/a-conceptual-framework-for-data-driven-decision-making.

https://ies.ed.gov/ncee/edlabs/regions/midatlantic/
 What’s so important about data that is relevant and diagnostic?

Data must be relevant to the decision maker in order to guide the

improvement of practice or outcomes. Its relevance may depend,
for example, on whether data are related to students, staff, or programs
or on how frequently data are updated and delivered.

Data also must be diagnostic for the issue at hand, which means
it must be reliable and valid (see text boxes). The same data can
be diagnostic for some decisions and not for others. Schoolwide
student growth, for example, might be diagnostic for identifying
a high-performing school, but by itself has limited use for identifying
why the school is high performing.

Even when data are reliable, they may not be valid for informing
Reliable measures do not show big,
the decision at hand. Validity depends on the purpose for which
random changes up or down. If a school’s
data are used. Schoolwide standardized test scores may be valid
performance is in the top 10 percent one
measures of average student achievement levels, but may be
year and the bottom 10 percent the next,
misleading measures of principal performance (if, for example,
the measure is probably unreliable.
a principal is new to the school).

Correct diagnosis can become increasingly challenging at higher levels because the
decisions require correctly identifying underlying causes—in circumstances with many
possible causes. A well-designed assessment can quickly help a teacher diagnose
specific skills that a particular student needs to work on. When a principal sees that an
entire class isn't doing well, however, an examination of average student test scores
isn't enough information to conclude that the teacher needs to do better or how to
support improvement. And when a school district has a chronically low-performing
school, there is an even wider range of possible explanations—including leadership,
teacher quality, curriculum, external factors, and many others—that could have
produced the low achievement, so diagnosing the right explanations (and subsequently
designing and implementing an appropriate intervention) is even more difficult. The
figure below illustrates how the level of detail, the frequency of data collection, and the
difficulty of making strong inferences vary for decision makers at different levels.

Data Semi- Infrequent

Frequent frequent annual,
refresh weekly,
rate quarterly, semi-
daily monthly annual
Relevant

Level of Specific Detailed Less

detail individuated disaggregated detailed

Are the data sufficiently reliable to use for decision making?

Diagnostic

Inferential Least More Most

difficulty difficult difficult difficult
Are the data valid for informing the decision at hand?

This work was funded by the U.S. Department of Education’s Institute of Education Sciences (IES) under contract ED-IES-17-C-0006, with REL Mid-Atlantic, administered by Mathematica
Policy Research. The content does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or
organizations imply endorsement by the U.S. government.