ReseaRch RepoRt

occasional papeR

Verifying Data Migration

Correctness:
The Checksum Principle
Bin Wei and Tennyson X. Chen

March 2014

RTI Press

About the Authors RTI Press publication OP-0019-1403

Bin Wei, MS, is a senior researcher at
the Pacific Islands Fisheries Science This PDF document was made available from www.rti.org as a public service
Center, a National Oceanic and of RTI International. More information about RTI Press can be found at
Atmospheric Administration (NOAA)
research branch within the Joint http://www.rti.org/rtipress.
Institute for Marine and Atmospheric
Research, a NOAA Cooperative
RTI International is an independent, nonprofit research organization
Institute at the University of Hawaii at dedicated to improving the human condition by turning knowledge into
Manoa.* His areas of expertise include practice. The RTI Press mission is to disseminate information about
designing systems for remote data
collection and data transformation. RTI research, analytic tools, and technical expertise to a national and
Tennyson X. Chen, MS, is a senior international audience. RTI Press publications are peer-reviewed by at least
research analyst and software system two independent substantive experts and one or more Press editors.
architect in RTI International’s Research
Computing Division. His main focus is
Suggested Citation
the National Survey of Drug Use and
Health (NSDUH) project, for which he Wei, B. and Chen, T. X. (2014). Verifying data migration correctness: The
is a key system designer and database
manager.
checksum principle (RTI Press publication OP-0019-1403). Research Triangle
*Although Mr. Wei is employed by the
Park, NC: RTI Press.
NOAA Joint Institute for Marine and
Atmospheric Research, this paper was
produced independently and is not a
work product of NOAA.

This publication is part of the

RTI Research Report series. Occasional
Papers are scholarly essays on policy,
methods, or other topics relevant to
RTI areas of research or technical focus.
RTI International
3040 East Cornwallis Road
PO Box 12194
Research Triangle Park, NC ©2014 Research Triangle Institute. RTI International is a trade name of Research Triangle

27709-2194 USA Institute.

Tel: +1.919.541.6000 All rights reserved. This report is protected by copyright. Credit must be provided to the author

Fax: +1.919.541.5985
and source of the document when the content is quoted. Neither the document nor partial or

E-mail: [email protected]
Website: www.rti.org entire reproductions may be sold without prior written permission from the publisher.

doi:10.3768/rtipress.2014.op.0019.1403 www.rti.org/rtipress
Verifying Data Migration Correctness: Contents

The Checksum Principle Introduction

errors in Data Migration
2
2
current Methods for Detecting
Data Migration errors 3
checksum Methodology in
network transmission error
Detection 4

Bin Wei and Tennyson X. Chen A New Data Migration

Correctness Verification
Method That Applies the
Checksum Methodology 5
implementation of the Method 5
Discussion 6
Future Work 7
References 8
Abstract Acknowledgments inside back cover
Data migration, generally referred to as the process of reading data from their source
and inserting them into a target database, is an important element of data extract,
transform, and load (ETL) systems. During data migration, errors can occur during
data transmission. These errors can directly affect the quality of the data in the target
database. Therefore, verifying the correctness of the outcome is a critical component of
a data migration operation. Current methods in data migration correctness verification
have many limitations, including incompleteness and inaccuracy. This paper describes
our innovative method that applies the well-proven checksum methodology to verify
the correctness of the data migration outcome. This method performs a thorough and
accurate verification on the correctness of the migrated data, mitigating most of the
weaknesses associated with current verification methods. This method is also easy to
implement and will greatly enhance the quality of data migration operations.
2 Wei and Chen, 2014 RTI Press

A number of factors can cause errors during data

Introduction migration. Leaving these errors undetected during
Many organizations, including service providers and the data migration process can cause data to not be
research organizations, rely on large amounts of data correctly transferred to the destination.
to manage their operations. Moving data is a common
and necessary operation in today’s information Errors in Data Migration
systems. As a result, designing and implementing Commonly, errors in data migration are caused
data extract, transform, and load (ETL) systems is by faulty hardware, network disturbances, format
a hot topic for programmers, developers, and the mismatches, and database definition mismatches.
computing community. For example, a corrupted hard disk or computer
According to Kimball and Caserta, “A properly memory can prevent the system from reading and
designed ETL system extracts data from the source transmitting the source table content correctly. If the
systems, enforces data quality and consistency system receives incorrect data from the source table,
standards, conforms data so that separate sources the data that are inserted into the target table cannot
can be used together, and . . . delivers data in a be correct.
presentation-ready format.”1[pxxi] ETL systems are Network transmission noise, interference, or
commonly used to transfer data from one database to distortion can also prevent an accurate migration.
another, to form data marts and data warehouses, and Data are migrated increasingly via the Internet, which
to convert fields in databases from one format or type can introduce errors to the data content. For example,
to another. An important function of an ETL system is in Table 1, an error during the transmission of the
the process of data migration. second record might change the City value from
Data migration can be defined many ways. According Amazon to Amazen. Although Internet protocols can
to Johny Morris,2[pxv] data migration is “the selection, detect and correct most transmission errors, some
preparation, extraction, transformation, and erroneous data can still be transmitted without being
permanent movement of appropriate data that is of detected.4
the right quality to the right place at the right time Format mismatch between the source and target
and the decommissioning of legacy data stores.” Data databases is the most common error that we have
migration is also known as “the one-time movement observed in designing data migration tools and
of data from a legacy source, or multiple sources, to a managing large-scale data migration projects.5 In
new target database.”3[p4] Although these definitions Table 1, the definition of the Address field is “string”
vary, data migration is generally defined as reading with a width of 35 characters. If the table in the
data from a source database and inserting them into a target database defines the same field with a width
target database. of only 20 characters, the Address information of

Table 1. Employees source table to be migrated

ID Name Address City State Zip Birthday Height Weight
1 John 1321 IPhone Road Apple XX 00011 1/7/1991 5.82 169
2 David 1322 Kindle street Amazon YY 00012 2/7/1992 6.18 193
3 Matthew 1323 IPad Road Apple XX 00013 3/7/1993 5.73 160
4 Mark 1324 Galaxy Lane Samsung ZZ 00014 4/7/1992 6.10 210
5 Luke 1325 Pink Drive Blue City YY 00015 5/7/1993 5.69 145
6 Skip 1326 Humuhumunukunukuapuaa Fish City NN 00016 6/7/2994 5.32 136
Avenue
 Data Migration Correctness Verification 3

the sixth record would be truncated to be “1326 the transfer. While these methods work well for a
Humuhumunukunuk” in certain database platforms small amount of data, using them in large-scale data
and result in the loss of address information. migration operations is prohibitively expensive in
Likewise, the definition of the Height field is decimal terms of time and effort.
with 2-digit precision in the source table. If the height
Programmers can also perform post-migration testing6
field is erroneously set up as an integer field or a
or validation testing.7,8 For these tests, programmers
decimal with 1-digit precision, the target table will
implement a series of predefined queries that examine
round all numbers received in this field up or down,
the content of the source and target databases. The
making the numbers unusable. In both scenarios,
result is a summary report that provides discrepancy
data lost during data migration result in inaccurate
information once the data migration operation is
and unusable data.
complete. In a previous work,5[p7] we illustrated one
In addition, errors are caused by different definitions example of validation testing as follows:
between the source and target databases in primary • For each field with “numeric” data type, the
key or constraint settings. This difference usually validation test compares maximum, minimum,
causes some records in the source table to be rejected average, and summary (indicating) values between
from the target table. For example, before migrating the two databases. For example, before migrating
data from Table 1, if the target table has ID field as its the data from Table 1, we can calculate these
primary key and the table already has a record with indicating values from the Weight field in the
value 5 in this field, trying to insert record number source table. After the migration, we can perform
5 of Table 1 into the target table would cause a key the same calculation from the Weight field of the
violation, and the record would not be migrated target table. If the results of these two calculations
correctly. do not match, errors have occurred during
Because of these potential errors, verifying the migration.
correctness of the data is an important component • For each field with “string” data type, the validation
of the data migration process. Next, we discuss a few test compares string length between the two
popular methods that are currently used to verify databases. For example, before migration, we can
whether data have been migrated correctly and some make note of the total string length of the Address
of the limitations of each. field of all records and compare it with the same
measure after the migration. If the numbers do not
Current Methods for Detecting Data match, migration errors have occurred.
Migration Errors • For each field with “date/time” data type, the
Once the migration is complete, most data migration validation converts the values into a numeric
systems provide only a simple report stating the representation and compares the databases using
number of records that have been migrated. Although the method applied on “numeric” fields.
this record count provides an important clue as to any
records that were rejected during the migration, it In any of these comparisons, if the values in the
does not provide any information on any problems or target database do not match those of the source
ensure that the data have been transferred correctly. database, errors must have occurred during the
Further verification of the data content is needed. migration process. While this method provides
important information on the successfulness of the
The first and most obvious way to verify data migration, it does not check the data at the record
correctness is by manual inspection and ad hoc level. If programmers detect errors, they still do not
queries. Programmers can examine the data visually know which particular records have been migrated
to verify that the content in the target database incorrectly. Therefore, they may have to repeat the
matches that of the source database. Programmers entire migration process. Even so, repeating the
can also run ad hoc queries to check the quality of process does not guarantee success because the
4 Wei and Chen, 2014 RTI Press

factors that caused error in the first migration may Checksum Methodology in Network
come into play during the second, or other factors Transmission Error Detection
may cause additional problems. Parity bit check and its extension called checksum
are popular methods of detecting network data
Reconciliation testing examines the data correctness at
transmission errors. There are variations on how
the record level.9-11 This testing method retrieves and
to implement a checksum. Bolton12 describes this
compares the content of each record in the source and
methodology as follows:
target databases, making it the most comprehensive
way to verify whether data were migrated correctly. The movement of digital data from one location
to another can result in transmission errors. For
Because this method performs correction verification
example, the transmitted sequence 1001 may be
at the record level, it will normally pinpoint the exact incorrectly received as 1101. In order to detect
records that are in error. This is an improvement over such errors a parity bit is often used. A parity bit
the validation testing method mentioned above, but is an extra 0 or 1 bit attached to a code group at
reconciliation testing has its own drawbacks: transmission. In the even parity method the value
of the bit is chosen so that the total number of 1s in
• Depending on the data volume, reconciliation the code group, including the parity bit, is an even
testing may be very time-consuming. number. For example, in transmitting 1001 the parity
• If the source database is located remotely, this test bit used would be 0 to give 01001, and thus an even
number of 1s. In transmitting 1101 the parity bit
may become difficult to perform and the testing
used would be 1 to give 11101, and thus an even
itself may be error-prone. number of 1s. With odd parity the parity bit is chosen
• If the primary key definitions are inconsistent so that the total number of 1s, including the parity
between the source and target tables, or if the bit, is odd. Thus if at the receiver the number of 1s
in a code group does not give the required parity,
source table does not have a primary key, matching
the receiver will know that there is an error and can
source and target records exactly would be request that the code group be retransmitted.
impossible.
An extension of the parity check is the checksum in
Because of the drawbacks of the methods discussed which a block of code may be checked by sending a
series of bits representing their binary sum.
so far, we have implemented a new data migration
correctness verification method using the checksum For example, for each binary packet sent, an 8-bit
methodology. This method verifies data content at checksum can be attached. The checksum bits can
the record level, so it is more comprehensive than hold 28 different values, so the checksum’s value
validation testing. It also does not require the system ranges from 0 to 255. If the value of the packet is less
to retrieve the source data again, so it avoids the than 255, the checksum is identical to the packet
weaknesses of reconciliation testing. Our method is value; otherwise, the checksum value is the remainder
novel in that it integrates the well-known checksum of the packet value divided by 256.
methodology used in network data transmission
error detection into data migration correctness In summary, checksum methodology basically
verification. Before we provide additional detail on derives extra content from the original information,
how to implement this method, we first briefly review attaches this content to the original information, and
the checksum methodology and how it is used in transmits the combined information to the receiving
network data transmissions. end. Once the information is received, the receiver
re-derives the checksum content and compares it
with the information received. If the checksum values
have discrepancies, errors have occurred. In the
next section, we discuss how to apply the checksum
methodology to verify whether data have migrated
correctly.
 Data Migration Correctness Verification 5

our example, we use the popular MD5 algorithm to

A New Data Migration Correctness generate the hash codes. To illustrate this process, we
Verification Method That Applies the used the Employees information from Table 1. Table 2
Checksum Methodology shows the source table to be migrated, with the new
We first discuss the implementation details of this field containing the checksum hash codes calculated
correctness verification method. Then we discuss its for each record.
strengths and weaknesses. To generate the checksum hash codes in the source
table, we retrieve the content of each record, convert
Implementation of the Method the value in every field into alpha character type,
The checksum methodology uses extra content concatenate these values into a single string, and
to verify the correctness of the original data. This then pass the string as a parameter (inputString) into
methodology is the basic mechanism of the data a programming code module (written in C# in this
migration correctness verification method discussed example) as follows:
in this paper. In this method, additional content is
attached to each record to verify the correctness of private string GetHashString(inputString)
the record after migration. Therefore, the first step is {
to add a new checksum field to each table that is to //calculate MD5 hash from input
be migrated from one database to another. This field System.Security.Cryptography.MD5 md5;
md5 = System.Security.Cryptography.MD5.
holds the extra content for each record that is going
Create();
to be used to verify the record. The second step is to string contentstr = inputString;
decide what content to put into the checksum field. if (contentstr == null) return null;
byte[] inputBytes = System.Text.Encoding.
A common and reliable method for creating content ASCII.GetBytes(contentstr);
for the checksum field that uses fixed-length and byte[] hash = md5.ComputeHash(inputBytes);
shortened output to reference the original data is
//convert byte array to hex string
hash function. A hash function can generate a code StringBuilder sb = new StringBuilder();
for each record, based on the content of the record, for (int i = 0; i < hash.Length; i++) {
for the checksum field of the source table. Once the sb.Append(hash[i].ToString(“x2”));
table arrives at the receiving end, programmers apply }
the same hash function algorithm that was used to return sb.ToString();
}
encode the source to generate a hash code for each
record received. For any records received, if the
calculated hash codes on the receiving end do not Implementing this hash code method is simple and
match the hash codes transferred with the record, straightforward. This same module is used on the
errors have occurred in these particular records receiving end to calculate the checksum hash codes
during migration. in the target table. This method is very effective in
detecting most data migration errors, as discussed
There are several mature and well-known hash next.
code algorithms, such as MD5 and SHA-1.13 For

Table 2. Employees source table to be migrated using the checksum verification method
ID Name Address … Checksum Hash Code
1 John 1321 IPhone Road … 652adc7cc073fe69b94a48b3704e9f12
2 David 1322 Kindle street … 745f6845ff3ebf33ab618aae567f3926
3 Matthew 1323 IPad Road … 3d88779f639bb5e2bc32441009e7bb00
4 Mark 1324 Galaxy Lane … a88798f5a9025645020026f11c35c93f
5 Luke 1325 Pink Drive … 131ac65ce64cacbd3168afed2a504a30
6 Skip 1326 Humuhumunukunukuapuaa Avenue … 17c126a5e3c06f20a8f36a3f1703778c
6 Wei and Chen, 2014 RTI Press

In the scenario in which the Height field (from

Discussion Table 1) is defined with 1-digit decimal precision in
We previously listed four main causes of potential the target table, Table 4 illustrates the comparison
errors: faulty hardware, network transmission noise, between the received checksum values and the
database format mismatch between source and target calculated hash codes based on the target table
databases, and database primary key and constraint content.
definition differences in source and target databases.
The checksum verification method proposed in this Other than the fourth record, which has value 6.10 in
paper can be used effectively in detecting the first the Height field of the source table (and is therefore
three types of errors, particularly the errors caused by not affected by the new format definition of the target
database format mismatch between source and target table), the calculated hash codes are different from
databases, the most common in data migration. We those that were received. Table 4 reveals that data
use this error type to illustrate the effectiveness of migration error has occurred in almost every record.
checksum verification. Based on the above analysis on detecting data
In Table 3, the scenario is that the target table defines migration errors, one can easily infer that this method
the Address field with a width of 20 characters rather would be useful in identifying errors caused by faulty
than 35. hardware as well as by network transmission noise.
This method checks the calculated hash code of
Because of this 20-character definition, the each record and therefore provides a comprehensive
Address information in record 6 is truncated examination of the data migration outcome.
due to the target table’s field format. When we In addition, this method performs correctness
calculate the hash code for each record in the verification solely on the target database and does
target table, the hash code of record 6 would be not require the repeated retrieval of the source data,
9448b731deacdf2942c85bc8d5a0af66, which is which makes it more feasible and easier to implement
different from the checksum hash code received. than validation testing or reconciliation testing.
Therefore, this comparison reveals that a data
migration error occurred and also pinpoints the exact While checksum verification is effective and can be
record that was migrated erroneously. applied to detect most of the data migration errors,

Table 3. Employees target table received with error

ID Name Address … Checksum Hash Code Received
1 John 1321 IPhone Road … 652adc7cc073fe69b94a48b3704e9f12
2 David 1322 Kindle street … 745f6845ff3ebf33ab618aae567f3926
3 Matthew 1323 IPad Road … 3d88779f639bb5e2bc32441009e7bb00
4 Mark 1324 Galaxy Lane … a88798f5a9025645020026f11c35c93f
5 Luke 1325 Pink Drive … 131ac65ce64cacbd3168afed2a504a30
6 Skip 1326 Humuhumunukunuk … 17c126a5e3c06f20a8f36a3f1703778c

Table 4. Comparison of received and calculated hash codes

ID Checksum Hash Code Received Hash Code Calculated After Receipt
1 652adc7cc073fe69b94a48b3704e9f12 dcc120f003d4c49f6f94bb0fb6810df3
2 745f6845ff3ebf33ab618aae567f3926 48e83caff5f7a3805c4abd68855e523e
3 3d88779f639bb5e2bc32441009e7bb00 58168cfa64cbbeee27c0b0a82d0b0a8d
4 a88798f5a9025645020026f11c35c93f a88798f5a9025645020026f11c35c93f
5 131ac65ce64cacbd3168afed2a504a30 ddc375d542d35eca0a4dc15e5c037cc9
6 17c126a5e3c06f20a8f36a3f1703778c 3f2ce06d09d7bf5835041c1eece0a0b3
 Data Migration Correctness Verification 7

programmers do need to be mindful of its limitations However, these potential problems should rarely
and drawbacks. First, checksum verification increases occur because our proposed method is based on
the data volume being transferred because of the the well-tested and commonly used checksum
extra checksum field in each table. However, with methodology. The limitations are merely the
improving technologies in data communications and inheritance of this methodology. Given the success
with storage becoming less expensive, this concern of the checksum methodology, and the multiple
becomes less of an issue almost daily. advantages of the checksum data migration
verification method as illustrated in our examples,
Second, if data migration errors occur in transmitting
we believe this method provides an easy and effective
the checksum values but not in transmitting the
way to verify the correctness of a data migration
original record content, the checksum method may
operation. Implementing the checksum correctness
actually cause false alarms.
verification method in transmitting data from one
Third, the checksum method can only detect errors database to another can greatly enhance the quality of
on the records that are received in the target table. a data migration outcome.
If records are rejected because of differences in
primary key and constraint definition between the
Future Work
source and target tables, the checksum method is not
effective. Fortunately, the error caused by records Much work has been done on not only detecting
being rejected is easy to detect. A simple record-count errors with the parity bit and checksum method in
comparison before and after the migration, which network data transmission, but correcting those
almost all data migration tools perform, is sufficient errors as well. Further research and exploration
to detect this sort of error, overcoming the method’s is needed to expand the checksum correctness
biggest shortcoming. verification method described here to detect and
correct errors more effectively and overcome the
Fourth, the checksum method described here can slight possibility that some errors pass undetected.
leave errors undetected, just like the parity bit and Further research in studying the error-missing rate
checksum methods in network data transmission can. will be helpful in establishing the tolerance level when
This is because converting the content of a record to applying this method.
a hash code is not a 1-to-1 relationship. Although
the possibility is extremely low, errors could alter the
record content while the resulting hash code remains
identical to that of the original record. In such a
scenario, the errors will escape the detection of the
checksum method.
8 Wei and Chen, 2014 RTI Press

References
1. Kimball R, Caserta J. The data warehouse ETL 8. Paygude P, Devale P. Automated data validation
toolkit: practical techniques for extracting, testing tool for data migration quality assurance.
cleaning, conforming, and delivering data. Int J Mod Eng Res (IJMER). 2013 Jan-Feb;3(1)
Indianapolis (IN): Wiley Publishing; 2004. 599-603.
2. Morris J. Practical data migration. 2nd ed. 9. Matthes F, Schulz C, Haller K. Testing &
Swindon (UK): BCS Learning & Development, quality assurance in data migration projects.
Ltd.; 2012. In proceeding of: IEEE 27th International
3. A roadmap to data migration success: Conference on Software Maintenance, ICSM
approaching the unique issues of data migration 2011; Williamsburg (VA), September 25-30,
[Internet] San Jose (CA): SAP Business Objects; 2011.
2008. Available from: http://fm.sap.com/data/ 10. Haller K. Towards the industrialization of data
upload/files/A_Road_Map_to_Data_Migration_ migration: concepts and patterns for standard
Success_2010.3.17-17.29.55.pdf software implementation projects. In: Eck P,
4. White C. Data communications and computer Gordijn J, Wieringa R, editors. 21st International
networks: a business user’s approach. Boston: Conference on Advanced Information Systems
Course Technology; 2011. Engineering (CAiSE) Proceedings; 2009;
Amsterdam, The Netherlands; June 8–12, 2009;
5. Wei B, Chen TX. Criteria for evaluating
pp. 63-78.
general database migration tools (RTI Press
publication No. OP-0009-1210). Research 11. Manjunath T, Hegadi R, Mohan H. Automated
Triangle Park (NC): RTI Press; 2012. Available data validation for data migration security. Int J
from: http://www.rti.org/publications/rtipress. Computer App. 2011 Sep;30(6):41-6.
cfm?pubid=20204 12. BoltonW. Mechatronics: electronic control
6. Katzoff D. How to implement an effective data systems in mechanical and electrical
migration testing strategy [Internet]. Data engineering. 3rd ed. Longman (NY): Prentice
Migration Pro; 2009. Available from: http:// Hall; 2004.
www.datamigrationpro.com/data-migration­ 13. Mironov I. Hash functions: theory, attacks, and
articles/2009/11/30/how-to-implement-an­ applications. Mountain View (CA): Microsoft
effective-data-migration-testing-strateg.html Research; 2005.
7. Singh I. How to do database migration testing
effectively and quickly? [Software testing blog
on the Internet]. Delhi: Inder P. Singh; 2010.
Available from: http://inderpsingh.blogspot.
com/2010/03/how-to-do-database-migration­
testing.html
Acknowledgments
We would like to thank Craig Hollingsworth and Joanne Studders of RTI
International for making substantial editorial contributions to this paper.
We would also like to thank our colleagues at NOAA and RTI’s Research
Computing Division for helping us shape and improve the idea of the method
discussed in this paper.
RTI International is an independent, nonprofit research organization dedicated
to improving the human condition by turning knowledge into practice. RTI
offers innovative research and technical solutions to governments and businesses
worldwide in the areas of health and pharmaceuticals, education and training,
surveys and statistics, advanced technology, international development,
economic and social policy, energy and the environment, and laboratory and
chemistry services.

The RTI Press complements traditional publication outlets by providing another

way for RTI researchers to disseminate the knowledge they generate. This PDF
document is offered as a public service of RTI International.
www.rti.org/rtipress RTI Press publication OP-0019-1403