Scribd
Upload
53 views

FastWARC Optimizing Large-Scale Web Archive Analyt

FastWARC is a new Python library for processing Web ARChive (WARC) files that provides significant performance improvements over the existing WARCIO library. It achieves speedups from 1.6x to 8x through optimizations to stream decompression speeds, record parsing, and skipping non-response records. The largest gains come from supporting the more efficient LZ4 compression algorithm, which can save hundreds of hours of processing time on large web archives like Common Crawl. FastWARC is open source and available to download, providing a high-performance alternative for large-scale web archive analytics.

Uploaded by

Manoj Kumar Maurya
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
53 views

FastWARC Optimizing Large-Scale Web Archive Analyt

FastWARC is a new Python library for processing Web ARChive (WARC) files that provides significant performance improvements over the existing WARCIO library. It achieves speedups from 1.6x to 8x through optimizations to stream decompression speeds, record parsing, and skipping non-response records. The largest gains come from supporting the more efficient LZ4 compression algorithm, which can save hundreds of hours of processing time on large web archives like Common Crawl. FastWARC is open source and available to download, providing a high-performance alternative for large-scale web archive analytics.

Uploaded by

Manoj Kumar Maurya
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

FASTWARC: OPTIMIZING LARGE-SCALE WEB ARCHIVE ANALYTICS

Janek Bevendorff∗ , Martin Potthast† , Benno Stein∗


∗ Bauhaus-UniversitätWeimar, † Leipzig University
Abstract Comp. Parser Records/s Speedup
Web search and other large-scale web data analytics rely AMD Ryzen Threadripper 2920X (NVMe SSD)
on processing archives of web pages stored in a standard- None WARCIO 16 945.5 –
ized and efficient format. Since its introduction in 2008, None FastWARC 108 488.0 6.4
the IIPC’s Web ARCive (WARC) format1 has become the None WARCIO+HTTP 11 661.6 –
None FastWARC+HTTP 79 297.0 6.8
standard format for this purpose. As a list of individually None WARCIO+HTTP+Checksum 6 986.7 –
compressed records of HTTP requests and responses, it al- None FastWARC+HTTP+Checksum 21 320.9 3.1
lows for constant-time random access to all kinds of web GZip WARCIO 6 460.1 –
data via off-the-shelf open source parsers in many program- GZip FastWARC 10 413.4 1.6
arXiv:2112.03103v1 [cs.IR] 22 Nov 2021

ming languages, such as WARCIO,2 the de-facto standard GZip WARCIO+HTTP 5 435.6 –
for Python. When processing web archives at the terabyte or GZip FastWARC+HTTP 10 101.5 1.9
GZip WARCIO+HTTP+Checksum 4 121.6 –
petabyte scale, however, even small inefficiencies in these GZip FastWARC+HTTP+Checksum 7 433.0 1.8
tools add up quickly, resulting in hours, days, or even weeks
LZ4 FastWARC 49 825.4 7.7∗
of wasted compute time. Reviewing the basic components LZ4 FastWARC+HTTP 42 394.5 7.8∗
of WARCIO and analyzing its bottlenecks, we proceed to LZ4 FastWARC+HTTP+Checksum 16 992.2 4.1∗
build FastWARC, a new high-performance WARC process- Intel(R) Xeon(R) CPU E5-2620 v2 (remote Ceph storage)
ing library for Python, written in C++ / Cython, which yields
None WARCIO 7 969.1 –
performance improvements by factors of 1.6–8x. None FastWARC 49 396.5 6.2

INTRODUCTION GZip WARCIO 3 555.7 –


GZip FastWARC 6 335.1 1.8
The earliest open source implementations of the WARC LZ4 FastWARC 28 313.8 8.0∗
format were provided for Java, namely Lin’s ClueWeb Tools3
(initially used by the research search engine ChatNoir),4 fol- Table 1: Evaluation of FastWARC and WARCIO on two
lowed by a more standards-compliant reference implementa- systems. Runs are (1) without payload parsing, (2) with
tion from the IIPC.5 Meanwhile, the IR, NLP, and machine automatic HTTP header parsing, and (3) with record check-
learning communities have largely transitioned to Python, in- summing. ∗ LZ4 speedup is over WARCIO with GZip, since
stead adopting WARCIO as a native implementation in that WARCIO does not support LZ4.
language. Processing large samples of the Common Crawl
and web archive data from the Internet Archive, however, better decompression speed of gzipped streams, FastWARC
we observed that the library did not match our performance interfaces directly with zlib, achieving compute time sav-
expectations. Even compiling it to native C code using ings of roughly 2.1 hours per TiB or 2 200 hours per PiB
Cython yielded only marginal improvements. Analyzing its over WARCIO. The largest performance penalty, however,
bottlenecks, three key causes can be discerned: (1) stream comes from the decompressor itself. While still saving about
decompression speed, (2) record parsing performance, and 135 hours overall on a Common Crawl, the relative speedup
(3) lack of efficient skipping of non-response records. Our shrinks to only 1.6–1.8x. For this reason, we decided to add
contribution is to rectify these issues with FastWARC, a support for the more recent and much faster LZ4 algorithm.
rewrite of the entire WARC parsing pipeline from scratch. With LZ4, we can save another 168 hours on top (a speedup
of 4.8x over FastWARC with GZip), or 300 hours compared
FASTWARC VS. WARCIO to WARCIO (speedup of up to 8x).
FastWARC is a reimplementation of WARCIO in C++ CONCLUSION
with Cython, making it both fast and perfectly integrated
into the Python ecosystem, yet allowing for more language FastWARC can speed up WARC processing significantly,
bindings if required. Table 1 compiles detailed performance saving hundreds of hours of compute time on large-scale
comparisons: On an uncompressed WARC file, it gains an web archive analytics. By far the largest speedup, though,
overall 6.4x speedup over WARCIO, or 4x over a naively can be gained from using LZ4 over GZip. Considering an
“cythonized” WARCIO. With an average processing time of additional storage overhead of only about 30–40 %, recom-
1.2 vs. 8 seconds for a single WARC file, this already saves at pressing GZip WARCs with LZ4 is certainly an option to
least 115 hours of compute time on a recent Common Crawl be considered, especially in cases where processing speed
with 64 000 individual WARCs (62.5 TiB compressed). For is more important than storage efficiency.
FastWARC is released under the Apache 2.0 license and
1 ISO 28500:2017; https://iipc.github.io/warc-specifications/ can be downloaded from Github6 or PyPi.7
2 https://github.com/webrecorder/warcio
3 https://github.com/lintool/clueweb
4 6 https://github.com/chatnoir-eu/chatnoir-resiliparse
https://chatnoir.eu/
5 7 pip install fastwarc
https://github.com/iipc/jwarc

You might also like