Frontera распределенный робот для обхода веба в больших объемах / Александр Сибиряков (Scrapinghub)

Frontera: open
source, large scale
web crawling
framework
Alexander Sibiryakov,
Scrapinghub Ltd.
sibiryakov@scrapinghub.com

Здравствуйте участники!

• Software Engineer @
Scrapinghub

Scrapinghub
• Born in Yekaterinburg,
RU

Scrapinghub
RU
• 5 years at Yandex:  
social & QA search,
snippets.

Scrapinghub
RU
• 5 years at Yandex:  
social & QA search,
snippets.
• 2 years at Avast!
antivirus: false
positives, malicious
downloads

We help turn web content into
useful data {
"content": [
{
"title": {
"text": "'Extreme poverty' to fall below 10% of
world population for ﬁrst time",
"href": "http://www.theguardian.com/society/
2015/oct/05/world-bank-extreme-poverty-to-fall-
below-10-of-world-population-for-ﬁrst-time"
},
"points": "9 points",
"time_ago": {
"text": "2 hours ago",
"href": "https://news.ycombinator.com/item?
id=10352189"
},
"username": {
"text": "hliyan",
"href": "https://news.ycombinator.com/user?
id=hliyan"
}
},

useful data
• Over 2 billion requests per month (~800/sec.)
{
"content": [
{
"title": {
},
"time_ago": {
id=10352189"
},
"username": {
"text": "hliyan",
id=hliyan"
}
},

useful data
• Over 2 billion requests per month (~800/sec.)
• Focused crawls & Broad crawls
{
"content": [
{
"title": {
},
"time_ago": {
id=10352189"
},
"username": {
"text": "hliyan",
id=hliyan"
}
},

Broad crawl usages
• News analysis

Broad crawl usages
• News analysis
• Topical crawling

Broad crawl usages
• News analysis
• Plagiarism detection

Broad crawl usages
• News analysis
• Sentiment analysis (popularity, likability)

Broad crawl usages
• News analysis
• Due diligence (profile/business data)

Broad crawl usages
• News analysis
• Lead generation (extracting contact
information)

Broad crawl usages
• News analysis
• Lead generation (extracting contact
information)
• Track criminal activity & find lost persons
(DARPA)

Saatchi Global Gallery Guide
• www.globalgalleryguide.com

• Discover 11K online
galleries.

galleries.
• Extract general
information, art
samples,
descriptions.

galleries.
• Extract general
information, art
samples,
descriptions.
• NLP-based
extraction.

galleries.
• Extract general
information, art
samples,
descriptions.
• NLP-based
extraction.
• Find more galleries
on the web.

Task
• Spanish web: hosts and their sizes statistics.

Task
• Only .es ccTLD.

Task
• Only .es ccTLD.
• Breadth-ﬁrst strategy:

Task
• Only .es ccTLD.
• ﬁrst 1-click environ,

Task
• Only .es ccTLD.
• 2,

Task
• Only .es ccTLD.
• 2,
• 3,

Task
• Only .es ccTLD.
• 2,
• 3,
• …

Task
• Only .es ccTLD.
• 2,
• 3,
• …
• Finishing condition: 100 docs from host max., all
hosts

Task
• Only .es ccTLD.
• 2,
• 3,
• …
• Finishing condition: 100 docs from host max., all
hosts
• Low costs.

Spanish, Russian and world
Web, 2012
Sources: OECD Communications Outlook 2013, statdom.ru
* - current period (October 2015)
Domains
Web
servers
Hosts DMOZ*
Spanish (.es) 1,5M 280K 4,2M 122K
Russian
(.ru, .рф, .su) 4,8M 2,6M ? 105K
World 233M 62M 890M 1,7

Solution
• Scrapy (based on Twisted) - async
network operations.

Solution
network operations.
• Apache Kafka - data bus (offsets,
partitioning).

Solution
network operations.
partitioning).
• Apache HBase - storage (random access,
linear scanning, scalability).

Solution
network operations.
partitioning).
• Apache HBase - storage (random access,
linear scanning, scalability).
• Snappy - efficient compression algorithm
for IO-bounded applications.

Architecture
Kafka topic
SW
Crawling strategy
workers
Storage workersDB

1. Big and small hosts problem

• Queue is flooded
with URLs from the
same host.

with URLs from the
same host.
• → underuse of
spider resources.

with URLs from the
same host.
• → underuse of
spider resources.
• additional per-host
(per-IP) queue and
metering algorithm.

with URLs from the
same host.
• → underuse of
spider resources.
• additional per-host
(per-IP) queue and
metering algorithm.
• URLs from big hosts
are cached in
memory.

2. DDoS DNS service Amazon AWS

Breadth-first strategy →

first visiting of
unknown hosts →

first visiting of
unknown hosts →
generating huge
amount of DNS reqs.

first visiting of
unknown hosts →
generating huge
amount of DNS reqs.
Recursive DNS server

first visiting of
unknown hosts →
generating huge
amount of DNS reqs.
• on every spider node,

first visiting of
unknown hosts →
generating huge
amount of DNS reqs.
• upstream to Verizon &
OpenDNS.

first visiting of
unknown hosts →
generating huge
amount of DNS reqs.
• upstream to Verizon &
OpenDNS.
We used dnsmasq.

3. Tuning Scrapy thread pool
for efficient DNS resolution

• OS DNS resolver,

• blocking calls,

• blocking calls,
• thread pool to
resolve DNS name to
IP.

• blocking calls,
• thread pool to
resolve DNS name to
IP.
• numerous errors
and timeouts 🆘

• blocking calls,
• thread pool to
resolve DNS name to
IP.
• numerous errors
and timeouts 🆘
• A patch for thread
pool size and
timeout adjustment.

4. Overloaded HBase region
servers during state check
3Tb of metadata.
URLs, timestamps,…
275 b/doc

• 10^3 links per doc,
3Tb of metadata.
275 b/doc

• state check: CRAWLED/NOT
CRAWLED/ERROR,
3Tb of metadata.
275 b/doc

CRAWLED/ERROR,
• HDDs.
3Tb of metadata.
275 b/doc

CRAWLED/ERROR,
• HDDs.
• Small volume 🆗
3Tb of metadata.
275 b/doc

CRAWLED/ERROR,
• HDDs.
• With ⬆table size, response
times ⬆🆘
3Tb of metadata.
275 b/doc

CRAWLED/ERROR,
• HDDs.
times ⬆🆘
• Disk queue ⬆
3Tb of metadata.
275 b/doc

CRAWLED/ERROR,
• HDDs.
times ⬆🆘
• Disk queue ⬆
• Host-local fingerprint
function for keys in HBase.
3Tb of metadata.
275 b/doc

CRAWLED/ERROR,
• HDDs.
times ⬆🆘
• Disk queue ⬆
• Host-local fingerprint
function for keys in HBase.
• Tuning HBase block cache
to fit average host states
into one block.
3Tb of metadata.
275 b/doc

5. Intensive network traffic
from workers to services

• Throughput
between workers
and Kafka/HBase  
~ 1Gbit/s.

• Throughput
between workers
and Kafka/HBase  
~ 1Gbit/s.
• Thrift compact
protocol for
HBase

• Throughput
between workers
and Kafka/HBase  
~ 1Gbit/s.
• Thrift compact
protocol for
HBase
• Message
compression in
Kafka with Snappy

6. Further query and traffic
optimizations to HBase

• State check: lots of
reqs and network

reqs and network
• Consistency

reqs and network
• Consistency
• Local state cache
in strategy worker.

reqs and network
• Consistency
• Local state cache
in strategy worker.
• For consistency,
spider log was
partitioned by
host.

State cache
• All ops are batched:

State cache
– If no key in cache→
read HBase

State cache
read HBase
– every ~4K docs →
flush

State cache
read HBase
flush
• Close to 3M (~1Gb)
elms → flush & cleanup

State cache
read HBase
flush
• Close to 3M (~1Gb)
elms → flush & cleanup
• Least-Recently-Used
(LRU) 👍

Spider priority queue (slot)
• Cell:

• Cell:
Array of: 
- ﬁngerprint,  
- Crc32(hostname),  
- URL,  
- score

• Cell:
Array of: 
- ﬁngerprint,  
- URL,  
- score
• Dequeueing top N.

• Cell:
Array of: 
- ﬁngerprint,  
- URL,  
- score
• Prone to huge hosts

• Cell:
Array of: 
- ﬁngerprint,  
- URL,  
- score
• Prone to huge hosts
• Scoring model: document
count per host.

7. Problem of big and small
hosts (strikes back!)

• Discovered few very
huge hosts (>20M
docs)

huge hosts (>20M
docs)
• All queue partitions
were flooded with
huge hosts,

huge hosts (>20M
docs)
were flooded with
huge hosts,
• Two MapReduce
jobs:

huge hosts (>20M
docs)
were flooded with
huge hosts,
• Two MapReduce
jobs:
– queue shuffling,

huge hosts (>20M
docs)
were flooded with
huge hosts,
• Two MapReduce
jobs:
– queue shuffling,
– limit all hosts to
100 docs MAX.

Hardware requirements
• Single-thread Scrapy spider →  
1200 pages/min. from ~100 websites in parallel.

• Spiders to workers ratio is 4:1 (without content)

• 1 Gb of RAM for every SW (state cache, tunable).

• Example:

• Example:
– 12 spiders ~ 14.4K pages/min.,

• Example:
– 3 SW and 3 DB workers,

• Example:
– 3 SW and 3 DB workers,
– Total 18 cores.

Software requirements
CDH (100% Open source
Hadoop package)

• Apache HBase,
Hadoop package)

• Apache HBase,
• Apache Kafka,
Hadoop package)

• Apache HBase,
• Apache Kafka,
• Python 2.7+,
Hadoop package)

• Apache HBase,
• Apache Kafka,
• Python 2.7+,
• Scrapy 0.24+,
Hadoop package)

• Apache HBase,
• Apache Kafka,
• Python 2.7+,
• Scrapy 0.24+,
• DNS Service.
Hadoop package)

Maintaining Cloudera Hadoop
on Amazon EC2

on Amazon EC2
• CDH is very sensitive to free space on root
partition, parcels, and storage of Cloudera
Manager.

on Amazon EC2
Manager.
• We’ve moved it using symbolic links to separate
EBS partition.

on Amazon EC2
Manager.
EBS partition.
• EBS should be at least 30Gb, base IOPS should be
enough.

on Amazon EC2
Manager.
EBS partition.
enough.
• Initial hardware was 3 x m3.xlarge (4 CPU, 15Gb,
2x40 SSD).

on Amazon EC2
Manager.
EBS partition.
enough.
• Initial hardware was 3 x m3.xlarge (4 CPU, 15Gb,
2x40 SSD).
• After one week of crawling, we ran out of space,
and started to move DataNodes to d2.xlarge (4
CPU, 30.5Gb, 3x2Tb HDD).

Spanish (.es) internet crawl
results

results
• fnac.es, rakuten.es, adidas.es,
equiposdefutbol2014.es, druni.es,
docentesconeducacion.es - are the
biggest websites

results
biggest websites
• 68.7K domains found (~600K
expected),

results
biggest websites
expected),
• 46.5M crawled pages overall,

results
biggest websites
expected),
• 1.5 months,

results
biggest websites
expected),
• 1.5 months,
• 22 websites with more than 50M pages

where are the rest of
web servers?!

Bow-tie model
A. Broder et al. / Computer Networks 33 (2000) 309-320

Y. Hirate, S. Kato, and H. Yamana, Web Structure in 2005

12 years dynamics
Graph Structure in the Web — Revisited, Meusel, Vigna, WWW 2014

Main features
• Online operation: scheduling of new batch,
updating of DB state.

Main features
• Storage abstraction: write your own backend
(sqlalchemy, HBase is included).

Main features
• Canonical URLs resolution abstraction: each
document has many URLs, which to use?

Main features
• Canonical URLs resolution abstraction: each
document has many URLs, which to use?
• Scrapy ecosystem: good documentation, big
community, ease of customization.

Distributed Frontera features
• Communication layer is Apache Kafka: topic
partitioning, offsets mechanism.

• Crawling strategy abstraction: crawling
goal, url ordering, scoring model is coded in
separate module.

separate module.
• Polite by design: each website is
downloaded by at most one spider.

separate module.
• Polite by design: each website is
downloaded by at most one spider.
• Python: workers, spiders.

References
• Frontera. https://github.com/scrapinghub/frontera
• Distributed extension. https://github.com/
scrapinghub/distributed-frontera
• Documentation:
– http://frontera.readthedocs.org/
– http://distributed-frontera.readthedocs.org/
• Google groups: Frontera (https://goo.gl/ak9546)

Future plans
• Lighter version, without HBase and Kafka.
Communicating using sockets.

Future plans
• Revisiting strategy out-of-box.

Future plans
• Watchdog solution: tracking website content
changes.

Future plans
changes.
• PageRank or HITS strategy.

Future plans
changes.
• Own HTML and URL parsers.

Future plans
changes.
• Integration into Scrapinghub services.

Future plans
changes.
• Integration into Scrapinghub services.
• Testing on larger volumes.

Run your business using Frontera

SCALABLE

SCALABLE
OPEN

SCALABLE
OPEN
CUSTOMIZABLE

Made in Scrapinghub
(authors of Scrapy)
SCALABLE
OPEN
CUSTOMIZABLE

Здесь может быть ВАШ код!

• Web scale crawler,

• Historically first
attempt in Python,

• Historically first
attempt in Python,
• Truly resource-
intensive task:
CPU, network,
disks.

We’re hiring!
http://scrapinghub.com/jobs/

Спасибо!
Alexander Sibiryakov,
sibiryakov@scrapinghub.com

Frontera распределенный робот для обхода веба в больших объемах / Александр Сибиряков (Scrapinghub)

Recommended

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Frontera распределенный робот для обхода веба в больших объемах / Александр Сибиряков (Scrapinghub) (20)

More from Ontico (20)

Recently uploaded (20)

Frontera распределенный робот для обхода веба в больших объемах / Александр Сибиряков (Scrapinghub)