Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamSets Open Source Community

1© StreamSets, Inc. All rights reserved.
Project Ouroboros
Using StreamSets Data Collector to Help Manage
the StreamSets Open Source Community
Pat Patterson / Director of Evangelism
@metadaddy / pat@streamsets.com

Who Am I?
Pat Patterson / pat@streamsets.com / @metadaddy
Past: Sun Microsystems, Salesforce
Present: Director of Evangelism, StreamSets
I run far 🏃♂️

Who is StreamSets?
Seasoned leadership team Customer base from global
8000
50%
Unique commercial
downloaders
2000+
Open source downloads
worldwide
3,000,000+
Broad connectivity
50+
History of innovation
streamsets.com/about-us

The StreamSets DataOps Platform
Data Lake

A Swiss Army Knife for Data

Parse Fastly CDN logs
Extract records relating to downloads
Gain insights
Companies downloading the binaries
Geographic reach
Metrics for different binary artifacts
Objective

Bash script to download S3 objects using AWS CLI tool
sed, grep, sort, uniq, awk, diff, xargs, curl
Complex, hard to maintain, slow, essentially ‘write-only’ code
cut -f 1 -d ' ' merge.log|sort|uniq > ips
diff --new-line-format="" --unchanged-line-
format="" ips allips > newips
cat newips|xargs -L 1 -I% curl -s
http://ipinfo.io/%/org|cut -f 2- -d '
'|sort|uniq>orgs && subl orgs
Before

Mission creep
Inertia
Why???
Image Nyah S / Pexels / Pexels License

Data Flow
StreamSets
Data Collector
↘
↘
Amazon S3
MySQL

Parse Fastly CDN log lines, send data to MySQL
<134>2017-07-09T12:01:13Z cache-sjc3636
StreamSetsS3Bucket[60550]: 104.155.191.102 "-" "-"
Sun, 09 Jul 2017 12:01:12 GMT GET
/datacollector/latest/parcel/manifest.json 200 1295
Let’s Get Started!

Grok Patterns are designed for exactly this!
Standard patterns for timestamps, HTTP verbs, filenames
<%{NUMBER:priority}>%{TIMESTAMP_ISO8601:timestamp}
%{HOSTNAME:cachenode}
%{WORD:logname}[%{NUMBER:pid}]: %{IP:ip} "-" "-"
%{DATESTAMP_FASTLY:datestamp} %{WORD:verb}
%{PATH:file} %{NUMBER:code} %{SIZE_OR_NULL}
Simple, Right?

First Cut

What??? An HTTP request isn’t supposed to include the protocol like that!
Fastly records whatever the client sends, no matter how dumb.
But...
Record1-Error SERVICE_ERROR_001 - Cannot parse record from message 'rawData':
com.streamsets.pipeline.api.service.dataformats.DataParserException:
LOG_PARSER_03 - Log line '<134>2017-07-09T12:01:13Z cache-sjc3636
StreamSetsS3Bucket[60550]: 104.155.191.102 "-" "- Sun, 09 Jul 2017 12:01:12 GMT
GET
https://archives.streamsets.com/datacollector/latest/parcel/STREAMSETS_DATAC
OLLECTOR-1.1.4-el6.parcel 404 0' does not conform to 'Grok Format

<%{NUMBER:priority}>%{TIMESTAMP_ISO8601:timestamp}
%{HOSTNAME:cachenode}
%{WORD:logname}[%{NUMBER:pid}]: %{IP:ip} "-" "-"
%{DATESTAMP_FASTLY:datestamp} %{WORD:verb}
%{NOTSPACE:file} %{NUMBER:code} %{SIZE_OR_NULL}
Solution: Be Permissive with your Input

Even if you think you know the data
schema - test with real data!
First Lesson Learned

Second Cut

But
Performance SUCKED!

Solution: Duplicate the Data
CREATE TABLE download (
id int(11) AUTO_INCREMENT,
ip varchar(64),
date datetime,
file varchar(767),
PRIMARY KEY (`id`),
KEY `date_idx` (`date`),
KEY `file_idx` (`file`)
);

Third Cut

30x Better Performance!

Filtering Downloads

Fit the data model to the data
Second Lesson Learned

Lookup company details from IP via Kickfire API
What’s Next?

Fourth Cut

com.streamsets.pipeline.api.base.OnRecordErrorException: HTTP_01 -
Error fetching resource. Status: 429 Reason: You have reached the
maximum calls per second
org.glassfish.jersey.message.internal.EntityInputStream@4cb3922b
But...
Kickfire API is rate limited!
To deliver optimum performance to all of our API customers, KickFire
balances transaction loads by using rate limits

Solution - Rate Limit

com.streamsets.pipeline.api.base.OnRecordErrorException: HTTP_01 - Error
fetching resource. Status: 429 Reason: You have reached the maximum calls
per month org.glassfish.jersey.message.internal.EntityInputStream@4cb3922b
But...
Kickfire API has a monthly call limit!

Solution - Don’t Ask For Data We Already Have

Know your API’s
non-functional constraints!
Third Lesson Learned

Fifth Cut

com.streamsets.pipeline.api.base.OnRecordErrorException: HTTP_01 -
Error fetching resource. Status: 429 Reason: You have reached the
maximum calls per month
org.glassfish.jersey.message.internal.EntityInputStream@4cb3922b
But...
Kickfire’s monthly call limit strikes again!

Root Cause
Seeing large numbers of downloads from the same few IP addresses
Data Collector has a microbatch architecture - database writes are
committed at the end of the batch
New IP address isn’t visible in the database until the start of the next batch
Still making repeated requests to Kickfire for the same IP address!

Solution - Deduplicate records on IP Address

Data Collector operates batch-by-batch
-
design your pipelines accordingly!
Fourth Lesson Learned

The Finished Article

A Closer Look

No plan survives first
contact with the enemy
Helmuth von Moltke the Elder, "On Strategy"
(1871)
Ultimate Lesson Learned
Image in the public domain

or

September 3-5, 2019
Tue, Sep 3 - Training & Tutorials
Wed-Thu, Sep 4-5, Keynote & Breakouts
Hilton Financial District
(Tue|Wed|Thur)

Questions?

Thank you
Pat Patterson / Director of Evangelism
@metadaddy / pat@streamsets.com

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamSets Open Source Community

More Related Content

What's hot (20)

Similar to Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamSets Open Source Community (20)

More from Pat Patterson (20)

Recently uploaded (20)

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamSets Open Source Community