Real-Time Fraud Detection with Storm and Kafka
0 likes1,177 views
This document discusses an advertising technology company's approach to preventing ad fraud. It describes how the company processes billions of ad impressions daily across multiple data centers. Key aspects of its fraud detection system include identifying bot activity, ensuring quality control of ad placements, and using machine learning models to analyze terabytes of daily log data to detect fraudulent patterns in real time.
Ad
More Related Content
Similar to Real-Time Fraud Detection with Storm and Kafka (20)
Recently uploaded (20)
Ad
Real-Time Fraud Detection with Storm and Kafka
- 2. 1 • Ad Fraud – We eliminate fraudulent impressions by adbots and make sure ads don’t show up on fraudulent web sites • Brand Safety – We make sure ads don’t show up in places that brands don’t want them • Viewability – We measure whether a person actually viewed an ad WE MEASURE AND ENSURE QUALITY
- 3. 2 • Ad impressions processed: 5+ Billion/day • HTTP Requests: 50+ Billion/day • Data Centers: 10+ (AWS and on-premises) • Data stored in clusters: 6+ petabytes • New data collected daily: 20+ terabytes • Hadoop cluster processing cores ~ 20,000 INTEGRAL ENGINEERING BY THE NUMBERS
- 5. H20 World, 11/10/2015 4 FRAUD DETECTION facebook cnn ebay nothingtoseehere.com thisisnotabotnet.com
- 6. H20 World, 11/10/2015 5 FRAUD DETECTION facebook cnn ebay nothingtoseehere.com thisisnotabotnet.com
- 7. 6 REQUIREMENTS • Quickly idenVfy freshly acVvated bot • High accuracy of detecVon algorithms • Avoid transfer of personal informaVon across borders • Withstand single data center failure
- 9. 8 EVENT SESSIONIZATION TimeTransaction 1 Transaction 2 Join Window Impression 1 Impression 2 UnloadDTDTDTInit DTDTDTInit Timeout Emit Emit Impression 3 DTDTDT Timeout Drop
- 11. 10 • Local log aggregaVon and processing • Transfer over long links causes all sorts of synchronizaVon problems • Intra-DC links are reliable, Internet is NOT. We can keep data locality and log Vme coherence • Single firewall server failure is not “stop-the-world” event. Data present on Kaaa cluster. • A completely autonomous system • Higher availability due DC redundancy INTRA-DC DATA PROCESSING
- 13. LOG SOURCING: TAILER AGENT ● Non-invasive event sourcing ● Decoupled data publica[on and event processing ● Data fan-out ● Hard latency requirements ● <10ms response ● Periodic checkpoints to recover acer failure
- 14. RECOVERY STRATEGY • Read logs in micro-batches and maintain state in memory • Reliable Processing - On success operaVon - write checkpoint - On failure return to previous checkpoint - On catastrophic failure rewind data feed to a point before the problem started
- 15. LOGICAL TIME ● Wall-clock does not work ● Load spikes ● Recovery rewinds data feed to previous Vme ● Logical clock ● Maximum Vmestamp seen by Bolt ● New messages with smaller Vmestamp are late ● No clock synchronizaVon ● All bolts are in “weak synchrony”
- 16. DEBUGGING AND MONITORING • Metrics recording and visualizaVon is essenVal component of development cycle - Ease failure symptoms correlaVon - Accelerate build/deploy/debug cycle - Provide trace for producVon issues • Monitor business metrics - This is the only thing you care - Technical issues may or may not have consequences • Do it a lot - 150K metrics/sec 15
- 17. GLOBAL CONFIGURATION 16 EAST COAST EUROPEDC-X Stream Mirror Stream Mirror Stream Mirror Kafka Backbone Spark CENTRAL Hadoop
- 18. LESSONS LEARNED • Use staged roll-out - Start from minimal infrastructure for logs delivery • Do not try to build a fortress - It is much easier to build a systems accepVng limited data loss • Minimize persistent state - Slows system down - Expensive to maintain • Hardware magers 17