Scalable and Reliable Logging at Pinterest
6 likes2,396 views
Pinterest uses Kafka as the central logging system to collect over 120 billion messages per day from thousands of hosts. They developed Singer, a lightweight logging agent, to reliably upload application logs to Kafka with low latency. Data is then moved from Kafka to cloud storage using systems like Secor and Merced that ensure exactly-once processing. Maintaining high log quality requires monitoring for anomalies, auditing new features, and catching issues both before and after releases through automated tooling.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Scalable and Reliable Logging at Pinterest (20)
Ad
Recently uploaded (20)
Scalable and Reliable Logging at Pinterest
- 1. Scalable and Reliable Logging at Pinterest Krishna Gade [email protected] Yu Yang [email protected]
- 2. Agenda • What is Pinterest? • Logging Infrastructure Deep-dive • Managing Log Quality • Summary & Questions
- 4. What is Pinterest? Pinterest is a discovery engine
- 5. What is the weather in SF today?
- 7. What is central limit theorem?
- 10. What do I cook for dinner today?
- 11. What’s my style?
- 12. Where shall we travel this summer?
- 13. Pinterest is solving this discovery problem
- 20. Kafka App Data Architecture Singer Qubole (Hadoop, Spark) Merced Pinball Skyline Redshift Pinalytics Product Storm Stingray A/B Testing
- 22. Logging Infrastructure Requirements • High availability • Minimum data loss • Horizontal scalability • Low latency • Minimum operation overhead
- 23. Pinterest Logging Infrastructure • thousands of hosts • >120 billion messages, tens of terabytes per day • Kafka as the central message transportation hub • >500 Kafka brokers • home-grown technologies for logging to Kafka and moving data from Kafka to cloud storage AppServers events Kafka Cloud storage
- 24. Logging Infrastructure v1 events Kafka 0.7 Host app app app data uploader Real-time consumers
- 25. Problems with Kafka 0.7 pipelines • Data loss • Kafka 0.7 broker failure —> data loss • high back pressure —> data loss • Operability • broker replacement —> reboot all dependent services to pick up the latest broker list
- 26. Challenges with Kafka that supports replication • Multiple copies of messages among brokers • cannot copy message directly to S3 to guarantee exact once persistence • Cannot randomly pick Kafka brokers to write to • Need to find the leader of each topic partition • Handle various corner cases
- 27. Logging Infrastructure v2 events Kafka 0.8 Host app log files Singer Secor/Merced Sanitizer Real-time consumers
- 28. Logging Agent Requirement • reliability • high throughput, low latency • minimum computation resource usage • support various log file format (text, thrift, etc.) • fairness scheduling
- 29. Singer Logging Agent • Simple logging mechanism • applications log to disk • Singer monitors file system events and uploads logs to Kafka • Isolate applications from Singer agent failures • Isolate applications from Kafka failures • >100MB/second for log files in thrift • Production Environment Support • dynamic configuration detection • adjustable log uploading latency • auditing • heartbeat mechanism Host app log files Singer
- 30. Singer Internals Singer Architecture LogStream monitor Configuration watcher Reader Writer Log repository Reader Writer Reader Writer Reader Writer Log configuration LogStream processors A - 1 A -2 B - 1 C - 1 Log configuration Staged Event Driven Architecture
- 31. Running Kafka in the Cloud • Challenges • brokers can die unexpectedly • EBS I/O performance can degrade significantly due to resource contention • Avoid virtual hosts co-location on the same physical host • faster recovery
- 32. Running Kafka in the Cloud • Initial settings • c3.2xlarge + EBS • Current settings • d2.xlarge • local disks help to avoid EBS contention problem • minimize data on each broker for faster recovery • availability zone aware topic partition allocation • multiple small clusters (20-40 brokers) for topic isolation
- 33. Scalable Data Persistence 33 • Strong consistency: each message is saved exactly once • Fault tolerance: any worker node is allowed to crash • Load distribution • Horizontal scalability • Configurable upload policies events Kafka 0.8 Secor/Merced
- 34. Secor 34 • Uses Kafka high level consumer • Strong consistency: each message is saved exactly once • Fault tolerance: any worker node is allowed to crash • Load distribution • Configurable upload policies events Kafka 0.8 Secor
- 35. Challenges with consensus-based workload distribution • Kafka consumer group rebalancing can prevent consumer from making progress • It is difficult to recover when high-level consumer lags behind on some topic partitions • Manual tuning is required for workload distribution of multiple topics • Inconvenient to add new topics • Efficiency
- 36. Merced • central workload distribution • master creates tasks • master and workers communicate through zookeeper
- 37. Merced
- 38. Log Quality
- 39. Log Quality Log quality can be broken down into two areas: • log reliability - Reliability is fairly easy to measure: did we lose any data? • log correctness - Correctness, on the other hand, is much more difficult as it requires the interpretation of data.
- 40. Challenges • Instrumentation is an after-thought for most feature developers • Features can get shipped breaking existing logging or no logging • Once an iOS or Android release is out, it will keep generating bad data for weeks • Data quality bugs are harder to find and fix compared to code quality
- 41. Tooling
- 42. Anomaly Detection • Started with a simple model based on the assumption that daily changes are normally distributed. • Revised that model until it has only a few alerts, mostly real and important. • Hooked it up to a daily email to our metrics avengers.
- 43. How did we do? • Interest follows went up after we started emailing recommended interests to follow • Push notifications about board follows broke • Signups from Google changed as we ran experiments • Our tracking broke when we released a new repin experience • Our tracking of mobile web signups changed
- 44. Auditing Logs Manual audits will have their limitations, especially with regards to coverage but will catch critical bugs. • However, we need two things: • Repeatable process that can scale • Tooling required to support the process • Regression Audit • Maintain a playbook of "core logging actions" • Use tooling to verify the output of the actions • New Feature Audit • Gather requirements for analysis and produce a list of events that need to be captured with the feature • Instrument the application • Test the logging output using existing tooling
- 45. Summary • Invest in your logging infra pretty early on. • Kafka has matured a lot and with some tuning works well in the Cloud. • Data quality is not free, need to proactively ensure it. • Invest in automated tools to detect quality issues both pre- and post-release. • Culture building and education go a long way.
- 46. Thank you! Btw, we’re hiring :)
- 47. Questions?