NickKallen_DataArchitectureAtTwitterScale
5 likes804 views
This document discusses how Twitter handles big data in real-time. It describes three key data problems: tweets, timelines, and social graphs. For tweets, the data was originally stored relationally but now uses partitioning by time and user ID to improve performance. Timelines were initially slow to retrieve but are now pre-computed and stored in memory. The social graph was challenging due to its unbounded nature; it is now partitioned and stored with edges in both directions. Principles for handling real-time big data at scale include partitioning, indexing, replication, and keeping working data in memory.
Ad
More Related Content
Viewers also liked (20)
Similar to NickKallen_DataArchitectureAtTwitterScale (20)
Ad
More from Kostas Mavridis (9)
Ad
NickKallen_DataArchitectureAtTwitterScale
- 1. Big Data in Real-Time at Twitter by Nick Kallen (@nk)
- 2. What is Real-Time Data? • On-line queries for a single web request • Off-line computations with very low latency • Latency and throughput are equally important • Not talking about Hadoop and other high-latency, Big Data tools
- 3. The three data problems • Tweets • Timelines • Social graphs
- 5. What is a Tweet? • 140 character message, plus some metadata • Query patterns: • by id • by author • (also @replies, but not discussed here) • Row Storage
- 6. Find by primary key: 4376167936
- 7. Find all by user_id: 749863
- 8. Original Implementation id user_id text created_at 20 12 just setting up my twttr 2006-03-21 20:50:14 29 12 inviting coworkers 2006-03-21 21:02:56 34 16 Oh shit, I just twittered a little. 2006-03-21 21:08:09 • Relational • Single table, vertically scaled • Master-Slave replication and Memcached for read throughput.
- 9. Original Implementation Master-Slave Replication Memcached for reads
- 10. Problems w/ solution • Disk space: did not want to support disk arrays larger than 800GB • At 2,954,291,678 tweets, disk was over 90% utilized.
- 11. PARTITION
- 12. Dirt-Goose Implementation Queries try each Partition by time partition in order id user_iduntil enough data Partition 2 24 ... is accumulated 23 ... id user_id Partition 1 22 ... 21 ...
- 13. LOCALITY
- 14. Problems w/ solution • Write throughput
- 15. T-Bird Implementation Partition by primary key Partition 1 Partition 2 id text id text 20 ... 21 ... 22 ... 23 ... 24 ... Finding recent tweets 25 ... by user_id queries N partitions
- 16. T-Flock Partition user_id index by user id Partition 1 Partition 2 user_id id user_id id 1 1 2 21 3 58 2 22 3 99 2 27
- 17. Low Latency PK Lookup Memcached 1ms T-Bird 5ms
- 18. Principles • Partition and index • Index and partition • Exploit locality (in this case, temporal locality) • New tweets are requested most frequently, so usually only 1 partition is checked
- 19. The three data problems • Tweets • Timelines • Social graphs
- 21. What is a Timeline? • Sequence of tweet ids • Query pattern: get by user_id • High-velocity bounded vector • RAM-only storage
- 22. Tweets from 3 different people
- 23. Original Implementation SELECT * FROM tweets WHERE user_id IN (SELECT source_id FROM followers WHERE destination_id = ?) ORDER BY created_at DESC LIMIT 20 Crazy slow if you have lots of friends or indices can’t be kept in RAM
- 25. Current Implementation • Sequences stored in Memcached • Fanout off-line, but has a low latency SLA • Truncate at random intervals to ensure bounded length • On cache miss, merge user timelines
- 26. Throughput Statistics date daily pk tps all-time pk tps fanout ratio deliveries 10/7/2008 30 120 175:1 21'000 11/1/2010 1500 3'000 700:1 2'100'000
- 28. MEMORY HIERARCHY
- 29. Possible implementations • Fanout to disk • Ridonculous number of IOPS required, even with fancy buffering techniques • Cost of rebuilding data from other durable stores not too expensive • Fanout to memory • Good if cardinality of corpus * bytes/datum not too many GB
- 30. Low Latency get append fanout 1ms 1ms <1s* * Depends on the number of followers of the tweeter
- 31. Principles • Off-line vs. Online computation • The answer to some problems can be pre-computed if the amount of work is bounded and the query pattern is very limited • Keep the memory hierarchy in mind
- 32. The three data problems • Tweets • Timelines • Social graphs
- 34. What is a Social Graph? • List of who follows whom, who blocks whom, etc. • Operations: • Enumerate by time • Intersection, Union, Difference • Inclusion • Cardinality • Mass-deletes for spam • Medium-velocity unbounded vectors • Complex, predetermined queries
- 35. Inclusion Temporal enumeration Cardinality
- 36. Intersection: Deliver to people who follow both @aplusk and @foursquare
- 37. Index Index Original Implementation source_id destination_id 20 12 29 12 34 16 • Single table, vertically scaled • Master-Slave replication
- 38. Problems w/ solution • Write throughput • Indices couldn’t be kept in RAM
- 39. Edges stored in both directions Current solution Forward Backward source_id destination_id updated_at x destination_id source_id updated_at x 20 12 20:50:14 x 12 20 20:50:14 x 20 13 20:51:32 12 32 20:51:32 20 16 12 16 • Partitioned by user id Partitioned by user • Edges stored in “forward” and “backward” directions • Indexed by time • Indexed by element (for set algebra) • Denormalized cardinality
- 40. Challenges • Data consistency in the presence of failures • Write operations are idempotent: retry until success • Last-Write Wins for edges • (with an ordering relation on State for time conflicts) • Other commutative strategies for mass-writes
- 41. Low Latency write cardinality iteration write ack inclusion materialize 1ms 100edges/ms* 1ms 16ms 1ms * 2ms lower bound
- 42. Principles • It is not possible to pre-compute set algebra queries • Partition, replicate, index. Many efficiency and scalability problems are solved the same way
- 43. The three data problems • Tweets • Timelines • Social graphs
- 44. Summary Statistics writes/ reads/second cardinality bytes/item durability second Tweets 100k 1100 30b 300b durable Timelines 80k 2.1m a lot 3.2k volatile Graphs 100k 20k 20b 110 durable
- 46. Principles • All engineering solutions are transient • Nothing’s perfect but some solutions are good enough for a while • Scalability solutions aren’t magic. They involve partitioning, indexing, and replication • All data for real-time queries MUST be in memory. Disk is for writes only. • Some problems can be solved with pre-computation, but a lot can’t • Exploit locality where possible