More Related Content
What's hot (20)
Similar to Transactional Streaming: If you can compute it, you can probably stream it. (20)
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Transactional Streaming: If you can compute it, you can probably stream it.
- 1. Transactional Streaming If you can compute it, you can probably stream it. John Hugg March 30th, 2016 @johnhugg / [email protected]
- 2. Who Am I? • First developer on the VoltDB project. • Previously at Vertica and other data startups. • Have made so many bad decisions over the years, that now I almost know what I'm talking about. • [email protected] • @johnhugg • http://chat.voltdb.com
- 4. Operations at Scale • Ingest data from several sources into a horizontally scalable system. • Process data on arrival (i.e., transform, correlate, filter, and aggregate data). • Understand, act, and record. • Push relevant data to a downstream, big data system.
- 5. Data Movement Processing Logic State Management
- 8. Right Now
- 9. Right Now
- 10. Right Now
- 11. One Size Fits All • Analytics and operational stateful stores require different storage engines to be optimal. Columns vs. Rows Vertica vs. VoltDB • Machine Learning Multi-Dim Math Search • Microservices? • Data Value?
- 12. Specifically: Operational Stream Processing and Operational State Where integration makes sense: Leading Edge Operations
- 13. What’s the Difference? • Non-integrated systems means you write glue code, or you use someone’s glue code. • Operational glue code is different from batch-oriented glue code. • Batch or OLAP has huge safety nets for glue code: • HDFS, CSV, immutable data sets • “Blow it away and reload” • Much less time pressure
- 14. Glue Glue You wrote this. 1 User. Tested Well 1000s of users Tested Well 1000s of users Tested Well 1000s of users Community Supplied Many Users
- 15. But I’m not writing “glue code” “I’m just using the well-tested Cassandra driver in my Storm code.” • You’re using a computer network. They are not always reliable. • Storm might fail in the middle of processing. • Cassandra might fail in the middle of processing. • Both systems are tested for this, but not together, using your glue code.
- 16. Operational Glue Code is Hard Main Point: Minimize it
- 18. Use the same system for state and processing. Ensures they are tested together. No independant failures.
- 19. 1 Transaction = 1 Event ACID • Atomic: Either 100% done or 0% done. No in-between. • (Consistent) • Isolated: Two concurrent operations can’t interfere with each other • Durable: If it says it’s done, then it is done.
- 20. Processing Code for a Single Event Database / State
- 21. Processing Code for a Single Event Database / State x x x x Not Atomic
- 22. Romeo And Juliet Explain “Atomicity” Operation 1: Fake your death Operation 2: Tell Romeo
- 23. Processing Code for a Single Event Database / State Processing Code for a Single Event Not Isolated
- 24. “A good example is the best sermon.” - Benjamin Franklin
- 25. Call Center Management http://www.publicdomainpictures.net/ 3000 AgentsMillions of Customers Dashboards & Alerts Billing Actions Events Processing State
- 26. Call Center Management Events • “Begin Call” Calling Number, Agent Id, Start Time, etc… • “End Call” Calling Number, Agent Id, End Time, etc…
- 27. What Kind of Problems • Correlation - Streaming Join • Out-of-order delivery • At least once delivery - How to dedup • Generate new event on call completion - once • Precise Accounting • Precise Stats - Event time vs processing time
- 28. Public Code https://github.com/VoltDB/app-callcenter It’s not finished as of today…
- 29. What’s the Hardest Part? BeginCall code EndCall code State Fake Call Generator (Makes event pairs with delay) Bad Network Transformer (Duplicate & delay) My Client Code
- 31. Schema for Call Center Example CREATE TABLE opencalls ( call_id BIGINT NOT NULL, agent_id INTEGER NOT NULL, phone_no VARCHAR(20 BYTES) NOT NULL, start_ts TIMESTAMP DEFAULT NULL, end_ts TIMESTAMP DEFAULT NULL, PRIMARY KEY (call_id, agent_id, phone_no) ); CREATE TABLE completedcalls ( call_id BIGINT NOT NULL, agent_id INTEGER NOT NULL, phone_no VARCHAR(20 BYTES) NOT NULL, start_ts TIMESTAMP NOT NULL, end_ts TIMESTAMP NOT NULL, duration INTEGER NOT NULL, PRIMARY KEY (call_id, agent_id, phone_no) ); Unpaired call begin/end events Can arrive in any order Any match transactionally moves to the completed calls table
- 33. is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application. Idempotence
- 34. Idempotent Not Idempotent set x = 5; same as set x = 5; set x = 5; x++; not same as x++; x++; if (x % 2 == 0) x++; same as if (x % 2 == 0) x++; if (x % 2 == 0) x++; if (x % 2 == 0) x *= 2; not same as if (x % 2 == 0) x *= 2; if (x % 2 == 0) x *= 2; spill coffee on brown pants eat whole plate of spaghetti
- 35. Idempotent Operations Exactly Once Semantics At-Least-Once Delivery + =
- 36. How to make BeginCall Idempotent? • If call record is in completed calls, ignore. • If the call record is in open calls and is missing end time, ignore. • If call record is in open calls, check if this event completes the call. Yes, handle swapped begin & end • Otherwise, create an new record in open calls table. open calls completed calls Tables
- 37. How to make BeginCall Idempotent? • If call record is in completed calls, ignore. • If the call record is in open calls and is missing end time, ignore. • If call record is in open calls, check if this event completes the call. Yes, handle swapped begin & end • Otherwise, create an new record in open calls table. open calls completed calls TablesIdempotency
- 38. • If call record is in completed calls, ignore. • If the call record is in open calls and is missing end time, ignore. • If call record is in open calls, check if this event completes the call. Yes, handle swapped begin & end • Otherwise, create an new record in open calls table. This thing to the left is a transaction.
- 39. Actual Math https://www.flickr.com/photos/kimmanleyort/13148718593 Accounting & Statistics May Require:
- 40. Counting • Counting is hard at scale. • 2 Kinds of fail: • Missed counts • Extra counts
- 41. Counting Read Read x=27 Write 28 x=27 x=28 Write 28 x=28Value: Incrementer 1: Incrementer 2:
- 42. Processing Code for a Single Event Database / State Processing Code for a Single Event Not Isolated
- 43. Counting Systems with single-key consistency Systems with special features to enable counters ACID transactional systems Systems that enforce a single writer As we say in New England… Performance is wicked variable. Not “Read Committed”
- 44. Accounting • Accounting is just counting, but more so. • Need to be able to increment by amount (or decrement). • Often need to increment/decrement things in groups.
- 45. Accounting • When gamer buys a Mystical Sword of Hegemony, update the following: • Debit the gamer’s rubies or whatever. • Update real-world region stats, like swords sold in gamer’s geo- region, total money spent in gamer’s geo-region etc… • Update game region stats for the current game location, say the “Tar Shoals of Dintymoore”, like number of MSoHs in the region. • Increment any offer-related stats, like record whether the MSoH was offered because of customer engagement algorithm X15 or B12.
- 46. Processing Code for a Single Event Database / State x x x x Not Atomic
- 47. Accounting Systems with single-key consistency Systems with special features to enable counters ACID transactional systems Systems that enforce a single writer As we say in New England… Performance is wicked variable. ?
- 48. Last Dollar Problem • Ad-Tech app wants to show a user an ad from a campaign. • The price of the ad is $0.90. • Advertiser has $1.00 campaign budget left. • If the budget check and the display aren’t ACID, it’s possible to decide to show the ad twice. • Ad-Tech app is forced to choose between over or under-billing.
- 49. Aggregation • Aggregation is just counting and accounting that the system does for you. • Often this is counting chopped up by groups. • Eg. Sword sales by region. % success by offer. • In Call Center, it could be average call length by agent.
- 50. Accounting Aggregation Systems with single-key consistency Systems with special features to enable counters ACID transactional systems Systems that enforce a single writer As we say in New England… Performance is wicked variable. ?
- 51. How to Aggregate Without Consistency? • Use a stand-alone stream processor. • Best fit for aggregation by time, and specifically by processing time, not event time. • Run a query on all the data every time you want the aggregation. • BOO!
- 52. Actual Math What’s the mean and standard deviation of call length chopped up various ways?
- 53. Running Variance is my next band name.
- 55. The Details (mostly) Don’t Matter • Still need to think about performance and likely horizontal partitioning of work. • Integration of State & Processing + Full ACID Transactions => I can program this math without thinking about: • Failure • Interference from weak isolation. • Partial Visibility to State
- 56. Bonus Topics!
- 57. Latency
- 58. Low Latency Can Affect the Decision 500ms Want to be here You lose money here
- 59. Get Into the “Fast Path” • Policy Enforcement in Telco • Fraud Detection “Smoke Tests” • Change what a user sees in response to action: • Change the next webpage content based on recent website actions. • Pick what’s behind the magic door based on how the game is going.
- 61. Problem Factory full of robots Sometimes they break They log metadata
- 62. When Imperfect is Enough • Before: No metadata. Maintenance works on stuff based on their experience, schedules and visual inspection. • Now: Basic stream processing system is up 99% of the time, and provides a much richer guidance to maintenance. Robots fail less often and cost less to operate. • Possible Future: More sophisticated stream processing is up 99.99% of the time and offers even more insight. Robots fail a tiny bit less often and costs are a tiny bit down.
- 63. When Imperfect Isn’t Worth It Probability of Failure (under system X) Expected Average Failure Cost # of Operations x xCost of System X + • I’ve worked on Ad-Tech use cases => High # Operations • Complex Multi-Cluster/System Monsters => High % failure • Billing systems and fraud systems => High cost per failure Licenses Hardware Engineering (Switching Tech)
- 64. More consistent systems don’t have to be more expensive Easier to develop => Less Engineering More Efficient => Less Hardware
- 65. Conclusion - Thank You! • Operations => Integration Wins Analytics, Batch => Use Specialized Tools • With transactions, complex math becomes mostly typing. • Many of these problems can be solved without transactional streaming, but… • It’s going to be harder • It might be less accurate BS Stuff I Don't Know Stuff I Know T H I S TA L K http://chat.voltdb.com @johnhugg [email protected] all images from wikimedia w/ cc license unless otherwise noted