Apache Kafka® Delivers a Single Source of Truth for The New York Times
13 likes2,227 views
The New York Times stores every piece of content it has ever published in Apache Kafka. It takes a log-based approach where all content is published as events to a central "Monolog" topic in Kafka. This log becomes the single source of truth, and all applications consume from the log. The content is stored in Kafka as Protobuf messages with a well-defined schema. A separate "Skinny Log" topic contains notifications that content was processed, to enable caching and track service level objectives. This log-based architecture aims to decouple producers and consumers of content and make the system more flexible and scalable.
Ad
More Related Content
What's hot (20)
Similar to Apache Kafka® Delivers a Single Source of Truth for The New York Times (20)
Ad
More from confluent (20)
Ad
Recently uploaded (20)
Apache Kafka® Delivers a Single Source of Truth for The New York Times
- 1. The Source of Truth 2018-05-09 The New York Times Why The New York Times Stores Every Piece of Content Ever Published in Kafka
- 2. 1 Boerge Svingen was a founder of Fast Search & Transfer (alltheweb.com, FAST ESP). He was later a founder and CTO of Open AdExchange, doing contextual advertising for online news. He is now working on search and backend platforms at The New York Times. Boerge Svingen Director of Engineering, The New York Times
- 3. 2 Housekeeping Items " This session will last about an hour. ● This session will be recorded. " You can submit your questions by entering them into the GoToWebinar panel. " The last 10-15 minutes will consist of Q&A. ● The slides and recording will be available after the talk.
- 4. Boerge Svingen Director of Engineering at the New York Times, working on backend systems.
- 5. Topic: How is published content made available to applications and services?
- 8. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 9. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 13. 20 years on the web
- 14. The New York Times Company Archives
- 15. The New York Times Company Archives
- 16. The New York Times Company Archives
- 17. The New York Times Company Archives
- 18. The New York Times Company Archives
- 19. The New York Times Company Archives
- 20. The New York Times Company Archives
- 21. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 23. A rather typical API-based architecture.
- 24. Disadvantages with this approach … The consumers have to know about all the producers of content.
- 25. Disadvantages with this approach … Producer APIs have to live forever.
- 26. Disadvantages with this approach … Every API tends to be different.
- 27. Disadvantages with this approach … Every API tends to return data with a different (implicit) schema.
- 28. Disadvantages with this approach … We have no efficient way of reading old content in bulk, so it’s hard to replace service stores.
- 29. Disadvantages with this approach … Most services have to manage permanent state.
- 30. Disadvantages with this approach … It is difficult to change the (non-existent) schema, leading to inconsistencies and duplication.
- 31. Disadvantages with this approach … We get monoliths that try to be everything for everyone.
- 32. Disadvantages with this approach … It’s hard to develop new products and change current ones.
- 33. We wanted to decouple producers and consumers of content
- 34. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 35. Make the log the source of truth
- 36. First covered by Martin Kleppmann Turning the database inside-out with Apache Samza Designing Data-Intensive Applications.
- 37. The database becomes derived The database is now secondary, and can be recreated by replaying the log at any time.
- 38. Always be replaying A consumer can start consuming at any point in the log, and just keep going.
- 39. Always be replaying This means that there’s no distinction between reading archive events and live events.
- 40. Materialized views Every consumer can now have its own, custom, materialized view of the log.
- 41. Materialized views The database schema can be specific to the needs of each consumer.
- 42. Why not PubSub or AWS SNS/SQS/ Kinesis? For most use cases, Kafka is used as a message broker, and the log is an implementation detail.
- 43. Why not PubSub or AWS SNS/SQS/ Kinesis? For this use case, the log is the point.
- 44. Why not PubSub or AWS SNS/SQS/ Kinesis? Two requirements: 1. We most retain the log forever. 2. Since messages have causal relationships, consumption must be ordered.
- 45. Why not PubSub or AWS SNS/SQS/ Kinesis? Only Kafka gives us an ordered log with infinite retention.
- 46. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 47. The schema
- 48. Protobufs All assets are published to Kafka as protobuf binaries.
- 49. The schema. All published assets are identified by a URI: nyt://article/186faf12-24a0-4dda-b737-018cee0b81cd
- 50. Protobufs The schema has lots of different types of messages.
- 51. Article Video Image Slideshow Interactive Playlist Person Title Organization Subject Location Program Promo Package List
- 52. Protobufs A single Event type is used for all messages, wrapping the actual asset.
- 53. Protobufs Assets are published in normalized form.
- 54. Article 1 Byline 1 Tag 1 Section 1 Image 2 Image 1 Tag 2 Article 2 Section 2 Image 3
- 55. Shared properties Asset types have shared properties, but protobufs don’t have interfaces.
- 56. Shared properties We really want interfaces.
- 57. Shared properties We use composition instead.
- 61. message Article { PublicationProperties publicationProperties = 1; // Publication metadata CreativeWork creativeWork = 2; // Editorial metadata PromotionalProperties promotionalProperties = 3; // Promotional metadata PrintInformation printInformation = 4; // Print metadata Dateline dateline = 5; // Where reporting for the article occurred int32 wordCount = 6; // The number of words of body text. DocumentBlock body = 7; // The body of the article }
- 62. Validation All assets are validated before they are published to Kafka.
- 64. Schema governance Custom linter to check for forwards and backward compatibility.
- 65. Schema governance Breaking changes are not allowed, like deleting a field or changing a type.
- 66. Schema governance The linter will warn against risky changing, like changing a custom type.
- 67. “Virtual” schema team We have a cross-functional team of people across the organization who manage the schema evolution.
- 68. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 69. Monolog
- 70. The Monolog Single partition, totally ordered, infinite retention.
- 71. The Monolog The Source of Truth for published content.
- 72. The Monolog Contains everything published since 1851.
- 73. Article 1 Dateline 1 Credit 1 Section 1 Image 2 Image 1 Credit 2 Article 2 Section 2 Image 3
- 76. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 77. The Skinny Log
- 78. The Skinny Log A Kafka topic containing notifications signifying some system acting on a Monolog event.
- 79. The Skinny Log A Skinny Log message says, “I processed event X at time Y”.
- 80. The Skinny Log Notification are acted on by downstream systems.
- 81. The Skinny Log Example: A list service consumes a publish for an asset, updates the list, and posts a notification saying that list has been updated.
- 83. Active Cache Invalidation The Skinny Log is used for active cache invalidation.
- 84. Active Cache Invalidation A notification saying that an asset has been updated is used by a downstream system to invalidate its cache.
- 85. CMS WebMonolog Gateway Asset store GraphQLAPI CMS WebMonolog Gateway Asset store GraphQLAPI Fastly CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway Invalidator CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway Invalidator CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway Invalidator CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway Invalidator Invalidator CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway Invalidator Invalidator CMS WebMonolog Gateway Asset store GraphQLAPI Fastly Skinny log Skinnygateway Invalidator Invalidator
- 86. Tracking SLOs
- 87. Metrics/SLOs The Skinny Log has a complete record of when every update is processed by every system.
- 88. Metrics/SLOs We use the Skinny Log to calculate metrics and check our SLOs.
- 89. Metrics/SLOs As in, how long does it take from something is published until it’s available on the site, or in a search result.
- 90. Agenda 1. A little history 2. How things used to work 3. Log-based architectures 4. The schema 5. The Monolog 6. The Skinny Log 7. Some challenges
- 92. Timestamps before 1970 Do not work in Kafka.
- 95. Failover We want to fail over between Kafka clusters, transparently, with consumers keeping their offsets.
- 96. Auth is hard
- 97. Auth on GCP On GCP, all brokers are on the public internet, requiring stronger auth.
- 98. 98 Q & A
- 99. 99 Thank you for joining us!