Introducing Events and Stream Processing into Nationwide Building Society (Robert Jackson and Pete Cracknell, Nationwide Building Society) Kafka Summit London 2019
3 likes1,486 views
Nationwide Building Society is implementing a 'speed layer' using Kafka to enhance data processing capabilities, improve agility, and enable real-time data access in response to regulatory and customer demands. This investment, which is part of a larger £4.1 billion digitization initiative, aims to reduce load on existing systems and streamline data consumption across various business units. Progress includes successfully completing architectural proofs of concept and the launch of initial use cases for efficient event-driven data handling.
Introducing Events and Stream Processing into Nationwide Building Society (Robert Jackson and Pete Cracknell, Nationwide Building Society) Kafka Summit London 2019
- 1. Introducing Streaming at Nationwide Building Society May 2019 Rob Jackson and Pete Cracknell Nationwide Building Society
- 2. 2 Contents 1. Nationwide Building Society 2. What is the business challenge we’re responding to? 3. What is the Speed Layer? 4. Typical current state architecture 5. Target state architecture 6. How does data flow through the Speed Layer? 7. How we consume data from the Speed Layer 8. How the Speed Layer is deployed 9. Progress 10. Streaming assessment 11. Value achieved 12. Demo
- 3. 3 Nationwide Building Society • Formed in 1884 and renamed to become the Nationwide Building Society in 1970 • We’re the largest building society in the world. • A major provider of mortgages, loans, savings and current accounts in the UK and launched the first (or 2nd) Internet Banking Service in 1997 • We recently announced an investment of an additional £1.4 billion (total £4.1bn) over 5 years to simplify, digitise and transform our IT estate. • Confluent and Kafka form the heart of an important part of that investment.
- 4. 4 • Regulation such as Open Banking • Business growth • 24 x 7 availability expectations from customers and regulators • Cloud adoption • Capitalising on our data • A need for agility and innovation … and our existing platforms were making this harder than we’d like.
- 5. The Speed Layer will be the preferred source of data for high-volume read-only data requests and event sourcing. It will deliver secure, near real data from back end systems with speed and resilience. It will use the latest technologies built for cloud and designed to be highly available. It will provide NBS with the first event based real-time data platform ready for digital.DEFINITION FOUR KEY CHARACTERISTICS SCALABILITY: The Speed Layer platform will be built using internet scale technologies hosted on cloud ready PaaS architecture. FAST AND AGILE: The Speed Layer will unlock data from our Systems of Record enabling digital and agile development teams to rapidly deliver new features and services RICH DATA SET: Provide a rich data set enhanced with 3rd party data and analytics RESILIENT: Reduce the load on core systems and isolate them from the demands of the digital platforms. Designed to be highly resilient and tolerate multiple infrastructure failures 5 What is the Speed Layer?
- 6. 6 As is logical E2E Architecture API Gateway Channel Web Services Enterprise Web Services Back-end Services Mainframes Fairly normal, is there a problem?
- 7. 7 API Gateway Stream Processing Mainframes + other sources of data Kafka Topics Target System Architecture CDC Kafka Topics Microservices Channel Services Enterprise Services Protocol adapters WritesReads
- 8. System of Record(s) CDC Replication Engine Source DB Kafka Raw Topic – raw data Stream processingA Microservice Kafka Published Topic – processed data Materialisation Microservice NoSQL tables {REST APIs} 1. Change Data Capture (CDC) is deployed to the System of Record (SoR) and pushes changes from source database to Kafka Topic 2. Kafka topics contain data in the format of the source system. There will be one raw topic per table replicated. Data is typically held here for c7 days. 3. Streams processing (Kafka Streams framework) is used to transform data into processed data made available to consumers through “Published Topics” 4. Kafka Published Topics retain data long term (in line with retention policies and GDPR) and can be used by many Speed Layer Microservices. 5. Speed Layer Microservices are consumers of Kafka Published Topics and push the data they need into their persistence store (NoSQL, in-memory, etc.) 6. APIs expose data to consumers 7. Channel applications call Speed Layer Microservices to request data 8. Note, applications can subscribe to events and respond to events without materialising them in a database, e.g., push notification to device. 124 5 3 6 Consuming Applications 7 8 Data Flow Diagram
- 9. 9 There are three main approaches for consumption of data from Speed Layer. 1. Immediate real time message consumption in the Event Driven pattern, 2. Usage specific data sets are materialised and exposed through APIs in the Request Driven pattern. 3. Functionally aligned enterprise level data stores are materialised. Enterprise/ Functional microservices Consumption Ms & apps Consumers are microservices that subscribe to topics and materialise data to their requirements. A set of functional microservices are created. For example an “account” microservice from which all consuming microservices and applications read account data when needed. Kafka consumers listen and respond to messages that are arriving in near real time and take immediate action on receipt of the message. In this pattern there is no need to materialise the data. SL subs Producer Producer Producer Producer SL subs Producer Producer Producer Producer SL Producer Producer Producer Producer FUNCTIONAL SERVICEEVENT DRIVEN REQUEST DRIVEN Applications and/or services can be re-written to consume data from the Speed Layer to improve performance and reduce demand on back-end systems. Consumption Patterns Overview
- 10. 10 Multi-site deployment and resilience Primary DC for SORs Standby DC for SORs Cloud hosting Deployment 1. CDC writes to a local Kafka Cluster, i.e., in the same DC as the mainframe 2. Kafka topics are replicated to a separate Kafka cluster in our 2nd DC 3. Independent database clusters in each datacentre. 4. When required, Kafka topics are replicated using Confluent Replicator to cloud providers DB
- 11. Progress so far… • Architectural PoC completed: 1. Initial logical proving 2. Functional and non functional proving 3. Load testing/benchmarking in Azure and IBM labs, > 80k TPS through a single broker • Project launched to deliver the production capability and first use cases 1. Split into 3 use cases, with the first one code complete, 2 & 3 progressing well • Adopting Confluent Kafka across multiple lines of business 1. Speed Layer 2. Event Based designs for originations journeys 3. High volume messaging in Payments • Working on Streaming Maturity Assessment with Confluent
- 12. Adopting an Enterprise Event-Streaming Platform is a Journey Nationwide nearly here - with Speed Layer + platforms for Mortgages & Payments - but more potential to share common ways of working and utilise a common platform for more use cases VALUE 1 Early interest 2 Identify a project / start to set up pipeline 3 Mission-critical, but disparate LOBs 4 Mission-critical, connected LOBs 5 Central Nervous System Projects Platform Developer downloads Kafka & experiments, Pilot(s). LOB(s); Small teams experimenting; → 1-3 basic pipeline use cases - moved into Production - but fragmented. Multiple mission critical use cases in production with scale, DR & SLAs. → Streaming clearly delivering business value, with C-suite visibility but fragmented across LOBs. Streaming Platform managing majority of mission critical data processes, globally, with multi-datacenter replication across on-prem and hybrid clouds. All data in the organization managed through a single Streaming Platform. Typically → Digital natives / digital pure players - probably using Machine Learning & AI.
- 13. Expected value (this time next year) Ø Enables agility and autonomy in digital development teams Ø The first use case alone will remove c7bn requests / year from the mainframes. Ø Will help us maintain our service availability despite unprecedented demand Ø Kafka and streaming being adopted across multiple lines of business Ø The move to micro services and Kafka enables Nationwide to onboard new use cases quickly and easily Ø Speed Layer, Streaming and Kafka will help Nationwide head-off the threat from agile challenger banks The Speed Layer will help Nationwide provide customers with a better customer experience leading to better customer retention and new revenue streams.
- 14. 14 Demo of Speed Layer Ø Why we did the Proof of Concept Ø Functional walk through Ø Non-functional view