Centralized logging in a changing environment at the UK’s DVLA
0 likes911 views
This presentation discusses logging strategies for a centralized logging environment at the DVLA as their applications move to cloud platforms and new technologies. Initially, application teams were isolated and responsible for their own stacks. The DVLA standardized on Elastic Stack for logging and adopted a shared "Open Services Landscape" platform. They worked with application teams to establish logging standards and libraries. For Kubernetes, they deployed Filebeat on each node and sent logs to Logstash. For serverless applications on AWS, they configured CloudWatch Logs to send to Kinesis and then to Logstash. Elastic tools like curated indices, rollover indices, and index lifecycle management help manage large and changing volumes of log data in the cloud.
Ad
More Related Content
What's hot (20)
Similar to Centralized logging in a changing environment at the UK’s DVLA (20)
Ad
More from Elasticsearch (20)
![Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]](https://cdn.slidesharecdn.com/ss_thumbnails/modernisingonelegalserchwithelastic-jsedit14thjune2021-210625013449-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Centralized logging in a changing environment at the UK’s DVLA
- 1. 1 ElasticON Global Matt Dendle Engineering Team Lead - Cloud Engineering, DVLA
- 2. 2 This presentation and the accompanying oral presentation contain forward-looking statements, including statements concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future operations and expected performance. These forward-looking statements are subject to the safe harbor provisions under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently available information regarding these matters may not materialize. Actual outcomes and results may differ materially from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business and our customers and partners; our ability to continue to deliver and improve our offerings and successfully develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings; our ability to realize value from investments in the business, including R&D investments; our ability to maintain and expand our user and customer base; our international expansion strategy; our ability to successfully execute our go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer retention and expansion; and general market, political, economic and business conditions. Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s website at ir.elastic.co and the SEC’s website at www.sec.gov. Any features or functions of services or products referenced in this presentation, or in any presentations, press releases or public statements, which are not currently available or not currently available as a general availability release, may not be delivered on time or at all. The development, release, and timing of any features or functionality described for our products remains at our sole discretion. Customers who purchase our products and services should make the purchase decisions based upon services and product features and functions that are currently available. All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not currently intend to, update any forward-looking statements or statements relating to features or functions of services or products, except as required by law. Forward-Looking Statements
- 3. 3 Centralised logging in a changing environment
- 4. 4 Agenda Centralised logging in a changing environment Moving to the cloud platform Kubernetes Serverless How elastic helps Summary 1 2 3 4 5
- 5. 5 Get the right drivers and vehicles taxed and on the road as simply, safely and efficiently as possible The Goal of the DVLA
- 6. 6 Application Squads - at the start Isolated, outsourced teams responsible for entire stack of their app
- 7. 7 DVLA - The Open Services Landscape (OSL)
- 8. 8 Moving applications onto the OSL App squads moving/rewriting applications OSL PLATFORM packetbeat metricbeat filebeat auditbeat
- 9. Logging Standards Technical Reference Architecture Self-Service • Agreed with all app squads • Standard log format • Reusable logging libraries • Support for custom fields • Agreed approaches to common problems • New app squads learn the techniques employed here • Embrace automation • Easy on-boarding • Engineers can focus on delivering value rather than repetitive tasks Key concepts that helped
- 10. 10 Agenda Centralised logging in a changing environment Moving to the cloud platform Kubernetes Serverless How elastic helps Summary 1 2 3 4 5
- 11. 11 Logging data flow from Kubernetes Kubernetes + Filebeat per node Logstash MIBI BMC Security Search
- 12. 12 Dealing with new applications Using rollover indices with dynamic index names Logstash Search Use pod namespace as index name
- 13. 13 Dealing with new applications Using rollover indices and dynamic index names Logstash Search Use pod namespace as index name Namespace scanner On new namespace, set up Index template and create first index
- 14. 14 Agenda Centralised logging in a changing environment Moving to the cloud platform Kubernetes Serverless How elastic helps Summary 1 2 3 4 5
- 15. 15 Logging for Lambdas The Simple Way Workload (App) Cloudwatch Logging Lambda Logstash
- 16. 16 Logging for Lambdas Using kinesis data streams Workload (App) Cloudwatch Logging Lambda LogstashKinesis Data Stream
- 17. 17 Automating Lambda Logging Setup Using AWS Cloudwatch CloudTrail Subscribe to “New Log Group Created” event Set log retention period to 7 days Subscribe log groups to Kinesis data stream
- 18. 18 Agenda Centralised logging in a changing environment Moving to the cloud platform Kubernetes Serverless How elastic helps Summary 1 2 3 4 5
- 19. 19 Date Based Indices The benefits ● Index names are composed of the date ● Use curator to delete indices based on this date - (E.g. delete over 90 days old)
- 20. 20 Date Based Indices The drawbacks ● Some days might not have many logs at all - waste of shards! ● Other days might be huge volumes of logs - could fill up disks (especially if you lower shard count)
- 21. 21 Rollover Indices Automatically creating new indices ● All writes reference the alias ● One index is set as the writeable one ● Rollover on size, doc count or time (or combination) ● Need to set up - “bootstrapped”
- 22. 22 Hot/Warm architecture Host your data on suitable nodes
- 23. 23 ILM - Managing your data Index Lifecycle Management
- 24. 24 New Feature - Data Streams Better way for time based data
- 25. 25 Careful with Warm/Cold node disk IOPs
- 26. The list goes on! • Machine learning features to make ML easily accessible • Huge offering for security, with a SIEM app and joining forces with ENDGAME to make Elastic Endpoint • Elastic Cloud offering allows for a single place to administer all of your organizations clusters - one team can be a cluster broker Regular new features and improvements
- 27. 27 Agenda Centralised logging in a changing environment Moving to the cloud platform Kubernetes Serverless How elastic helps Summary 1 2 3 4 5
- 28. In Summary • Isolated app squads work, but with duplicated effort and tech proliferation • Agreeing architectural and patterns based standards really helps • A centralised platform to deliver apps upon is a great bonus • Setting up data collection for K8s and serverless is easy • Elastic have the tools to manage large volumes of data, and the features keep coming • Elastic Cloud - Less time managing infrastructure - more time to add value & new features are easier to use in cloud! Centralised cloud platform helps us deliver valuable apps to UK citizens