CSX: Real-time Business Discovery with the Elastic Stack
0 likes2,745 views
This document discusses how CSX, a freight railroad company, used Elastic to gain real-time insights into their business data. It describes CSX's need to analyze data from various systems and sensors to improve visibility, maintain train-locomotive associations, and democratize access to data. The document outlines how CSX implemented Elastic to explore data without impacting systems, synthesize data streams, and develop proofs of concept like tracking PTC compliance and locomotive faults. This has helped CSX better understand their data and share insights with business partners.
CSX: Real-time Business Discovery with the Elastic Stack
- 1. Chris Jesseman & Evan King CSX Technology January 8, 2019 Real time business discovery with Elastic
- 3. ABOUT CSX ▪ Over 21,000 route miles of track in 23 states, DC, and Canada ▪ Nearly two-thirds of Americans live within CSX’s service territory ▪ Operates an average of 1,300 trains per day ▪ Transports more than 6.5 million carloads of products and raw materials a year ▪ Maintains a fleet of more than 4,000 locomotives and approximately 70,000 freight cars ▪ Safety is a way of life 3
- 4. BUSINESS NEED ▪ Ingest varying data at increasing speeds and density ▪ Support legacy and modern systems ▪ Real-time data discovery that enables development of business rules ▪ Improve analytical insight ▪ Provide an enterprise solution that empowers the business 4
- 5. DATA LANDSCAPE ▪ Mutable, logical entities compound the complexity of capturing and acting on data over satellite, cellular, 220mHz, WiFi, and the track sensors ▪ Data from systems (what should happen) and sensors (what is happening) ▪Formats: EMP, XML, JSON, Protobuf 5
- 6. INITIAL BUSINESS PROBLEMS ▪ Improved visibility of PTC status and compliance ▪ Maintaining Locomotive to Train Association (LTA) ▪ Access to PTC telemetry ▪ Democratizing data 6
- 7. SEVERAL PROBLEMS…ONE STARTING PLACE
- 8. BUSINESS DISCOVERY IN ELASTIC ▪ No need for custom apps ▪ Data exploration does not impact running transaction systems ▪ Let the data be the analyst's tour guide ▪ Use real data to confirm, dispel anecdotes ▪ Synthesize data streams from various sources to 'enrich' the story ▪ Easy to jump right in 8
- 9. SHARED UNDERSTANDING WITH BUSINESS PARTNERS 9 Hey Matt. Here is a good example of Q02802 transitioning to PTC Active track. The algorithm would find them out of compliance. Perfect. That’s the information we need to know when it’s happening.
- 11. DEVELOPING PROOF OF CONCEPTS 11
- 12. DOES THAT ANECDOTE HOLD WATER? 12
- 13. BUSINESS VALUE ▪ Kibana can help to tell the story back to the business partner ▪ Visualizations and dashboard 'POCs' show the business partner what’s possible ▪ More efficient testing…..compare the app with the ‘Kibana truth’ ▪ Democratization of data makes us all better analysts 13
- 15. ARCHITECTURE TENETS ▪ Capture events at their source, even the edge ▪ Humans can't log onto databases or servers in production ▪ Cloud Native ▪ Downstream entities can’t tell upstream what to do ▪ Separation of concerns ▪ Applies traditionally to software architecture ▪ Also apply it to data producers/consumers ▪ Don't query transaction systems for analytics data 15
- 16. LOGICAL ARCHITECTURE SUPPORTS FLOWING DATA ▪ Aqueduct ▪ Distillery ▪ Reservoir ▪ Tap ▪ Data Lake 16 Discovery
- 18. PROTOBUF ▪ Protobuf on the wire forces schema creation and enforcement as data enters Aqueduct ▪ One Protobuf schema per topic ▪ One Elastic index per Protobuf schema ▪ Protobuf schemas and Elastic templates checked into Git happy as a clam ▪ Structured data is good for you ▪ Logstash pipelines use Protobuf codec and can be scaled like microservices ▪ Schemas and templates provide isolation for reliability and scaling ▪ Kafka is partitioned knowing that all data flows will also be going to Elastic 18
- 20. HOW FAST IS THE NETWORK MOVING? 20
- 21. HOW IS PTC COMPLIANCE TRACKING? 21
- 22. WHAT LOCOMOTIVE FAULTS ARE OCCURRING? 22
- 23. MACHINE LEARNING AND PTC COMPLIANCE 23