BDW16 London - Scott Krueger, skyscanner - Does More Data Mean Better Decision Making?
0 likes421 views
The document discusses the importance of data quality in decision making and the challenges faced in ensuring data integrity and validity. It introduces a unified log approach for assessing data quality, highlights the significance of a collaborative culture in data handling, and emphasizes the necessity of measuring decision quality. Ultimately, it depicts a scenario where improved data practices lead to a measurable increase in sales performance.
![+ float device_diagonal_screen_size = 19;
+ float device_diagonal_screen_size = 19 [deprecated=true];
+ DisplayMeasurement diagonal_screen_size = 26;
DATA CHANGE:WE DON'T ALWAYS GET IT RIGHT
0
5
Time
Event Definition Changes
T1
T2
T3
+ // float device_diagonal_screen_size = 19 [deprecated=true];
+ DisplayMeasurement diagonal_screen_size = 26;
2 rules:
1. Maintain
backwards
compatibility.
2. Rebuild.](https://image.slidesharecdn.com/bdw16london-scottkruegerskyscanner-doesmoredatameanbetterdecisionmaking-161107140721/85/BDW16-London-Scott-Krueger-skyscanner-Does-More-Data-Mean-Better-Decision-Making-29-320.jpg)
BDW16 London - Scott Krueger, skyscanner - Does More Data Mean Better Decision Making?
- 1. DOES MORE DATA MEAN BETTER DECISION MAKING? (Assessing Data Quality with a Unified-Log and a bit of Stream- Processing) Scott Krueger Data Architect
- 2. ATALE OF DATA-DRIVEN DECISION MAKING - ACT ONE Me: "@StreamEngine - how many user sessions have we had from Illinois in the last 4 hours from those #ChicagoRocks tweets?" StreamEngine:" In the last 4 hours, we have had 43, 578 new user sessions from Ohio as a result of the #ChicagoRocks tweets" Me: "How confident are you about that?" (I need to make a quick call here) StreamEngine: "Sorry, what was that?"
- 3. WHY ARE WETALKING ABOUTTHIS Analytical Sciences Data Quality Business and Culture Tech Talks at Big Data Week, London 2016
- 4. 2020
- 5. MOTIVATIONS
- 7. MOTIVATIONS "From now on, our cars will more deeply understand that buses (and other large vehicles) are less likely to yield to us than other types of vehicles, and we hope to handle situations like this more gracefully in the future." http://www.bbc.co.uk/news/technology-35692845
- 8. SO WHAT'S GOING ON HERE? WE ARE CREATORS OF POOR QUALITY DATA Machines (we make machines that consume / create data) Software (we create software that consumes / creates data sets) photo: Faruk Ates, https://flic.kr/p/stxXK
- 10. Pillar 1: Data Integrity Data completeness - is it there? is it in tact? are the ‘required-to-be-of-value’ fields there Data Interpretation - what is that thing? what does ‘cost’ mean? Data change - we don't use this anymore so I’m not writing it anymore. Oh, you’re still reading it? Pillar 2: Data validity - what's in there? Values make sense? Values expected? Data presence* - are the messages making it? Are there as many as there should be when data was created? Is this an expected volume? THE 2 (OR SO) PILLARS OF DATA QUALITY http://www.newyorkinternationallimousines.com/
- 11. DATA INTEGRITY A definition of what the data is so it can be turned into a meaningful piece or set of information Varies with approach and ‘structure’ of data Event Schemas (not to be confused with the relational DB term) Examples: protocol buffers, thrift, avro
- 12. DATA INTEGRITY:THE GREATTRADE-OFF Somewhere between left and right something has to prepare data for usage elsewhere.There is cost associated with every position. Data-In Data-Out Schema-on-Write Schema-on-Read Schema-Inbetween
- 13. ATALE OF DATA-DRIVEN DECISION MAKING - ACTTWO Me: "@StreamEngine - I would like to measure how effective all of our data-driven decision making is. I need a measure of quality. I think you can help me with this." StreamEngine: "Are you from the future?" Me: "I'm not from it, but I'm thinking about it..."
- 14. WHAT ARE WE DOING ABOUT IT? This requires a brief understanding of our 'unified log' approach at skyscanner
- 15. SKYSCANNER EVENT DATA PLATFORM
- 16. EXPLOITTHE PLATFORM - MAKE USE OF WHAT YOU HAVE
- 17. DATA INTEGRITY (SCHEMAVALIDATION) Data definition - a message that doesn't fit throws an exception Try...Catch...Log To SchemaValidation Failure “stream/topic” with message
- 18. SchemaValidation
- 19. DATAVALIDATION: EVERYONE PLAYS A PART Everywhere between left and right everything has a data validation opportunity Data-In Data-Out Validation-on-Write Validation-on-Read Validation-Inbetween
- 22. WHAT DOES IT ALL LOOK LIKE? alert! alert!
- 23. TO IMPROVE QUALITY…. … ISTO CLOSETHE LOOP
- 24. EVERYONE PLAYS A PART A shared repository for event structure and validation rules Any service that logs events runs automated tests that use this repository A generic stream service that assesses data quality and gives the heads up to consumers Data consumers who find new data quality mishaps commit back to the repository
- 25. IFYOU CAN'T MEASURE IT HOW DOYOU KNOW YOU ARE IMPROVINGTHINGS? Quality of decision making = 100 - (((# of data issues detected + recent commits for improved detection) / # high quality events logged)))*100) example: 99.8 %
- 26. TIPS FOR IMPROVEMENT Master Data Management Metadata Management Handling Data Change Culture
- 27. MASTER DATA MANAGEMENT (“ONE SOURCE OF REFERENCE/LOOKUP DATA”) Simple rules to maintain consistency of reference data use of enums/constants in schemas for reference data sets you don't provide in your systems authoritative data sources (your internal data services; industry standard sets e.g. "IATA" for travel, ISO geography/timezones etc.) Bring this ref data as close to the processing as possible API, csv / json, tables, trans logs -> Unified Log Topic
- 28. METADATA MANAGEMENT (DATA PROVENANCE AND OTHER NICETIES) Data Debugging Data flow measurements - how long did it take for my message to go through this pipeline? Historical records - transparency for everyone (you the business operator, and you the customer) Governance and regulation - data quality laws? http://www.forbes.com/sites/forbestechcouncil/ 2016/04/29/how-companies-can-leverage-real-time-platforms-and-metadata-to-improve- healthcare-delivery/2/#53fbea89480b
- 29. + float device_diagonal_screen_size = 19; + float device_diagonal_screen_size = 19 [deprecated=true]; + DisplayMeasurement diagonal_screen_size = 26; DATA CHANGE:WE DON'T ALWAYS GET IT RIGHT 0 5 Time Event Definition Changes T1 T2 T3 + // float device_diagonal_screen_size = 19 [deprecated=true]; + DisplayMeasurement diagonal_screen_size = 26; 2 rules: 1. Maintain backwards compatibility. 2. Rebuild.
- 30. DATA QUALITY CULTURE photo: scott krueger
- 31. WHAT ARE YOU GOINGTO DO ABOUT IT? Understand the causes and details of data problems in your services Unify and simplify - one source of truth for everything: reference data, reports, archive, formulae, validation rules, data definitions, metdadata Start measuring - this is your baseline and allows you to measure confidence in decision making Work (or evolve) your tech Fix the system, stop moaning about it It’s never too late
- 32. ATALE OF DATA-DRIVEN DECISION MAKING - ACTTHREE -THE FINALE Me: "@StreamEngine - give me some decision quality numbers!" StreamEngine: "Right now, decision quality is at 96.05%. This time last week it was 92.4%. Well done! In 1 week sales are up 2 % and this is positively correlated to the decisions you made with this data. Would you like me to predict sales uplift over the next month if you improve decision quality by 1 %? Me: "You bet I would..."