Machine Learning for Product Managers
3 likes•898 views
The document presents an overview of machine learning (ML) tailored for product managers, focusing on key aspects such as technical feasibility, customer value, and commercial viability. It emphasizes the importance of balancing model complexity and data volume while highlighting potential pitfalls including overestimation of ML capabilities and ethical concerns related to bias and privacy. Additionally, the document outlines strategies for effective ML implementation, including the need for a customer-led approach and the value of combining multiple models.
Machine Learning for Product Managers
- 1. ©ThoughtWorks 2020 Commercial in Confidence ML for Product Managers
- 2. ©ThoughtWorks 2020 Commercial in Confidence Machine Learning for Product Managers Matt Travers & Mat Kelcey
- 3. ©ThoughtWorks 2020 Commercial in Confidence 3 What is Machine Learning? Technical Feasibility Customer Value Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery
- 4. ©ThoughtWorks 2020 Commercial in Confidence 4 What is Machine Learning? Technical Feasibility Customer Value Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery
- 5. ©ThoughtWorks 2020 Commercial in Confidence artificial intelligence data science machine learning What is machine learning? . . 5
- 6. ©ThoughtWorks 2020 Commercial in Confidence 6 Breaking down the jargon A house price example distance material age 31 BRICK 12 2 WOOD 7 15 BRICK 13 sale price $250,000 $670,000 ???????? .
- 7. ©ThoughtWorks 2020 Commercial in Confidence 7 Technical Feasibility Customer Value Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery What is Machine Learning? .
- 8. ©ThoughtWorks 2020 Commercial in Confidence Start small and add more data & model complexity in increments. 8 How models & data interact Getting the balance right MLMODELCAPACITY BESPOKE DATA ESTATE Investment .
- 9. ©ThoughtWorks 2020 Commercial in Confidence 9 Two dimensions of data Volume vs complexity. distance material age 31 BRICK 12 2 WOOD 7 15 BRICK 13 sale price $250,000 $670,000 $300,000 .
- 10. ©ThoughtWorks 2020 Commercial in Confidence 10 Two dimensions of data Volume vs complexity. distance material age 31 BRICK 12 2 WOOD 7 15 BRICK 13 17 WOOD 12 sale price $250,000 $670,000 $300,000 $250,000 .
- 11. ©ThoughtWorks 2020 Commercial in Confidence 11 Two dimensions of data Volume vs complexity. distance material age yard 31 BRICK 12 SMALL 2 WOOD 7 LARGE 15 BRICK 13 MEDIUM 17 WOOD 12 SMALL sale price $250,000 $670,000 $300,000 $250,000 .
- 12. ©ThoughtWorks 2020 Commercial in Confidence 12 Two dimensions of data Volume vs complexity. distance material age yard 31 BRICK 12 SMALL 2 WOOD 7 LARGE 15 BRICK 13 MEDIUM 17 WOOD 12 SMALL days to sell 7 14 6 5 .
- 13. ©ThoughtWorks 2020 Commercial in Confidence Never just a single model... . 13
- 14. ©ThoughtWorks 2020 Commercial in Confidence Model options Commodity vs bespoke models. Cloud services and existing models provide a commodity set of capability, but don't differentiate. Bespoke models give the most flexibility, but come at non-trivial engineering cost. . 14
- 15. ©ThoughtWorks 2020 Commercial in Confidence 15 Customer Value Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery What is Machine Learning? Technical Feasibility
- 16. ©ThoughtWorks 2020 Commercial in Confidence 16 Mapping value to investment Getting the balance right Customer value should increase to justify the investment in both data and models. (Just like all products) A clear measure of customer value and a non-ML baseline is essential. CUSTOMER VALUE 16MLMODELCAPACITY BESPOKE DATA ESTATE Investment
- 17. ©ThoughtWorks 2020 Commercial in Confidence 17 Mapping value to investment Getting the balance right If additional investment leads to no growth in user value, look for opportunities on the other dimension. MLMODELCAPACITY BESPOKE DATA ESTATECUSTOMER VALUE Balanced Zone Over-engineered model Redundant data
- 18. ©ThoughtWorks 2020 Commercial in Confidence The stalled value trap Enough already. Customer value plateaus as better models and more data provide no perceivable improvement. 18CUSTOMER VALUE 18MLMODELCAPACITY BESPOKE DATA ESTATE No growth in value
- 19. ©ThoughtWorks 2020 Commercial in Confidence The hype trap We just need some ML! The opportunity for ML is overestimated. Simple rules-based systems already deliver high value and a major ML investment is required to match the customer value. 19CUSTOMER VALUE 19MLMODELCAPACITY BESPOKE DATA ESTATE Marginal value growth after major investment
- 20. ©ThoughtWorks 2020 Commercial in Confidence The moonshot True faith. Value relies on complex models and high volume and complexity of data. High risk, with no feedback on progressing customer value. 20CUSTOMER VALUE 20MLMODELCAPACITY BESPOKE DATA ESTATE
- 21. ©ThoughtWorks 2020 Commercial in Confidence Complementary Approaches The most reliable path. Most successful ML products combine multiple models and datasets, often at different stages of development, which maximise customer value over the life of the product. 21CUSTOMER VALUE 21MLMODELCAPACITY BESPOKE DATA ESTATE
- 22. ©ThoughtWorks 2020 Commercial in Confidence 22 Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery What is Machine Learning? Technical Feasibility Customer Value
- 23. ©ThoughtWorks 2020 Commercial in Confidence Platforms and tooling How to take advantage of the burgeoning ecosystem. ● cloud APIs ● host your own model ● download and tune model ● train existing from own data ● novel model with own data . 23
- 24. ©ThoughtWorks 2020 Commercial in Confidence Data Cost Structure How to manage costs Expect high people costs in data engineering when first accessing data. Storage is cheap but grows as complexity and volume increases 2424 BESPOKE DATA ESTATE DATA ENGINEERS DATA STORAGE Money Icon by Matlo from the Noun Project
- 25. ©ThoughtWorks 2020 Commercial in Confidence ML Model cost Structure People and technology. Compute costs can be significant but typically have gradual growth. Data Scientists are expensive and can cause explosive increases during the transition from generic models to custom models. 2525MLMODELCAPACITY DATA SCIENTISTS COMPUTE COSTS MLaaS Generic Custom Money Icon by Matlo from the Noun Project Free!
- 26. ©ThoughtWorks 2020 Commercial in Confidence 26 Privacy & Ethics Land Mine Icon by Ethan Clark from the Noun Project Function ML unlocks new opportunities for powerful new products that are ethically debatable. Self driving vehicle safety, facial recognition in crime prevention. Explainability Many ML models lack explainability; compounding biases and frustrating those seeking redress. Fraud prevention Data Security All ML requires lots of data; holding this data is a privacy risk. Re-identification is a risk when PI data points can be reproduced from a model. Bias Models reflect their training data. Biased datasets produce biased models. Facial analysis model have perpetuated racism and sexism in hiring and criminal justice. .
- 27. ©ThoughtWorks 2020 Commercial in Confidence 27 What is Machine Learning? Technical Feasibility Customer Value Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery
- 28. ©ThoughtWorks 2020 Commercial in Confidence 28 Robust to failure Combine algorithms for flexibility and nuance. ➜ Zidane ➜ Humour ➜ Sport ➜ Football ➜ Football ➜ Zidane
- 29. ©ThoughtWorks 2020 Commercial in Confidence 29 Augmenting vs Automating Start with a humble UI 29MLMODELCAPACITY BESPOKE DATA ESTATE Confident UI Humble UI AUGMENTING AUTOMATING
- 30. ©ThoughtWorks 2020 Commercial in Confidence 30 What is Machine Learning? Technical Feasibility Customer Value Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery
- 31. ©ThoughtWorks 2020 Commercial in Confidence Custom combinations of known models Novel model IP Competitive Advantage Where are the moats? Best opportunities for competitive advantage are in: ● Data, rather than models ● Combinations ● Usage feedback loops creating a flywheel effect. 31HIGHEST COMPETITIVE. AD. 31MLMODELCAPACITY BESPOKE DATA ESTATE Generic models as a service Spreadsheet models Existing models on exclusive data Usage Data feed- back loops Existing models on replicable data
- 32. ©ThoughtWorks 2020 Commercial in Confidence 32 ML’s double flywheel How usage can create unassailable competitive advantage. EXPERIENCE QUALITY USAGE $ UNIQUE DATA (Volume & Complexity) MODEL INNOVATION
- 33. ©ThoughtWorks 2020 Commercial in Confidence 33 Value creation vs efficiency Where to look for opportunities. Value Creation Efficiency Definition Create customer new customer value which can be monetised. Reduce costs or mitigate risks through efficiency Examples Voice services, self driving vehicles Process prioritisation, fraud detection. Competitive Advantage Potential for flywheel. Incremental and replicable. Timing Explore urgently Build gradually. Likelihood of success Risky, moonshot More proven.
- 34. ©ThoughtWorks 2020 Commercial in Confidence 34 Tech-led or customer-led? What if your CEO just wants some AI. ● Generally we advise customer-led to ensure we are solving real problems ● But ML is a big enough step from known tech that there is the risk of Product teams not knowing what they don’t know. ● ML bootcamps, which are tech-led, for all teams to understand the potential of the tech and re-assess existing Opportunities with new ML-based solutions.
- 35. ©ThoughtWorks 2020 Commercial in Confidence 35 What is Machine Learning? Technical Feasibility Customer Value Commercial Viability What we’ll cover today A look at ML through six product management lenses ©ThoughtWorks 2020 Commercial in Confidence Usability Strategy Delivery .
- 36. ©ThoughtWorks 2020 Commercial in Confidence ML Maturity Where are the skills? There's usually a mix of skills and impacts within an organisation. Cross team Company wide Single team Descriptive Predictive Prescriptive Many teams doing simple things Deep expertise in few teams Capture expertise in tooling . 36
- 37. ©ThoughtWorks 2020 Commercial in Confidence Low risk with low impact? High risk with high impact? A mix? Moonshots or incremental? Research portfolio Portfolio Evolution . 37
- 38. ©ThoughtWorks 2020 Commercial in Confidence feedback? https://l.ead.me/bbRXBr Thank you! Matt Travers [email protected] Mat Kelcey [email protected]