Before Kaggle
4 likes1,951 views
Many think that a Data Science is like a Kaggle competition. There are, however big differences in the approach. This presentation is about designing carefully your evaluation scheme to avoid overfitting and unexpected production performances.
Ad
More Related Content
What's hot (20)
Viewers also liked (14)
Ad
Similar to Before Kaggle (20)
Ad
Recently uploaded (20)
Before Kaggle
- 1. Before Kaggle From a business goal to a ML problem Pierre Gu(errez @prrgu(errez
- 2. • Data Science competitions platform (There are others : DataScience.net in France) • 332,000 Data Scientists • today : 192 competitions, 18 active + 516 In class, 12 active • Prestigious clients : Axa, Cern, Caterpillar, Facebook, GM, Microsoft, Yandex… What is ?
- 3. • Price pool? • 325,000 $ to make on August 31st • Good luck with that ! • Not a good hourly wage • today : 192 competitions, 18 active Understand : • Lot’s of datasets about approximately every DS topic • Lot’s of winner solutions, tip and tricks, etc… • Lot’s of “beat the benchmark” for beginners I discovered/tested there : GBT, xgboost, Keras, word2vec, BeautifulSoup, hyperopt, ... Why should I join ?
- 4. Most of the time: • You have a train set with labels and a test set without labels. • You need to learn a model using the train features and predict the test set labels • Your prediction is evaluated using a specific metric • The best prediction wins What is a Data Science Competition ?
- 5. Most of the time: • You have a train set with labels and a test set without labels. • You need to learn a model using the train features and predict the test set labels • Your prediction is evaluated using a specific metric • The best prediction wins What is a Data Science Competition? Why AUC? F1 score? Log loss? Could that depend on my train/test split? Where do they come from ? Do you always have some? Why is the split this way? Random? Time?
- 6. What you don’t learn on Kaggle (or in class?): • How to model a business question into a ML problem. • How to manage/create labels. (proxy / missing…) • How to evaluate a model: • How to choose your metric • How to design your train/test split • How to account for this in feature engineering Understanding this actually helps you in Kaggle competition : • How to design your cross validation scheme (and not overfit) • How to create relevant features • Hacks and tricks (leak exploitation J) What is a Data Science Competition?
- 7. Scikit learn cheat sheet
- 8. Christophe Bourguignat DS cheat sheet @chris_bour Today
- 9. • Introduction • Labels? • Train and test split? • Feature Engineering? • Evaluation Metric? Introduction
- 10. • Introduction • Labels? • Train and test split? • Feature Engineering? • Evaluation Metric? Introduction The newcomer disillusion The produc(on bad surprise The business obfusca(on
- 11. • Senior Data Scientist at Dataiku (worked on churn prediction, fraud detection, bot detection, recommender systems, graph analytics, smart cities,…) • (More than) Occasional Kaggle competitor • Twitter @prrgutierrez Who I am
- 12. • Senior Data Scientist at Dataiku (worked on churn prediction, fraud detection, bot detection, recommender systems, graph analytics, smart cities,…) • (More than) Occasional Kaggle competitor • Twitter @prrgutierrez Who I am
- 14. • Everywhere is fraud E-business, Telco, Medicare,… • Easily defined as a classification problem • Target well defined ? • E-business : yes with lag • Elsewhere : need checks, labels are expensive Fraud Detection
- 15. • Wikipedia: “Churn rate (sometimes called attrition rate), in its broadest sense, is a measure of the number of individuals or items moving out of a collective group over a specific period of time” = Customer leaving Churn
- 16. • Subscription models: • Telco • E-gamming (Wow) • Ex : Coyote -> 1 year subscription -> you know when someone leave • Non subscription models: • E-Business (Amazon, Price Minister, Vente Privée) • E-gamming (Candy Crush, free MMORPG) -> you approximate someone leaving Candy Crush: days / weeks MMORPG: 2 months (holidays) Price Minister: months Two types of Churn
- 17. • Predict if a vehicle / machine / part is going to fail • Classification Problem: • Given a future horizon and a failure type. Will this happen for a given vehicle ? -> 2 parameters describe the target • Vary a lot the target -> spurious correlation • Just choose it as the result of the exact business need Predictive Maintenance
- 18. • Target is “will like” or “will buy” • Target is often proxy of real interest (implicit feedback) Recommender System
- 19. • Can you model the problem as a ML problem? • Ex : predictive maintenance • Ask the right question from a business point of view. Not what you know how to do. • Is your target a proxy? • Recommendation system • May need bandit algorithm • Is it easy to get labels? • Ex : Fraud detection • Can be expensive • Mechanical Turk can be the answer Summary on Labels
- 20. • Random Split • Just like in school Train / test split • When and why ? -‐> When each line is independent from the rest (not that common !) image, document classifica(on, sen(ment analysis (“but aha is the new lol” ) -‐> When you want to quickly iterate / benchmark: “is it even possible?” -‐> When you want to sell something to your boss
- 21. • Column / group based Ex : Caterpillar challenge • Predict a price • for each tube id • Tube id in train and test are different Objective : being able to generalize to other tubes! Train / test split
- 22. • Time based • Simply separate train and test on a time variable • When and Why? -> When you want a model that “predict the future” -> When things evolve with time! (most problems!) -> Examples : Add click prediction, Churn prediction, E-business Fraud detection, Predictive maintenance,… Train / test split
- 23. • No subscription example • Target : 4 month without buying • Features ? Train / test split : Churn example
- 24. Ex : Train and predict scheme Time T : present (me T – 4 month Data is used for target crea(on : ac(vity during the last 4 months Data is used for feature genera(on. Use model to predict future churn Train model using features and target
- 25. Ex : Train Evaluation and Predict Scheme Time T : present (me T – 4 month Data is used for target crea(on : ac(vity during the last 4 months Data is used for feature genera(on Valida&on set Use model to predict future churn Training Evaluate on the target of the valida(on set T – 8 month Data is used for features genera(on. Data is used for target crea(on : ac(vity during the last 4 months
- 26. • More complex design • Graph sampling (fraud rings ? ) • Random sampling in client / machine life • Mix of column based and time based … • The rule : 1) What is the problem ? 2) To what would I like to generalize my model ? Future ? Other individuals ? … 3) => Train / Test split Train / test split
- 27. • Predictive Maintenance problem • Objective : predict failure in next 3 days. • Metric is proportional to accuracy (and 0.57 is the best score !) • Link to data : https://www.phmsociety.org/events/conference/phm/14/data-challenge EX PHM Society (Fail example)
- 28. • Failures EX PHM Society
- 29. • Usage EX PHM Society
- 30. • Part Replacements EX PHM Society
- 31. • How to design the evaluation scheme? • What is the probability that an asset fail in the next 3 days from Now? -> classification problem -> Time based split -> but how do I create a train and a test? • Choose a date and evaluate what happens 3 days later? -> pb : not enough failures happening • Choose several dates for each asset? -> beware of asset over-fitting • In the challenge : random selection of (asset, date) in the future + over sampling of failures. EX PHM Society
- 32. • Basic Feature engineering EX PHM Society
- 33. • Random Sampling EX PHM Society This is decent! « With some more work I could have a model that beat randomness enough to be useful »
- 34. • Time based split EX PHM Society Wait what?
- 35. • TIME LEAK EX PHM Society
- 36. • TIME LEAK EX PHM Society Tree cuts
- 37. • Beware of the distribution of you features! • Is there a time dependency? • Ex : count, sum, … that will only increase with time • -> Calculate count and sum rescaled by time / in moving windows instead. • Can be found in Churn, Fraud detection, Ad click prediction,… • A categorical variable dependency? • Ex : email flag in fraud detection • Is there a Network dependency? • Ex : Fraud / Bot detection (network features can be useful but leaky) Feature Engineering
- 38. • Final trick : - Stack train and test and add is_test boolean - Try to predict is_test - Check if the model is able to predict - If so : - check the feature importance - Remove / modify feature and iterate Feature Engineering
- 39. • Final trick: • Back to Phm example: Feature Engineering Huge (me leak !
- 40. • “Treshold dependant” • Accuracy • Precision and Recall • F1 score • “Treshold independant” • AUC • Log Loss • Others (Mean average precision)… Evaluation metric : Classification
- 41. • “Treshold dependant” • Accuracy • Precision and Recall • F1 score • “Treshold independant” • AUC • Log Loss • Others (Mean average precision)… • Customs Evaluation metric : Classification Not good if unbalanced target When you have an order problem When you are going stochas(c When you need to s(ck to business Accuracy alterna(ve
- 42. • Custom metrics • Cost based • Ex Fraud: • Mean loss of 50 $ / fraud (FN) • Mean loss of 20 $ / wrongly cancelled transaction (FP) • F1 score often used in papers • in practice, you often have a business cost Evaluation metric : Classification TP FN TN FP
- 43. • Custom metrics • Fraud Example 1: • “I have fraudsters on my e-business website” • I generate a score for each transaction • I handle this by manually handling transactions with score higher than threshold • I have 1 person that does this fulltime and able to deal with 100 transactions / day • The rest is automatically accepted -> AUC is not bad -> Recall in 100 transactions / day -> Total money blocked 100 transactions / day In practice AUC more stable… But the money metric can also be used for communication. Evaluation metric : Classification
- 44. • Custom metrics • Fraud Example 2: • “I have fraudsters on my e-business website” • I generate a score for each transaction • I handle this automatically by blocking all transactions with score higher than threshold -> AUC is not bad… But don’t give threshold value. -> F1–Score? -> Cost based is better Evaluation metric : Classification
- 45. • My cheat sheet Evaluation metric : Classification Metric Op&mized By ML model ? Treshold Dependant Applica&on example Accuracy YES YES image classifica(on, nlp … F1-‐score NO YES ? Papers ? AUC NO NO fraud detec(on, churn, healthcare … Log-‐Loss YES NO add click predic(on Custom metric NO ? all ?
- 46. • Business Question dictates Evaluation Scheme! • test set design • evaluation metric • Indirectly impact feature engineering • Indirectly impact label quality • Think (not too much) before coding • Don’t try to optimize the wrong problem! Conclusion
- 47. Thank you for your attention!