To Explain Or To Predict?
2 likes•1,479 views
1) The document discusses the differences between explanatory and predictive modeling in scientific research. 2) Explanatory models are used to test causal theories, while predictive models are used to predict new records or scenarios. 3) Explanatory power and predictive accuracy are different and one cannot be inferred from the other. The best explanatory model is often not the best predictive model and vice versa.
![1072 articles of which52 empirical with predictive claims“Examples of [predictive] theory in IS do not come readily to hand, suggesting that they are not common”Gregor, MISQ 2006](https://image.slidesharecdn.com/toexplainortopredictgeorgetownoct09-091030194041-phpapp02/85/To-Explain-Or-To-Predict-18-320.jpg)
![Three Current ProblemsPrediction underappreciatedDistinction blurredInappropriate modeling/assessment“While the value of scientific prediction… is beyond question… the inexact sciences [do not] have…the use of predictive expertise well in hand.”Helmer & Rescher, 1959](https://image.slidesharecdn.com/toexplainortopredictgeorgetownoct09-091030194041-phpapp02/85/To-Explain-Or-To-Predict-49-320.jpg)
To Explain Or To Predict?
- 1. Galit ShmuéliGeorgetown UniversityOctober 30, 2009To Explain or To Predict?Explanatory vs. Predictive Modeling in Scientific Research
- 2. The path to discovery
- 5. Statistical modeling in social science researchPurpose: test causal theory (“explain”)Association-based statistical models Prediction nearly absent
- 6. Lesson #1:Whether statisticians like it or not,in the social sciences,association-based statistical models are used for testing causal theory.Justification: a strong underlying theoretical model provides the causality.
- 7. Definition: Explanatory ModelA statistical model used for testing causal theory(“proper” or not)
- 8. Definition: Predictive ModelAn empirical model used for predicting new records/scenarios
- 10. Multi-page sections with theoretical justifications of each hypothesis
- 11. Concept operationalizationPovertyTrustAngerEconomic stabilityWell-being4 pages of such tables
- 12. Statistical model (here: path analysis)
- 15. Lesson #2In the social sciences,empirical analysis is mainly used for testing causal theory.Empirical prediction is considered un-academic.Some statisticians share this view: The two goals in analyzing data... I prefer to describe as “management” and “science”. Management seeks profit... Science seeks truth.Parzen, Statistical Science 2001
- 16. Prediction in the Information Systems literature
- 17. Predictive goal stated?Predictive power assessed?
- 18. 1072 articles of which52 empirical with predictive claims“Examples of [predictive] theory in IS do not come readily to hand, suggesting that they are not common”Gregor, MISQ 2006
- 19. Breakdown of the 52 “predictive” articles
- 20. Why Predict?Scientific use of empirical modelsTo PredictTo Explaintest causal theory(utility)relevancenew theorypredictability
- 21. Why are statistical explanatory models different than predictive models?
- 22. Theory vs. its manifestation?
- 23. “The goal of finding models that are predictively accurate differs from the goal of finding models that are true.”
- 24. Given the research environment in the social sciences, two critically important points are:Explanatory power and predictive accuracy cannot be inferred from one another.The “best” explanatory model is (nearly) never the “best” predictive model, and vice versa.
- 25. Point #1Explanatory PowerPredictive Power≠Cannot infer one from the other
- 26. What is R2 ?
- 27. In-sample vs. out-of-sample evaluation
- 28. out-of-sampleinterpretationprediction accuracyp-valuesPerformance EvaluationR2costsgoodness-of-fitrun timeDanger: type I,II errorsDanger: over-fitting
- 29. Suggestion for social scientists:Report predictive accuracy in addition to explanatory power
- 31. Point #2Best explanatory model≠Best predictive model
- 32. Predict ≠ Explain“We should mention that not all data features were found to be useful. For example, we tried to benefit from an extensive set of attributes describing each of the movies in the dataset. Those attributes certainly carry a significant signal and can explain some of the user behavior. However, we concluded that they could not help at all for improving the accuracy of well tuned collaborative filtering models.” Bell et al., 2008 +?
- 33. Predict ≠ ExplainThe FDA considers two products bioequivalent if the 90% CI of the relative mean of the generic to brand formulation is within 80%-125%“We are planning to… develop predictive models for bioavailability and bioequivalence”Lester M. Crawford, 2005Acting Commissioner of Food & Drugs
- 34. Let’s dig in
- 35. Explanatory goal: minimize model biasPredictive goal: minimize MSE(model bias + sampling variance)
- 36. What isOptimized?BiasPrediction MSEorVar(Y)= uncontrollablebias2 = model misspecificationestimation (sampling variance)
- 37. Linear Regression ExampleUnderspecified modelEstimated modelTrue modelEstimated modelMSE2 < MSE1 when: σ2 large |β2| small corr(x1,x2) high limited range of x’s
- 38. China's Diverging Paths, photo by Clark SmithTwostatistical modeling paths
- 39. Design & CollectionData PreparationGoal DefinitionEDAVariables? Methods?Model Use & ReportingEvaluation, Validation & Model Selection
- 40. Hierarchical dataStudy design & data collectionObservational or experiment?Primary or secondary data?Instrument (reliability+validity vs. measur accuracy) How much data? How to sample?
- 43. Which variables?Multicollinearity?A, B, A*B?theoryassociationsex-post availability
- 44. Methods / ModelsbiasvarianceBlackbox / interpretableMapping to theoryridge regressionensemblesboostingPLSPCR
- 45. Model fit ≠ValidationExplanatory powerEmpirical modelTheoretical modelDataEvaluation, Validation& Model SelectionTraining dataEmpirical modelOver-fitting analysisHoldout dataPredictive power
- 46. Model UseInferenceTest causal theoryNull hypothesisPredictions(utility)RelevanceNew theoryPredictabilityPredictive performanceOver-fitting analysisNaïve/baseline
- 47. Design & CollectionData PreparationGoal DefinitionEDAVariables? Methods?Model Use & ReportingEvaluation, Validation, & Model Selection
- 48. How does all this impact research in the (social) sciences?
- 49. Three Current ProblemsPrediction underappreciatedDistinction blurredInappropriate modeling/assessment“While the value of scientific prediction… is beyond question… the inexact sciences [do not] have…the use of predictive expertise well in hand.”Helmer & Rescher, 1959
- 50. Why?What can be done?Statisticians should acknowledge the difference and teach it!
- 51. It’s time for ChangeTo PredictTo Explain