Prediction for Retrospection: Integrating Algorithmic Stress Prediction into Personal Informatics Systems for College Students’ Mental Health
- 1. 2022.09.22 Prediction for Retrospection: Integrating Algorithmic Stress Prediction into Personal Informatics Systems for College Students’ Mental Health CHI '22 Taewan Kim, Haesoo Kim, Ha Yeon Lee, Hwarang Goh, Shakhboz Abdigapporov, Mingon Jeong, Hyunsung Cho, Kyungsik Han, Youngtae Noh, Sung-Ju Lee, Hwajung Hong
- 2. 2 Personal Informatics System • Help to gain insights from self-reflection on past behaviors • Use passive sensing technology (smartphones and wearable devices) to collect behavioral data • Apply machine learning technologies to suggest proper actions from insights and patterns • Assist users’ self-reflection by incorporating algorithm and explainability
- 3. 3 MindScope Algorithm-assisted stress management system • Determine user stress levels • Explain how the stress level was computed Purpose Data Collection Process • Social • Location • Activity • Sleep • Phone usage 1) Data collection phase: get current stress level as an input four times a day (10 days) 2) Reflection phase: report current stress level determined by ML model and explanation of predictors four times a day(15 days)
- 4. 4 MindScope Type 1: Predicted stress level Type 2: Highlighted data category Type 3: More detailed explanation Level of explanability How explanation of prediction affect users’ perception of the system and self-reflection experience
- 5. 5 System Design • What to explain • How to explain • Level of detail in explanation Iterative Design Process • Pilot study with 30 college students
- 6. 6 System Design • Data • Accelerometer and GPS data, app usage, app type, noise levels in the surrounding environment • Commonly visited locations as context information • Stress data (low, moderately high, high) through ecological momentary assessment (EMA) at 4-hour intervals (11am, 3pm, 7pm, 11pm) • Qualitative information in a form of hashtag Modeling Phase
- 7. 7 System Design • Personalized Stress Prediction Reports • Type 1: Only stress level • Type 2: Categories highlighted among phone usage, social activity, movement, physical action and sleep • Type 3: Granularized context such as major deviations from the norm • Receive 4 reports a day • Confirm whether the prediction was correct • Answer whether the explanation was useful • Answer issue with the explanation every three times of not useful Prediction Phase
- 8. 8 System Design Prediction Phase • Personalized Stress Prediction Reports
- 9. 9 System Design • Stress Interventions through microtasks • Detect opportune moments to perform the microtask • To relieve stress • Send a push notification to suggestion the intervention’s execution • Detect habitual phone usage • Setup, new suggestion, something else • Log completion Prediction Phase
- 10. 10 Implementation Details • Social activity • Number of incoming and outgoing phone calls, call duration, missed call • Audio loudness (every 20 minutes for 5 seconds) • Silence threshold -65db • Location • 5-minute periodic check-up and 10 meters threshold • Sleep • Screen off durations from 6:00 pm to 10:00am • Physical action • Activity Recognition and Transition API • Still, walking, running, riding a bicycle, on a vehicle • App usage (when screen is unlocked) Data Collection (29 data features)
- 11. 11 Implementation Details Machine Learning • gRPC server • Data preprocessing • Synchronization • Normalization • XGBoost • To combine multiple decision trees • Shapley Additive explanations (SHAP) • To measure feature importance
- 12. 12 Method • 36 participants • Perceived Stress Scale (PSS) – 10 items from 0 to 4 • Before / 10 days of modeling / after • User Experience Questionnaire – 15 UX-related items with 7 Likert scale • Application Usage Log – MindScope app • Follow-up Interview • Motivation, stress-management skills, effect of app to perception of stress, acceptance of algorithm, usability 30-minute introductory 25-day field deployment 40-minute interview
- 13. 13 Analysis • PSS scores • One-way repeated measures ANOVA with Greenhouse-Geisser correction • Tukey’s HSD for post-hoc analysis • Open coding with thematic analysis by 3 researchers • How MindScope supported users’ stress management behavior • How users perceive prediction results Quantitative Qualitative
- 14. 14 Result • User engagement • 78% of average response rate for the modeling phase • 85% for the prediction phase • 86% users used intervention feature at least once • Stress level changes • Majority of self-reported stress levels were low and number of low increased slightly from modeling to prediction phase (56.3%, 61.5% resp.) • Ratio of reports having high stress slightly decreased slightly (8.5%, 5.8% resp.) Descriptive Statistics
- 15. 15 Result • Changes in prediction • 63.89% of cases consistent with self-reported stress level • 36.1% adjusted by the user • 59.97% of perceived accuracy • Retraining process in the prediction phase with confirmed stress level raise to 68% of performance Descriptive Statistics
- 16. 16 Result • Stress significantly reduced first, then persisted (A) • It is better than no explanation, but more doesn’t make it useful (B) • The more accurate the algorithm is perceived, the more useful it is Statistical Finding
- 17. 17 Qualitative Findings • The impact of algorithmic stress prediction and explainability on stress management practices • Identifying stress patterns through a data-driven approach • Supporting reminiscence with data about the past • Planning actionable stress intervention to offset identified stressors • User perceptions toward stress prediction algorithms • More data input, the better prediction accuracy • Challenges in data collection • Privacy concerns over personal data
- 18. 18 Qualitative Findings • The level of explainability and self-reflection for managing stress • Type 3 (62% preferred) • Guide specific action to mitigate the stress • Enable to reconstruct the past • Should guarantee accuracy • Type 2 (27%) • Allow proactive reflection by investigating stressor • Value more trustworthy • Require less effort to analyze • Type 1 (10%) • Easy to analyze • Explanability uncovered the reasoning process
- 19. 19 Discussion • Prediction for retrospection: Exploiting algorithms to facilitate technology mediated reflection • Objective algorithm decisions can supplement subjective judgement • Prediction with explanation is more useful and effective • Increase in level of explainability makes more sensitive to the accuracy • Explainability for supporting self-reflection in the algorithm-assised PI system • Stress prediction visualization for promoting user initiative and behavior change • Open-ended algorithmic stress prediction visualization for promoting self reflection
- 20. 20 Discussion and Limitation • Design suggestions for algorithm-assisted self-reflection in PI system • Progressively improve explainability as the interaction with the system increases and the model is trained sufficiently • Co-performing relationship between a user and an algorithmic system is critical to encourage active participation • Short period of examination • No control group • Not strictly counterbalanced explanation types Limitation
- 21. Thank you
Editor's Notes
- #3: Previous studies focused on detecting user’s state instead of assisting user’s self-reflection