Using Game Learning Analytics to Improve the Design, Evaluation and Deployment of Serious Games
1 like1,166 views
The document discusses the application of game learning analytics (GLA) to improve the design, evaluation, and deployment of serious games across various domains. It critiques traditional evaluation methods and proposes GLA as a solution for real-time learning progress analysis, offering insights for both educators and developers. Additionally, the document outlines a structured approach for implementing GLA, emphasizing the need for models and tools that facilitate data collection and analysis.
Using Game Learning Analytics to Improve the Design, Evaluation and Deployment of Serious Games
- 1. Using Game Learning Analytics to Improve the Design, Evaluation and Deployment of Serious Games Baltasar Fernandez-Manjon [email protected] @baltaFM Grupo e-UCM www.e-ucm.es 25/04/2018, Institute of Digital Games, Malta
- 2. Serious Games Applied successfully in many domains (medicine, military) with different purposes (knowledge, awareness) Serious Games follow a black box model: ● reporting of final results ● no information of real-time learning progress ● complicates external integration Usual evaluation method: pre-post questionnaires (Calderón & Ruiz, 2015). Very few formally evaluated serious games
- 3. Learning Analytics: “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environment in which it occurs” (Long & Siemens, 2011) Game Learning Analytics Game Learning Analytics breaking the black box model to obtain information while students play. Manuel Freire, Ángel Serrano-Laguna, Borja Manero, Iván Martínez-Ortiz, Pablo Moreno-Ger, Baltasar Fernández-Manjón (2016): Game Learning Analytics: Learning Analytics for Serious Games. In Learning, Design, and Technology (pp. 1–29). Cham: Springer International Publishing. http://doi.org/10.1007/978-3-319-17727-4_21-1.
- 5. H2020 Projets Ivan Perez-Colado, Cristina Alonso-Fernandez, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Game Learning Analytics is not informagic!. IEEE Global Engineering Education Conference (EDUCON), April 18-20, 2018, Santa Cruz de Tenerife, Canary Islands, Spain. Generalize and simplify GLA for serious games Realising an Applied Gaming Eco-System
- 6. Uses of Gaming Learning Analytics ● Game testing – game analytics ■ It is the game realiable? ■ How many students finish the game?, Average time to complete the game? ● Game deployment in the class – tools for teachers ■ Real-time information for supporting the teacher ■ “Stealth” student evaluation ● Supporting more scientific game-based research ■ Formal Game evaluation ■ From pre-post test to evaluation based on game learning analytics
- 7. Learning Analytics Model (LAM) Game Learning Analytics needs a model to drive the analysis For LAMs to provide insight into learning, it is necessary to clearly establish: - requirements to fulfill a LAM definition - realistic expectations of what the outcomes can be False expectations of GLA with deep meaning based on shallow interaction data, assuming the GLA system can infer the game educational design This is not Game Learning Analytics but informagic!.
- 8. Steps to define a LAM 1. Learning goals to be achieved in the game. 2. Game goals (e.g. tasks, levels) Correspondence between learning & game goals. 3. Traces to be sent by the game (follow some standard) 4. Analysis model defined how traces should be analyzed & interpreted. 5. Visualizations game-dependent to extend the default LAM.
- 9. Data Collection Experience API for Serious Games Profile (xAPI-SG): standard interactions model implemented in xAPI with ADL. The model allows tracking of all in-game interactions as xAPI traces (statements). It also simplifies data sharing. xAPI-SG open-source tracker https://github.com/e-ucm/unity-tracker Ángel Serrano-Laguna, Iván Martínez-Ortiz, Jason Haag, Damon Regan, Andy Johnson, Baltasar Fernández-Manjón (2017): Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces 50 (2017) 116–123,
- 10. Data Analysis Traces generated are sent to a LA server that analyzes the traces and generates both visualizations and metrics. There are two types of analysis + alerts: ● Default: is provided by the system and therefore generic for all location-based games. ● Custom Game-Specific: specific visualizations that requires programming to extract game-specific insigth ● Rule -based alerts. Simply enough to be created by instructors ⇒ Simple, but yet powerful, customization
- 11. Data Analysis and Visualization xAPI traces allows for: - a default set of analyses and visualizations - for different stakeholders (e.g. teachers, developers). Visualization for specific games can be developed based on game-specific LAMs. Correct/incorrect answers per question (alternative in xAPI-SG). Progress per player in each task / level (completable in xAPI-SG).
- 12. Real-time analytics: Alerts and Warnings ● Identify situations that may require teacher intervention ● Fully customizable alert and warning system for real-time teacher feedback 25/04/ Inactive learner: triggers when no traces received in #number of minutes (e.g. 2 minutes) High % incorrect answers: after a minimum amount of questions answered, if more than # % of the answers are wrong Students that need attention View for an specific student (name anonymized)
- 13. xAPI GLA in games authoring Previous game engine eAdventure (in Java) ● Helps to create educational point & click adventure games ● Platform updated to uAdventure (in Unity) ● Full integration of game learning analytics ● into uAdventure authoring tool ● uAdventure games with default analytics ● Include geolocalized games
- 14. uAdventure: geolocalized serious games Geolocalized default analytics visualization New Geolocalized game scenes and actions
- 16. Examples: First Aid - CPR validated game ● Collaboration with Centro de Tecnologias Educativas de Aragon, Spain ● Identify a cardiac arrest and teach how to do a cardiopulmonary resuscitation to middle and high school students ● Validated game, in 2011, 4 schools with 340 students Marchiori EJ, Ferrer G, Fernández-Manjón B, Povar Marco J, Suberviola González JF, Giménez Valverde A. Video-game instruction in basic life support maneuvers. Emergencias. 2012;24:433-7.
- 17. BEACONING Experiments: data collection 227 students from 12 to 17 years old Game rebuilt with uAdventure Each student completed: 1. Pre-test (multiple-choice questions) 2. Complete gameplay (xAPI traces) 3. Post-test (multiple-choice questions) 104 variables identified for each player.
- 18. Replicability of results: knowledge acquisition Original experiment with the game Original experiment, control group Current experiment Lower learning than in original experiment but still significative!
- 19. GLA for Improving SG evaluation Ideally: we would want to find a better evaluation method for SGs, avoiding pre-post experiments. High costs in time and effort. Our first approach: use data mining techniques to predict pre-test and post-test scores using the data tracked from in-game interactions. - To measure knowledge acquisition. - But it could also work to measure attitude change or awareness raised.
- 20. GLA data for Improving SG evaluation Use data mining techniques to predict pre-test and post- test scores using the data tracked from in-game interactions. 1. To avoid pre-test: - Determine the influence of previous knowledge in game results. 1. To avoid post-test: - Determine the capability of game interactions to predict post test results when combined with the pre test. - Compare the previous capability to that of game interactions on their own to predict post test results.
- 21. Improving SG evaluation To avoid pre-test: - Create prediction models of pre-test using only game data Score prediction: - As binary category pass / fail Prediction models used: - Naïve Bayes Method Precision Recall Naive Bayes 0.69 0.84 Results obtained:
- 22. Improving SG evaluation To avoid post-test: - Create prediction models of post-test score using pre-test + game data - Create prediction models of post-test score using only game data Score prediction: - As numeric value (from 1 to 15 in our game) - As binary category pass / fail Prediction models used: - Trees - Regression - Naïve Bayes to compare results
- 23. Improving SG evaluation To avoid post-test - summary of results obtained. -Prediction of post-test score (1 to 15) -Prediction of post-test pass / fail Worse predictions without pre-test data but still acceptable results. Method Pre- test ASE Regression trees Yes 4.92 No 5.68 Linear regression Yes 5.81 No 5.71 Method Pre-test Precision Recall Decision trees Yes 0.81 0.94 No 0.88 0.92 Logistic regression Yes 0.89 0.98 No 0.87 0.98 Naive Bayes Yes 0.92 0.89 No 0.89 0.90
- 24. New uses of games based on GLA - Avoiding pre-test: Games for evaluation - Avoiding post-test: Games for teaching and measure of learning With or without pre-test.
- 25. BEACONING GLA Case study: Downtown • Serious Game designed and develop to teach young people with Down Syndrome to move around the city using the subway • Evaluated with 51 people with cognitive dissabilities (mainly Down Syndrome) • 42 users with all data • 3h Gameplay/User • 300K analytics xAPI data (traces) to analyze
- 26. Case Study: Downtown ● From user requirements to a game design and its observables ● Know more about how and what is learn by people with Down Syndrome 26
- 28. Game Design and Analysis Workflow Individualized Learning Analysis Collective Learning Analysis …. Group 1 Group 2 Group 3 d1.a d1.n d2.a d3.a d2.n d3.n *d = Data collected during a game session User 1 User 2 User n User 1 User n User 3 User 2 User 5 User 4 User cognitive restrictions Formal Requirements Game & Learning Design Group of Observables xAPI 28 Ana Rus Cano, Alvaro Garcia-Tejedor, Baltasar Fernández-Manjón (2018): Using Game Learning Analytics for Validating the Design of a Learning Game for Adults with Intellectual Disabilities. The British Journal of Educational Technology (in press)
- 29. Hyp 1: Users prefer to identify themselves with the avatar ● REFUTED ● None of the users selected the avatar with Down features despite the trainers showed them the avatar and pointed that that character was Down. ● The majority of the users used the preconfigured character despite they were asked to customize the avatars at the beginning of the game session. ● We are not observing significative evidences in the users’ play patterns between those who customize the character and those who don’t, but it may be significative that the majority of the users that changed the avatar were Down. 29
- 30. Hyp: High-Functioning users do a better performance using the game ● To determine the cognitive skills and autonomy of the users we asked the trainers to complete a test about each student ● 6 intelectual dimensions were measured (5-point Likert scale) ○ General cognitive/intellectual ability ○ Language and communication ○ Memory acquisition ○ Attention and distractibility ○ Processing speed ○ Executive functioning ● Users were divided in two groups: Medium-Functioning (≤ 3 avg.) and High-Functioning (>3 avg.). ● MF = 19 (45.2%) HF = 23 (54.8%) 30
- 31. Hyp 2: High-Functioning users do a better performance using the game 31 Number of MF users that played each level Number of HF users that played each level
- 32. Hyp 2: High-Functioning users do a better performance using the game Average time completing levels for MF Average time completing levels for HF 0:31:21 0:37:02 0:33:16 EASY MEDIUM EXPERT 0:32:44 0:28:33 0:36:51 0:25:58 EASY MEDIUM HARD EXPERT
- 33. Hyp: ID users are engaged and motivated while learning with a videogame 33 0:01:30 0:01:03 0:00:590:00:59 0:00:50 0:01:00 0:00:360:00:38 0:00 0:17 0:35 0:52 1:09 1:26 1:44 1 2 3 4 5 6 7 8 • Inactivity times reduced in a 70,7% avg. from session #1 to session #8 • Positive and motivational learning environment (98,2% users show improvement and engagement performing the videogame tasks) Average inactivity time evolution avg time game session
- 34. Hyp: The game design of Downtown is effective as a learning tool 34 • 100% of the trainers agree that the use of Downtown would enhance the user learning adquisition (Perceived Usefulness) • 85,8% of the users were able to follow the right path (both LF and HF) • 50,8% of the wrong path occured during the first 30 min. of playing 0 20 40 60 80 100 120 140 160 180 1 2 3 4 5 6 Correct vs Incorrect Path per Game session (#correct stations vs #incorrect stations) cou nt game session
- 35. Cyberbullying: Conectado game Calvo-Morata, A. et al. Validation of a Cyberbullying Serious Game Using Game Learning Analytics (submitted for publication IEEE)
- 36. Educational desing ● Adventure game- sentiments and emotions are important ● Real situations familiar for students ● Events based on user decision making (but no agression options) ● Scenarios based on research about bullying and cyberbullying ● Different roles of bullying represented ● Designed to be used at classroom
- 37. Game mechanics Seminario eMadrid sobre Serious gaes 2017-02-24 37 New student in school Occurs during 5 days Minigames as “nightmares” Implications of the social networks
- 38. The experiment: initial validation With students from 3 schools( Madrid, Zaragoza, Teruel) 257 students 223 Valid pre-post and gameplays (121 males, 102 females)
- 39. Significant increase in the ciberbullying perception Wilcoxon paired test, p<0.001 5.72 6.38
- 40. ● 500+ players (1000+ expected by summer) ○ +380 students 12 to 17 years old ○ +120 Educational science students ○ +60 actual school and high school teachers ● Players from ○ Madrid ○ Murcia ○ Teruel ○ Zaragoza Conectado: current experiment current experiment
- 41. Teachers in training: Aplicability 85,5% consider the game is applicable and are ready to use it 13,7% consider that is applicable in their classroom and are willing to apply it 99% very useful for creating a controlled discussion 98% good representation of the problem, easy to be identified with the victim
- 42. Teachers results
- 43. Teachers: change cyberbullying perception 6.64 Significant increase of 0.57 points 6.07
- 44. Teachers: Aplicability 70 % consider the game is applicable and are ready to use it 26 % consider that is applicable in their classroom and are willing to apply it 98% very useful for creating a controlled discussion 100 % good representation of the problem, easy to be identified with the victim
- 45. Conclusions ● Game Learning Analytics has a great potential from the business, application and research perspective ○ Design, Evaluation and Deployment of Serious Games ● Still complex to implement GLA in SG ○ Increases the (already high) cost of the games ○ Requires expertise not always present in SME ● New standards specifications and open software development could greatly simplify GLA implementation and adoption 45
- 46. 46 Thank You! Gracias! ¿Questions? • Mail: [email protected] • Twitter: @BaltaFM • GScholar: https://scholar.google.es/citations?user=eNJxjcwAAAAJ&hl=en&oi=ao • ResearchGate: www.researchgate.net/profile/Baltasar_Fernandez-Manjon • Slideshare: http://www.slideshare.net/BaltasarFernandezManjon