Assessing cognitive presence using automated learning analytics methods
2 likes327 views
This doctoral thesis presentation discusses assessing cognitive presence using automated learning analytics methods. The thesis aims to develop models for assessing cognitive presence through analysis of online discussion transcripts and interaction log data. It seeks to address these goals in both traditional online courses and newer modalities like MOOCs. The thesis is comprised of several chapters that aim to develop assessment models, operationalize cognitive presence, create analytics systems, validate models in MOOCs, and identify study strategies in relation to cognitive presence. The overall goal is to advance understanding and assessment of cognitive presence using learning analytics.
More Related Content
What's hot (20)
Similar to Assessing cognitive presence using automated learning analytics methods (20)
More from Vitomir Kovanovic (20)
Recently uploaded (20)
Assessing cognitive presence using automated learning analytics methods
- 1. ASSESSING COGNITIVE PRESENCE USING AUTOMATED LEARNING ANALYTICS METHODS Vitomir Kovanović School of Informatics The University of Edinburgh Doctoral Thesis Presentation 30 June 2017
- 2. 2 Basic idea Use learning analytics to provide assessment (and understanding) of cognitive presence
- 3. 3 Community of Inquiry framework (Garrison, Anderson, and Archer, 1999)
- 4. 4 Cognitive presence (Garrison, Anderson, and Archer, 2001)
- 5. 5 General approach Method Use automated learning analytics methods for assessing cognitive presence Context Examine cognitive preesnce in “traditional” online courses and “new” modalities, such as MOOCs Data Utilize course discussion transcripts and usage log (trace) data
- 6. 6 Driving principles Theory-driven The use of learning analytics must be aligned with the existing educational theories. Data-informed Use data mining techniques to process large amounts of student-generated data to understand how student learn in online settings. Practical Develop automated learning analytics tools that can be used to assess student learning and improve research in online and distance education.
- 7. 7 Five research questions RQ1: Assessment model. How to develop theoretically sound model of cognitive presence assessment, that is detailed enough to provide practical guidance for analytics development, and also flexible enough to accommodate assessment in different contexts?
- 8. 8 Five research questions RQ1: Assessment model. How to develop theoretically sound model of cognitive presence assessment, that is detailed enough to provide practical guidance for analytics development, and also flexible enough to accommodate assessment in different contexts? RQ2: Operationalization. How are phases of cognitive presence expressed in discussion transcripts? What quantitative measures extracted from discussion transcripts are reflective of cognitive presence phases?
- 9. 9 Five research questions RQ1: Assessment model. How to develop theoretically sound model of cognitive presence assessment, that is detailed enough to provide practical guidance for analytics development, and also flexible enough to accommodate assessment in different contexts? RQ2: Operationalization. How are phases of cognitive presence expressed in discussion transcripts? What quantitative measures extracted from discussion transcripts are reflective of cognitive presence phases? RQ3: Analytics development. To what extent can we use learning analytics to assess cognitive presence? How accurately can an analytics system gauge students’ cognitive presence through the analysis of discussion transcripts and interaction log data?
- 10. 10 Five research questions RQ4: Analytics within MOOCs. Can we enable assessment of cognitive presence within newer modalities of online learning, such as MOOCs? How does cognitive presence development within MOOCs differ from traditional online courses?
- 11. 11 Five research questions RQ4: Analytics within MOOCs. Can we enable assessment of cognitive presence within newer modalities of online learning, such as MOOCs? How does cognitive presence development within MOOCs differ from traditional online courses? RQ5: CoI model within MOOCs. Can we use the CoI model to assess student learning experience in MOOCs? Are there any important differences with respect to three presences between MOOCs and traditional online courses? Are there any important differences regarding cognitive presence development between MOOC and traditional online courses?
- 12. 12 Table of contents Ch. 1. Introduction Ch. 2. Cognitive presence assessment model Ch. 3. Assessing cognitive presence from online discussion transcripts Ch. 4. Assessing cognitive presence using trace data Ch. 5. Methodological challenges with trace data usage for LA models Ch. 6. Validation of the CoI model within MOOC context Ch. 7. Assessing cognitive presence within MOOC courses Ch. 8. Conclusions and future directions
- 14. 14 Thesis structure & RQs Chapter Research Questions # Title RQ1 RQ2 RQ3 RQ4 RQ5 2 Cognitive presence assessment model 3 Assessing cognitive presence from online discussion transcripts 4 Assessing cognitive presence using trace data 5 Methodological challenges with trace data usage for learning analytics models 6 Validation of the Community of Inquiry model within MOOC context 7 Assessing cognitive presence within MOOC courses
- 15. 15 THESIS OVERVIEW Overview of thesis chapters and their contributions
- 16. 16 Table of contents Ch. 1. Introduction Ch. 2. Cognitive presence assessment model Ch. 3. Assessing cognitive presence from online discussion transcripts Ch. 4. Assessing cognitive presence using trace data Ch. 5. Methodological challenges with trace data usage for LA models Ch. 6. Validation of the CoI model within MOOC context Ch. 7. Assessing cognitive presence within MOOC courses Ch. 8. Conclusions and future directions
- 17. 17 Chapter 1: Introduction Overview of the thesis • Research questions & goals • Methodology • Thesis structure & overview Research contributions & output
- 18. 18 Chapter 2: Cognitive presence assessment model • Developed assessment model for cognitive presence using learning analytics methods. • Based on evidence-centered design (ECD) model. • Conceptual framework for designing, implementing, and customizing analytics in different contexts. • Shows how developed analytics provide a coherent and comprehensive assessment of cognitive presence.
- 19. 19 • Developed a discussion message classification system which can be used to classify student discussion messages based on their level of cognitive presence. • Obtained inter-rater reliability of Cohen’s κ = 0.63, which is significantly higher than for the existing classification systems (McKlin, 2004; Corich, Hunt, & Hunt, 2012). • Identified data-informed indicators of cognitive presence phases that provide insights into the nature of cognitive presence and inquiry-based learning. Ch. 3: Assessing cognitive presence from discussions
- 20. 20 Chapter 4: Assessing cognitive presence using trace data • Developed a method for the identification of students’ learning strategies in online learning environments. • Identified different study strategies and examined their association with the differences in cognitive presence development. • Results reveal several ways in which students in online courses can be successful and emphasize the need for specific interventions to cater to the needs of students with different study strategies.
- 21. 21 Chapter 5: Methodological challenges with trace data • Overview of time-on-task (ToT) estimation methods in the literature • Examined the effects of ToT estimation on statistical results. In particular, the relationship between ToT and course outcomes (including cognitive presence) • Replicated the analysis within blended courses, to examine the role of learning modalities. • Significant portion of variance (~15%) is explained by the ToT estimation method. • This suggest significant methodological challenges with ToT use as a measure of student engagement.
- 22. 22 Chapter 6: Validation of the CoI model within the MOOC context • Examined the validity of the CoI model within MOOCs. In particular, we examined the factor structure and reliability of the CoI survey instrument (Arbaugh et al., 2008) • We examined original 3-factor and optimal factor structures. • Results confirm the reliability and validity of the CoI instrument for assessing MOOC learning experience. • Results indicate significant differences regarding student perceptions of the three CoI presences.
- 23. 23 Chapter 7: Assessing cognitive presence within MOOC courses • Developed the method for study strategy identification within MOOCs. Draws heavily on the work from the Chapter four. • Identified three different study strategies within MOOC and examined their relationship with the perceived levels of three CoI presences. • Also investigated the relationship between study strategies and survey responses. • Results revealed statistically significant differences in levels of cognitive presence and commitment to learning, enrollment goals and motivation, approaches to learning, and goal orientation.
- 24. 24 Chapter 8: Conclusions and future directions Summary of contributions • RQ1: Assessment model • Developed a model for cognitive presence assessment based on ECD framework for assessment design • RQ2: Operationalization • Data-informed operationalization of the phases of cognitive presence • RQ3: Analytics development • Developed two different learning analytics models for assessing student cognitive presence in online learning. • The first model provides classification of student discussion messages, while the other model provides complementary insights into the students’ study strategies • RQ4: Analytics within MOOCs • Developed a method for the identification of students’ study strategies • Examined the differences between study strategies regarding perceived levels of three CoI presences • RQ5: CoI model within MOOCs • Examined the use of CoI model within MOOC courses for assessing students’ learning experience • Examined the validity and reliability of the CoI survey instrument through EFA
- 25. 25 Chapter 8: Conclusions and future directions Future work: • Role of course context • On classification accuracy • On identified study strategies • Integration of the analytics into the learning platform • How the developed analytics will be utilized by the practitioners • How the analytics results will be used to provide student and instructor feedback • Development of the graphical interface for the analytics systems • Examine the relationships between CoI presences within MOOCs • Structural-equation model of relationship between social, teaching, and cognitive presence. • Extend analytics to teaching and social presence • Additional classification systems • Explore the use of sentiment analysis
- 27. 27 References Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. The Internet and Higher Education, 11(3–4), 133–136. doi:10.1016/j.iheduc.2008.06.003 Corich, S., Hunt, K., & Hunt, L. (2012). Computerised Content Analysis for Measuring Critical Thinking within Discussion Forums. Journal of E-Learning and Knowledge Society, 2(1), 47-- 60. Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education. The Internet and Higher Education, 2(2–3), 87–105. doi:10.1016/S1096- 7516(00)00016-6 Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical Thinking, Cognitive Presence, and Computer Conferencing in Distance Education. American Journal of Distance Education, 15(1), 7–23. doi:10.1080/08923640109527071 McKlin, T. (2004). Analyzing Cognitive Presence in Online Courses Using an Artificial Neural Network. Georgia State University, College of Education, Atlanta, GA, United States.