Learning Interactions Across Spaces: a Framework for Contextualised Multimodal Observations

Editor's Notes

• #4: Learning processes increasingly happen across spaces To analyse educational practices we need data Data can be collected with different human-mediated and automated means Alignment of not only two learning contexts but also paradigms of research
• #5: The context and problems detected in the communities around this PhD lead to the main research question and objectives, as well as the contributions that emerged and the different evaluations they went through. THE MAIN RESEARCH OBJECTIVE IS: ALIGNMENT OF … To achieve my objectives I have taken several different steps across the thesis.
• #6: THIS PHD THESIS IS BASED ON SEVERAL FIELDS AND RESEARCH LINES THAT CREATED THE CONTEXT OF THE RESEARCH AND ITS CONTRIBUTIONS The field of technology enhanced learning and more specifically, blended learning is the broad context of this research. It is important to analyse educational processes in holistic manner, which involves two learning contexts and spaces and the use of mixed data sources Problem: teaching and learning processes that happen increasingly in hybrid spaces need mixed data sources to analyse these processes Problem: gathering evidence with different methods and sources of data is difficult from such blended context
• #7: There is a need for different, mixed sources of data in educational research and practice, especially in TEL settings (Mor 2015) Traditional (observations, surveys, interviews etc) or modern data collection methods (logs, sensors etc) Which Four paradigm view suggests that on top of qualitative, quantitatve and mixed methods there is new, analytics based paradigm and ome considerations for automatically collected data may include: context of data that might be unknown to researchers Or the researcher might use data already collected Emergent epistemology and ontology
• #8: Learning interaction definitions broadly describe emerging processes that unfold through (inter)actions between actors and objects/artefacts/resources. According to some authors the analysis of interactions shall happen in hierarchical manner and include different sources of data from distributed settings Problem: there is a need for collecting data on learning interactions across spaces Problem: there is a need for the definition of unit of analysis and technological infrastructure for across spaces learning interaction data collection
• #9: Unit of analysis is the “entity of analysis” — an analytical unit that conclusions are based on Consists of subjects and objects and action Justification for the choice is determined by the need for Interaction analysis in the online and distributed settings.
• #10: LA relies on data aggregated through automated means to track learning processes MMLA are modern data collection methods, that can add more data sources On the other hand, MMLA can only capture observable attributes (Di Mitri et al., 2018) In MMLA human inference is often used to make sense of the data or to label learning constructs Classroom Observations are highly contextually and dan help contextualisation Alignment between them, how they enrich eachother and how they complement each other Aside from this theory driven approaches or the us of LD can further contextualise the data collection and analysis Problem: there is a need to address the “street-light effect” often present in LA for meaningful analytics Problem: there is a need for guided collection, contextualization and sensemaking of learning interaction data in authentic contexts 
• #12: Unit of analysis express by the event can be recorded as xAPI statements
• #13: the overarching research is
• #14: TEL, blended learning/teaching and learning across spaces: 1.1. Problem: teaching and learning processes that happen increasingly in hybrid spaces need mixed data sources to analyse these processes 1.2. Problem: gathering evidence with different methods and sources of data is difficult from such blended context Educational research and practice in TEL: 2.1. Problem: the methodological dichotomy between modern and traditional research paradigms makes it difficult to align data collection methods Learning interactions across spaces (automated and human-mediated data-collection): 3.1. Problem: there is a need for collecting data on learning interactions across spaces 3.2. Problem: there is a need for the definition of unit of analysis and techno logical infrastructure for across spaces learning interaction data collection 4. MMLA and contextualization of automated data 4.1. Problem: there is a need to address the “street-light effect” often present in LA for meaningful analytics 4.2. Problem: there is a need for guided collection, contextualization and sensemaking of learning interaction data in authentic contexts 
• #15: This PhD thesis is based on design tradition but on specific, RBD methodology where ”the product should first communicate the theoretical findings made during the design process. Even if the process is developing a new theory or concept, the design or product should communicate these in an innovative, sophisticated and clear way. The design can be an extensive combination of process and system descriptions as well as collections of tools and artifacts. Design, is the end. In comparison, for design-based research, design is a means, not an end. In design-based research design is an instrument used in the research» Research-based design consisting of 4 stages where these phases are not detached from each other and go back and forth between stages. For instance, I have returned to the contextual inquiry many times. Because the thesis is rooted in theory and RP identified, the contextual inquiry was heavily theoretical, eliciting design challenges from the literature, validating concepts throughout. Contextual inquiry: The outcome of this phase was the development of the mental prototype of the app and the first version of the model that communicated main methodological considerations of the research (I, II, III) Participatory design a design session and scenario-based research where the paper first version of the model (IV) and corresponding conceptual design and a paper prototype for data collection instrument have been validated (V). The outcomes of this phase are validated paper prototypes of Observata and the model. + 4. Modelling, towards evaluation and producing software as a hypothesis: Further examined the alignment between LD and CO and defined the need for alignment, conceptualizing modern (automated) and traditional data collection methods and identified methodological considerations, needed infrastructure and challenges (VI), evaluating applicability of the model in authentic settings through data collected with Observata (VII). The final outcome of this phase is the software prototype as a hypothesis as indicated by the RBD research methodology . I have developed a conceptual design of the app based on the model. Validated throughout and evaluated in one context in the end of the research. Different methods for data collection and analysis have been applied throughout the research.  
• #16: Philosophical/research approach framing the purpose the purpose of the LA study; Theoretical dimension - the educational theory and the pedagogical background that sustains the learning scenario; Technological context - the technological and architectural aspects that condition the data gathering and integration of multiple and heterogeneous data sources; In action Step 1. Be aware of the elements that belong to the learning context. Step 2. Define the areas of focus, Step 3. Collect observable events. Step 4. Analyze and interpret the results.
• #18: The proof of concept papers contributed to the first version of the model and definition of requirements and a first version of the model and the mental prototype (conceptual design) of the app Observata. . Based on the results of the exploratory case study (with 12 lessons observed involving 2 coders) it proved feasible to collect xAPI statements from physical settings by coding learning events.. The lessons learnt from the case study was that for systematic aggregation of xAPI events not only a new approach, but also a technological solution was needed– a new classroom observation tool had to be developed.
• #19: Context: contextual information to guides the data collection and analysis process (Context and pedagogical design are reflected in the predefined LD, which is important for both – to guide data collection or to analyse/make sense of the data. LD should be extracted from machine-interpretable context that contains information on actors, activities, resources and objects (artefacts). Alternatively, lesson structure also can be inferred through observations Data collection from enactment: to enable systematic data collection, codes are predefined, while systematic data from logs are automatically collected. These codes can be pedagogy, theory-driven or action-driven (or both at the same time, since several code-sets can be used for codification). MMLA data: by observing lesson enactment, the data is collected by coding learning events through the codification of user actions and aggregation of xAPI statement. Then observations and logs are put together in order to enable automated or semi-automated analysis.
• #21: In this phase, I have returned to the contextual inquiry phase to re-examine initial concepts in the proposal. The study entailed developing personas (scenario-based research) for the possible stakeholders and the users of the app. The study involved 6 people but 4 were the envisioned personas, potential/hypothetical users of the app. Head teacher was the primary persona, a supervisor of intern teachers, where one teacher was the experienced teacher with inquiry motivations and another one was an intern. Other stakeholders were not envisioned as the users of the app/model, so they were not involved. The findings of the validation showed that the participants representing the main stakeholders (primary - supervisor of intern teachers, secondary – intern teacher, edutech startup head, teacher and a researcher) have accepted the model. important suggestions from this session was the use of open coding schemes and adding qualitative data collection possibilities (field notes/photos). through the use of Observata tool we can identify, code and combine LA-compliant observation data. Findings indicated to the exactment about the innovation but at the same time, indicated to the expected challenges of adoption in authentic scenarios, which can be explained by the complexity of the proposal Lessons learnt and suggestions informed the development of the app and refinmenet of the model
• #22: I envision the taxonomy as a classification of research designs in MMLA study. The taxonomy stems from the previous research and different cases as indicated in scenario-based research and conceptual design of the app, where different protocols and documentations are considered, according to different purposes and stakeholders. Ideal - Systematic documentation and data collection: In the most desirable case, the learning design (including actors, roles, resources, activities, timeline, and learning objectives) is set up-front and documented in an authoring tool. LD is automated. Authentic (baseline) - Non-systematic documentation but systematic data collection: a compromise between the limitations of authentic settings but still rich in terms of data. LD is not automated. Limited MMLA Limited - Non-systematic documentation or data collection: Data collection happens non-systematically. As in the previous case, no information about the learning design is available (i.e., actors are not known). Non systematic, not automated
• #23: Here I have examined LD and CO are connected which entails modern and human-driven classroom observations. The main idea was to understand to what extent LD contextualises data collection and analysis, for what aims and what are further methodological considerations. The outcomes of this paper inspired the Context-Aware MMLA taxonomy and the further understanding of the model and methodological considerations Based on the results of the literature review, there is a synergetic relationship between CO and LD at data gathering and analysis stage by LD informing CO at data gathering, analysis or both stages. Observations can inform the LD as an artefact or the process of designing for learning; In some cases, CO can contribute to theory or the field of LD in general;. PhD the benefits of this synergy are unidirectional i.e., reflects on how CO can benefit from LD. However, the introduction of LD due to different technological or adoption issues makes it difficult to adopt such contextualization effort widely. The synergy can be used for classroom orchestration and teacher professional development. However, based on the review we have also elicited some challenges in order to make use of this synergies: lack of explicit designs, lack of standards and need for conceptual and technological tools. Starting from the learning design, often this information is not explicit and formalized, therefore, in order to make use of the LD, practitioners are requested to create it, raising the workload. To enable inquiry processes it is suggested that MMLA solutions could contribute to reducing the burden by inferring the lesson plan and by automatically gathering parts of the observation through automated means.
• #26: In this research my aim was to understand the applicability of the model and understand the value of classroom observations in the dataset. The aim of the final evaluation was not to test Observata, but to evaluate the model behind it, using Observata as the main communication channel for research findingsAccording to the participants, systematic observations allow for quantitative analysis of data while still offering rich context derived through non-automated means. In authentic scenarios: HMO can contextualize data analysis and sense-making, where due to adoption or technological challenges LD is not available. Qualitative data such as teacher reflections, field notes etc creates an additional layer of contextual information, some of it can be gathered through Observata. This reinforces the need for a qualitative dimension of data sources. In ideal use of HMO can further enhance contextual information, where it is technically possible (actors are identified and cross-referenced across spaces, LD is provided). Several layers of contextual information make it possible to have context from: first, predefined LD, second - observed lesson structure, and the third - systematic observations. Data can be collected qualitatively (photos or fieldnotes) but can be structured later on using Observata post-editing feature of learning events. In all the scenarios, both, HMO and AO complement each other through different taxonomies and semantics, HMO provides contextual information for data analysis and can introduce theoretical (learning) constructs in the data sets, which is an important factor for data-analysis and sensemaking. Due to data analysis and sense-making considerations, the use of systematic observations is reinforced, so data can be gathered in qualitative form and later coded, or recoded to increase reliability. Multimodal Dashboards are to be developed in a way that they further contribute to data analysis and sensemaking, provided that the inclusion of qualitative data-sources is also important. According to participants, theoretical constructs can be introduced through CO: Sometimes theory in LD is not explicit, at least not in the LePlanner. Or sometimes there is no predefined LD present.
• #28: Used in a large-scale digital learning resources development project (Digiõpevaramu) CEITER* project LA toolkit for Estonian schools to analyse learning processes on a classroom level For evidence-based decision making in the educational innovation process and is integrated into the EDULABS** method. Observata has been deployed as an in/pre-service teacher education tool at Tallinn University. By other researchers from an institution outside Estonia to observe students affect and behaviour***
• #30: Concept of validity is different from quantitative research validity (Cohen et al., 2018) In qualitative settings the most important factor is the naturalistic approach (real-world, context-specific) (Golafshani, 2003) in DBR (and in RBD, as its adaptation), resulting [design] principles are perceived as having greater external validity than those developed in laboratory settings (Wang & Hannafin, 2004). The internal validity, in qualitative studies it lies is in the rigorous, truthful description of the research procedures and extensive data interpretation from different angles (Cohen et al., 2018).
• #33: The product design phase resulting in a hypothesis that needs to be further revisited and more participatory design sessions are needed for further versions. The final stage of the research identified some challenges: this entails data integration and architectures, l as well as data quality and reliability issues. to the ideal case, in the future it would be recommended to include more systematically collected data. Also, for further contextualization of the MMLA data for analysis some methodological, technological and research needs are to be addressed. Observata will need to be further developed according to the findings of the final co-design study. In addition, aspects such as data reliability and validity as well as data privacy issues should be solved in future both at the technological and methodological level by integrating inter-rater reliability solu tions. Observata will need to be further developed according to the findings of the final co-design study.