IVE 2024 Short Course - Lecture13 - Neurotechnology for Enhanced Interaction in Immersive Environments
0 likes117 views
The document outlines the research on brain-computer interfaces (BCIs) and their application in immersive environments like augmented and virtual reality. It discusses various BCI paradigms, techniques for measuring brain activity, and the integration of BCIs in 3D immersive settings. The presentation also covers advancements in user interaction, prototype designs, and open challenges in the regulation of neurotechnologies.
![Definitions
9
Interface: noun [C] a situation, method, or place
where two things come together and have an
effect on each other. [Cambridge dictionary]
A BCI is a system that translates measures of brain
activity into commands or messages for an
interactive application
J W p w EW W p w “Brain-Computer Interfaces: principles and practice” Ox U v s y P ss 2012
F. Lotte, L. Bougrain, M. Clerc, "Electroencephalography (EEG)-based Brain-Computer Interfaces",
Wiley Encyclopedia on Electrical and Electronics Engineering, 2015](https://image.slidesharecdn.com/lecture13-bcivrlecturehakimsmh-240721164534-a3583410/85/IVE-2024-Short-Course-Lecture13-Neurotechnology-for-Enhanced-Interaction-in-Immersive-Environments-9-320.jpg)
![Motor Imagery
18
Penfield homonculus [Penfield54]
µ (~8-12 Hz) oscillations
β (~12-30 Hz) oscillations
Pfurtscheller & Neuper “M y b - p ”
Proceedings of the IEEE, 2001
1: Modulation
2: Feedback](https://image.slidesharecdn.com/lecture13-bcivrlecturehakimsmh-240721164534-a3583410/85/IVE-2024-Short-Course-Lecture13-Neurotechnology-for-Enhanced-Interaction-in-Immersive-Environments-18-320.jpg)
![P300: aka Aha! Signal
22
[Stephanie et al. 2017]
P300Speller
Farwell & Donchin “Talking off the top of your head: toward a mental prosthesis utilizing event-related brain
potentials” E p p y N p ys y 1988](https://image.slidesharecdn.com/lecture13-bcivrlecturehakimsmh-240721164534-a3583410/85/IVE-2024-Short-Course-Lecture13-Neurotechnology-for-Enhanced-Interaction-in-Immersive-Environments-22-320.jpg)
![Digitally created 3D immersive environment
28
[Milgram and Kishino 1994]](https://image.slidesharecdn.com/lecture13-bcivrlecturehakimsmh-240721164534-a3583410/85/IVE-2024-Short-Course-Lecture13-Neurotechnology-for-Enhanced-Interaction-in-Immersive-Environments-28-320.jpg)
IVE 2024 Short Course - Lecture13 - Neurotechnology for Enhanced Interaction in Immersive Environments
- 1. Neurotechnology for Enhanced Interaction in Immersive Environments Hakim Si-Mohammed Associate Professor Univ. Lille, CRIStAL Brain-Computer Interfaces team Contact: [email protected] University of South Australia 2024
- 2. A few words about myself 2 Computer Science Engineer Higher National School of Computer Engineering Supervision: Pr. Karima Benatchba, Pr. Yacine Challal International mobility (Academic) Graz University of Technology (3 months) Supervision : Pr. Reinhold Scherer Internation mobility (Industrial) Microsoft Research (3 months) Supervision : Dr. Andrew Wilson PhD in Computer Science (Defended 12/2019) Inria/INSA Rennes Supervision: Dr. Anatole Lécuyer, Dr. Ferran Argelaguet, Pr. Géry Casiez Associate Professor Univ. Lille June 2016 September 2018 June 2019 October 2016 September 2020
- 3. The Brain-Computer Interfaces team 3
- 4. Agenda 4
- 5. Research Topic 5 Brain-Computer Interfaces (BCIs) Immersive Environments (AR/VR)
- 7. What is NOT a BCI 7
- 8. Let’s play a small game 8
- 9. Definitions 9 Interface: noun [C] a situation, method, or place where two things come together and have an effect on each other. [Cambridge dictionary] A BCI is a system that translates measures of brain activity into commands or messages for an interactive application J W p w EW W p w “Brain-Computer Interfaces: principles and practice” Ox U v s y P ss 2012 F. Lotte, L. Bougrain, M. Clerc, "Electroencephalography (EEG)-based Brain-Computer Interfaces", Wiley Encyclopedia on Electrical and Electronics Engineering, 2015
- 10. How does a BCI actually work 10 M. Clerc, L. Bougrain F. “Brain-Computer Interfaces 1: Foundations and Methods", ISTE-Wiley, 2016
- 11. History of brain activity measurement 11
- 12. Types of brain activity measurement techniques 12
- 13. Some widely used brain activity measurement techniques 13
- 15. The notion of BCI paradigm 15 1: Modulation 2: Feedback 2: Modulation 3: Feedback 1: Stimulation 1: Monitoring 2: Feedback
- 16. Active BCI paradigms 16 1: Modulation 2: Feedback
- 17. Slow Cortical Potentials 17 1: Modulation 2: Feedback
- 18. Motor Imagery 18 Penfield homonculus [Penfield54] µ (~8-12 Hz) oscillations β (~12-30 Hz) oscillations Pfurtscheller & Neuper “M y b - p ” Proceedings of the IEEE, 2001 1: Modulation 2: Feedback
- 21. Steady-State Visual Evoked Potentials 21 Legeny et al. 2013 Zhu et al. 2010
- 22. P300: aka Aha! Signal 22 [Stephanie et al. 2017] P300Speller Farwell & Donchin “Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials” E p p y N p ys y 1988
- 23. Passive BCI paradigms 23 A passive BCI is a system estimating one or several mental states from the user – without any voluntary action from this user – to adapt a human-computer interaction accordingly Zander, T. & Kothe . “T w s p ss v b -computer interfaces: applying brain-computer interface technology to human- sys s ” J N E 2011
- 24. Error Related Potentials 24 Putze, F., Schünemann, M., Schultz, T., & Stuerzlinger, W. Automatic classification of auto- correction errors in predictive text entry based on EEG and context information. ACM ICMI 2017 Correcting Auto-correct!
- 25. Error Related Potentials … For implicit control 25 Zander, T. O., Krol, L. R., Birbaumer, N. P., & Gramann, K. Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity. Proceedings of the National Academy of Sciences, 2016
- 26. A few tools for BCI development http://openvibe.inria.fr) https://mne.tools/ 26
- 28. Digitally created 3D immersive environment 28 [Milgram and Kishino 1994]
- 29. BCIs and AR/VR Part III:
- 30. Combining BCI and VR/AR: A land of opportunities 30
- 31. The use of SSVEP in AR/VR Part IV: Legeny et al. 2013
- 32. Is it possible to exploit an SSVEP based BCI in AR? 32 • Si-Mohammed et al., Towards BCI-based Interfaces for Augmented Reality: Feasibility, Design and Evaluation, IEEE TVCG, 2018
- 33. How to integrate SSVEP stimulations in the AR environment? 33 • Si-Mohammed et al., Towards BCI-based Interfaces for Augmented Reality: Feasibility, Design and Evaluation, IEEE TVCG, 2018
- 34. Robot control system based on AR and SSVEP 34 Si-Mohammed et al., Towards BCI-based Interfaces for Augmented Reality: Feasibility, Design and Evaluation, IEEE TVCG, 2018
- 35. Integrated AR user interface 35 Collaboration : Si-Mohammed, et al. "Designing Functional Prototypes Combining BCI and AR for Home Automation." International Conference on Virtual Reality and Mixed Reality. Springer, Cham, 2022. F. Bouchenak
- 36. How to improve the user friendliness of the SSVEP stimulations? 36 Collaboration : A. Wilson C. Holz Si-Mohammed, et al. "On the Effect of Size and Contrast of the SSVEP Visual Stimuations on Classification Accuracy and User- Friendliness in Virtual Reality." Winter BCI Conference. IEEE, Seoul, 2023.
- 37. How to improve the user friendliness of the SSVEP stimulations? 37 Classification accuracy Subjective preference
- 38. What if we could get rid of the flickering stimulations? 38 Patent: • Using Real-World Inspired Animations to Create Pleasant SSVEP-Based Brain-Computer Interfaces, US Patent 407846-US-NP Collaboration : A. Wilson
- 39. Towards gaze independent BCIs 39 Van Den Kerchove, et al. "Correcting for ERP latency jitter improves gaze-independent BCI decoding." Journal of Neural Engineering, 2024 A. Van Den Kerchove
- 40. Characterizing User Experience in VR using EEG Part V:
- 41. Anomaly detection in VR using Error Related Potentials 41 Collaboration : R. Scherer C. Lopes-Dias Si-Mohammed, Hakim, et al. "Detecting system errors in virtual reality using EEG through error-related potentials." 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2020.
- 42. EEG-based detection of cybersickness 42 • GENESIS: LeveraGing nEuromarkers for Next-gEneration immerSIve Systems GENESIS
- 43. Identification of markers of Vection in VR using EEG 43 M. Naud
- 44. Conclusion and perspectives Part III:
- 45. Towards the regulation of neurotechnologies 45
- 47. Summary of the presentation 47
- 48. Thank you for your attention Hakim Si-Mohammed Associate Professor Univ. Lille, CRIStAL Brain-Computer Interfaces team Contact: [email protected] University of South Australia, 2024