Publicação
Dynamically Predicting Content Comprehension Through Intelligent Wearables and Biofeedback Sensors (TellBack)
| Resumo: | This thesis introduces an innovative approach that utilizes wearable biosensors, low-cost eye-tracking technology, and machine learning to assess user comprehension and engagement at local levels of content. It addresses the significant challenge of identifying “when and where” users’ comprehension difficulties occur on screens identified by changes in Autonomic Nervous System (ANS) activity, such as heart rate variability (HRV) and Electrodermal Activity (EDA). The core concept, which we coined "TellBack," combines non-intrusive biosensors with eye tracking to assess cognitive load using ANS modulations and localize where comprehension challenges occur within digital content. This approach demonstrates promising classification accuracy for predicting difficulties and low engagement perceived by individuals in content regions compared to existing methods. The work further extends the core concept to two distinct application domains. The first application, "iReview," is developed for the software industry and aims to enhance code inspection (review) by evaluating the reviewer's understanding and engagement with the code under review. Using machine learning, we estimate the surrogate of cognitive load and classify code regions based on code comprehension, providing insights into reviewers’ performance and potentially improving the quality of code reviews by identifying code regions that were not well reviewed. The second application, "iMind," focuses on improving foreign language comprehension in educational and training settings. In this application, the goal is to predict comprehension difficulties within English texts, making it possible to provide personalized and contextualized help for nonnative speakers at local levels of text (e.g., paragraphs), by integrating different cognitive load manifestations encoded in the ANS using wearables and contextual information, "iMind" effectively identifies "hotspots" in texts that present comprehension challenges, thereby opening the possibility to develop cognitively aware interfaces for learning purposes. |
|---|---|
| Autores principais: | Hijazi, Haytham Wael Ismail |
| Assunto: | Artificial Intelligence Biofeedback Sensors Cognitive Load Content Comprehension Software Engineering Inteligência Artificial Sensores de Biofeedback Carga Cognitiva Compreensão de Conteúdo Engenharia de Software |
| Ano: | 2024 |
| País: | Portugal |
| Tipo de documento: | tese de doutoramento |
| Tipo de acesso: | acesso embargado |
| Instituição associada: | Universidade de Coimbra |
| Idioma: | inglês |
| Origem: | Estudo Geral - Universidade de Coimbra |
| Resumo: | This thesis introduces an innovative approach that utilizes wearable biosensors, low-cost eye-tracking technology, and machine learning to assess user comprehension and engagement at local levels of content. It addresses the significant challenge of identifying “when and where” users’ comprehension difficulties occur on screens identified by changes in Autonomic Nervous System (ANS) activity, such as heart rate variability (HRV) and Electrodermal Activity (EDA). The core concept, which we coined "TellBack," combines non-intrusive biosensors with eye tracking to assess cognitive load using ANS modulations and localize where comprehension challenges occur within digital content. This approach demonstrates promising classification accuracy for predicting difficulties and low engagement perceived by individuals in content regions compared to existing methods. The work further extends the core concept to two distinct application domains. The first application, "iReview," is developed for the software industry and aims to enhance code inspection (review) by evaluating the reviewer's understanding and engagement with the code under review. Using machine learning, we estimate the surrogate of cognitive load and classify code regions based on code comprehension, providing insights into reviewers’ performance and potentially improving the quality of code reviews by identifying code regions that were not well reviewed. The second application, "iMind," focuses on improving foreign language comprehension in educational and training settings. In this application, the goal is to predict comprehension difficulties within English texts, making it possible to provide personalized and contextualized help for nonnative speakers at local levels of text (e.g., paragraphs), by integrating different cognitive load manifestations encoded in the ANS using wearables and contextual information, "iMind" effectively identifies "hotspots" in texts that present comprehension challenges, thereby opening the possibility to develop cognitively aware interfaces for learning purposes. |
|---|