Human Machine Symbiosis Using Multimodal Trace Data for the Design, Development, Testing and Implementation of Human Digital Twins

Human-machine symbiosis using multimodal data for the development of human Digital Twins Professor Roger Azevedo and Dr. Megan Wiedbusch, School of Modeling Simulation and Training, the University of Central Florida (UCF) Dr. Azevedo is a Professor in the School of Modeling Simulation and Training at the University of Central Florida. He is also an affiliated faculty in the Departments of Computer Science and Internal Medicine at the University of Central Florida and the lead scientist for the Learning Sciences Faculty Cluster Initiative. His main research area includes examining the role of cognitive, metacognitive, affective, and motivational self-regulatory processes during learning with advanced learning technologies (e.g., intelligent tutoring systems, hypermedia, multimedia, simulations, serious games, immersive virtual learning environments). His overarching research goal is to understand the complex interactions between humans and intelligent learning systems by using interdisciplinary methods to measure cognitive, metacognitive, emotional, motivational, and social processes and their impact on learning, performance, and transfer. To accomplish this goal, he conducts laboratory, classroom, and in-situ (e.g., medical simulator) studies and collects multi-channel data to develop models of human-computer interaction; examines the nature of temporally unfolding self- and other-regulatory processes (e.g., human-human and human-artificial agents); and designs intelligent learning and training systems to detect, track, model, and foster learners, teachers, and trainers’ self-regulatory processes. A person with blonde hair wearing glasses Description automatically generatedDr. Wiedbusch is a postdoctoral researcher at the School of Modeling, Simulation, and Training at the University of Central Florida (UCF) with Dr. Roger Azevedo. Her research is focused on the measurement of the dynamics of metacognition and engagement using traditional (i.e., self-reports) and unobtrusive multimodal (e.g., eye tracking, facial expressions, log files) methodological and analytical approaches across contexts (e.g., health care, K-12 education, teacher training) and learning environments (e.g., VR, simulations, ITS, and GBLEs). She conducts laboratory, classroom, and in-situ studies to model human (meta)cognition and behavior during complex learning to inform the design of human-centered intelligent learning and training technologies. Abstract: This presentation explores the paradigm of human-machine symbiosis by leveraging multimodal data (e.g., eye movements, concurrent verbalizations, physiological data) to create human Digital Twins within the context of Human-Machine Systems and Digital Twin Technologies. As technology advances, integrating various data modalities such as physiological, behavioral, psychological, and environmental becomes pivotal in enhancing the fidelity and utility of human Digital Twins. The proposed framework establishes a symbiotic relationship between humans and machines, aiming to replicate and simulate human characteristics and functionalities (e.g., reasoning, problem-solving, learning) in digital environments. This integration involves real-time monitoring and analysis of diverse data sources, including biometric data, motion patterns, cognitive states, metacognitive monitoring processes, and contextual variables. Machine learning algorithms are crucial in deciphering intricate patterns and fostering a deeper understanding of individual behaviors and responses to external stimuli. The implications of this research extend to applications in healthcare, personalized medicine, military, K-16, and human-centered design. Human Digital Twins can serve as predictive models for learning outcomes and self-regulatory skills development, enabling proactive, intelligent, individualized interventions. Furthermore, in design and engineering, the development of highly responsive and adaptive systems can be realized through the insights gained from human Digital Twins. This talk contributes to the ongoing discourse on Human-Machine Systems and Digital Twin Technologies, presenting a comprehensive framework that advances the concept of symbiotic interaction, pushing the boundaries of technological integration to improve human well-being and system performance.