AR/VR and Human Comfort Index for Human Robot Collaboration in Industrial Manufacturing

AR/VR and Human Comfort Index for Human Robot Collaboration in Industrial Manufacturing Dr. Ferat Sahin, Department of Electrical and Microelectronics Engineering, Rochester Institute of Technology Organizing Committee – IEEE SMC 2023Dr. Ferat Sahin received his M.Sc. and Ph.D. degrees from Virginia Polytechnic Institute and State University. In September 2000, he joined Rochester Institute of Technology, where he is a Professor and Department Head of Electrical and Microelectronics Engineering department. He is also the director of Multi Agent Bio-Robotics Laboratory and Century Mold Collaborative Robotics Laboratory at RIT. His current research interests are Integration of Collaborative Robots in Manufacturing, Human-Robot Collaboration, Deep Learning Approaches for Grasping, Digital Twin, System of Systems Engineering, Machine Learning, Biological Signal Processing, Fault Analysis and Systemic Health Evaluation. Dr. Sahin has over 170 conference and journal publications in these research fields. In addition, he is also the co-author of two books: “Experimental and Practical Robotics” and “Intelligent Control Systems with an Introduction to System of Systems Engineering” by CRC Press. Dr. Sahin has been very actively serving in his scientific community through IEEE. He has served as in several capacities such as technical organizer, reviewer, referee, associate editor and deputy editor for leading journals and conferences in both the IEEE and other organizations. He serves as an Associate Editor for IEEE SMCS: Systems and SMCS Magazine. He is a member of the IEEE Systems, Man, and Cybernetics Society, Robotics and Automation Society, and Computational Intelligence Society. Currently, he is a member of Board of Governors and VP Finance of IEEE SMC Society. He has served as the Student Activities chair for the IEEE SMC Society in 2001, 2002, and 2003. He was the Secretary of the IEEE SMC Society from 2003 to 2006. Dr. Sahin received an “Outstanding Contribution Award” for his service as the SMC Society Secretary in 2007 and 2015. He also served as the Treasurer of the IEEE SMC Society in 2011, 2012, and 2013. Dr. Sahin has been heavily involved with technical conference organization. He has held many committee positions such as publication chair, finance chair, technical committee chair, and general chair of several IEEE sponsored international conferences. He has been mainly involved with two conferences: IEEE SMC International Conference on System of Systems Engineering (SOSE 2007-2023) and IEEE Annual Conference on Systems, Man, and Cybernetics (SMC). Currently, he is the general chair for 2024 International Conference on System of Systems Engineering and was the general co-chair for IEEE SMC 2023. Abstract: Collaborative robots are becoming very common in manufacturing industry because of their ability to work with humans safely without a cage around them. In the event of a collision collaborative robots are guaranteed not to jeopardize the safety of the human. However, a collision with a collaborative robot still requires the robot to stop completely and be started by an engineer which can cause significant decrease in throughput of the manufacturing line. Thus, collaborative robots need to avoid collision and maintain the highest throughput. Speed and separation monitoring (SSM) approaches are researched and developed to minimize collision and robot downtimes. There are also dynamic speed and separation monitoring approaches where the robot’s and human’s speed and positions are factored in for more optimized SSM. However, the SSM approaches only assess the position and speed of the human but does not care about human’s physical and mental state while working with a robot. If a person is not comfortable with or scared of a robot, there performance can degrade and resit to collaboration with the robot. We developed approaches where human comfort is assessed and factored into dynamic SSM approach to control the robot’s motion to ease human discomfort. We measure human comfortability through physiological signals such as pupil size, heart rate, heart rate variability and galvanic skin response to determine their emotional state based on circumflex model. In addition, we provide the human an AR set (Microsoft HoloLens II) to increase their situational awareness and possibly help with their comfort level. We have run four human subject test with regards to dynamic SSM, comfortability index estimation, human emotion estimation through physiological data and face image, and situational awareness assessment through AR headset. Our experimental results shows that including mental and emotional state (comfort) and situational awareness through AR can improve their performance and the throughput of the manufacturing line.