Phd in ENGINEERING IN COMPUTER SCIENCE

Thesis title: Multimodal Communication for Enhancing Human-Robot Interaction: Virtual Simulations to Real Robots

Human-robot interaction (HRI) is a rapidly evolving domain focused on the interaction between humans and robots, exploring robotic systems' design, functionality, and social implications in various environments. Virtual Reality (VR) has emerged as a valuable tool for evaluating HRI solutions before real-world deployment, ensuring safety and scalability. The main objective of this thesis is to present a multimodal interaction framework integrating speech and gesture recognition to enhance collaboration between humans and robots in precision agriculture, particularly in table-grape vineyards under the CANOPIES project, as well as in broader contexts of indoor and outdoor logistics. In collaborative robotics, where human-robot collaboration (HRC) is essential, multimodal communication between humans and robots is crucial during each interaction. To address this challenge, building on a categorization of the information content and speech act classification in the context of HRI in shared environments, Speech and Gesture recognition pipelines were designed and integrated into HRI architecture for cobots. Leveraging virtual reality (VR) as a testbed, the work generates synthetic datasets to train robust gesture and speech recognition models, overcoming the scarcity of real-world data in agricultural contexts. The framework is empirically validated through VR-based user studies and field experiments, demonstrating improved communication reliability in noisy vineyard environments and reduced task completion times. Notably, the system emphasizes modularity, allowing interchangeable components (e.g., pose estimators and speech classifiers) to adapt to dynamic tasks. Key contributions include (i) a standardized gesture taxonomy tailored to agricultural workflows, (ii) open-source datasets produced from both real and synthetic sources, (iii) a synthetic data generation pipeline for pose estimation, and (iv) a multimodal communication architecture augmented by large language models (LLMs) for contextual reasoning using limited computational capacity in agricultural logistics. By bridging virtual simulations and real-world deployment, this research advances human-robot collaboration in precision agriculture, offering interactive solutions for harvesting, pruning, and logistics tasks. The findings underscore the potential of multimodal HRI and immersive technologies to address collaboration between human expertise and robots and enhance safety and efficiency across both indoor and outdoor collaborative environments. Keywords: Human-Robot Interaction (HRI), Human-Robot Collaboration (HRC), Virtual Reality (VR), Synthetic Data Generation, Multimodal Communication, Gesture Recognition, User Evaluation, large language models (LLMs), Precision Agriculture, Collaborative Robotics.