Machine learning driven multimodal control for human-robot interaction

Abstract

The transition toward Industry 5.0 requires robotic systems that can work naturally and safely alongside humans. To achieve this, robots must be capable of understanding human inputs such as gestures and speech, rather than relying solely on fixed commands or pre-programmed behaviours. However, many existing Human–Robot Interaction (HRI) systems struggle to handle such multimodal inputs and often lack the ability to combine language understanding with task execution in a unified and scalable manner. This research addresses these challenges by developing a general multimodal HRI framework that connects high-level human intent with low-level robot actions.
The proposed framework integrates three principal components: a Convolutional Neural Network (CNN)-based gesture recognition system (GESTID) for real-time classification of static and dynamic hand gestures; a Goal-Oriented Action Planning (GOAP) engine for flexible and adaptive task sequencing; and a Large Language Model (LLM) interface for interpreting natural language commands. The framework focuses on mapping ambiguous human inputs into clear and executable robot actions through semantic reasoning and structured planning. As a practical demonstration, the framework is implemented and evaluated on a7-Degree-of-Freedom (DoF) Franka Emika Panda robotic manipulator, illustrating how the proposed approach can be applied to a real-world physical system.
Experimental evaluations were conducted across complex manipulation scenarios, including beverage preparation and object handling tasks. The vision-based gesture recognition module achieved a classification accuracy of 99.1% with an end to-end latency of 60–70 ms, enabling responsive real-time interaction. The speech based interaction system, combined with GOAP-driven task planning, achieved an overall task success rate of 85.4% under dynamic operating conditions. The LLM-based semantic mapping module correctly interpreted natural language instructions with an accuracy of 94.67% using a cloud-based model, and 94.2% using a fully offline Small Language Model (SLM), demonstrating deployability without network dependency. These results confirm that combining visual perception, speech understanding, and language-driven reasoning within a single planning governed framework significantly improves the flexibility, robustness, and reliability of human–robot interaction.
This thesis presents a validated and scalable multimodal HRI framework that is not limited to a single robot platform or task domain. The proposed approach is suitable for a wide range of applications, including industrial automation, healthcare, and assistive robotics, where intuitive, hands-free, and natural interaction between humans and robotic systems is essential.

Date of Award	Jun 2026
Original language	English
Awarding Institution	University of Brighton
Supervisor	Khizer Saeed (Supervisor), Almas Baimagambetov (Supervisor) & Nikolaos Polatidis (Supervisor)