Semantic information in human-robot teams for extreme environments

Ruan, Tianshu (2025). Semantic information in human-robot teams for extreme environments. University of Birmingham. Ph.D.

Ruan2025PhD.pdf
Text - Accepted Version
Available under License All rights reserved.
Download (21MB)

Abstract

This thesis explores the role of semantic-level information in Situational Awareness (SA) during robot deployments into hazardous environments. In such applications, robots are supervised remotely by a human operator who must remain in a safe zone, distant from the robot and its hazardous surroundings. In particular, we consider Human–Robot Teams (HRT) and Human–Robot Interaction (HRI) in the context of a Variable Autonomy (VA) framework, where the Level of Autonomy (LoA) of the robot can be changed dynamically during missions, varying between direct teleoperation by the human, and fully autonomous behaviours.

We begin with a discussion of different kinds of real-world deployments of robots for First Responder (FR) and Extreme Environment (EE) industries, and examine the types of semantic information that enable the operator to develop SA and understand the remote situation. We propose a taxonomy framework for semantics, describing low-level semantics, high-level semantics, and the relationship between these semantics and raw sensor data, mission tasks, and context.

Building on the taxonomy, we propose a semantic indicator metric and a framework that enables the robot (and its autonomous agent) to reason about the Situational Semantic Richness (SSR) of a scene. The framework shows how different types of semantic information (e.g. signs of human presence in Search and Rescue (SAR) missions, or Light Detection and Ranging (LiDAR) noise during navigation tasks) can each generate a semantic indicator that might be important for SA. These different indicators can then be combined, in a systematic way, to generate an overall SSR score for the current scene, in real-time during missions. This SSR score can then be used to flag semantically-rich situations that may require the high-level reasoning and attention of a human operator. This can help in e.g. guiding LoA changes in a VA paradigm.

Subsequently, we apply the semantic richness framework in a SAR mission in a mock-up disaster environment, where we incorporate the real-time semantic indicators into a Graphical User Interface (GUI) on the human operator’s control screen and the SSR score processing in the background. We investigate the impact of these semantic indicators on HRI, and on the overall mission performance of the HRT, the usefulness of the semantics-based SA framework, by successfully lowering the human’s cognitive workload and increasing human’s trust in the VA system. Moreover, some additional interesting findings were noticed in the experiments, e.g. a correlation between the LoA preference and trust in the robotic system.

Finally, we survey expert FRs and operators in EE industries to understand their perceptions, needs, and views with respect to semantic information. Their feedback is consistent with the findings of our research, and confirms the importance of semantic information in future robot developments and deployments.

In summary, the main contributions of the thesis include: a taxonomy framework for systematically reasoning about semantics; metrics and methods for autonomously computing the SSR of different scenes in real-time; an analysis of the impact of incorporating the SSR indicators into VA HRI; and insights and findings of interest for incorporating semantic information into future HRT/HRI developments.

Type of Work:

Thesis (Doctorates > Ph.D.)

Award Type:

Doctorates > Ph.D.

Supervisor(s):