Multi-scale energy management for multi-mode hybrid vehicles using multi-agent reinforcement learning

Hua, Min (2025). Multi-scale energy management for multi-mode hybrid vehicles using multi-agent reinforcement learning. University of Birmingham. Ph.D.

Preview

Hua2025PhD.pdf
Text - Accepted Version
Available under License All rights reserved.
Download (43MB) | Preview

Abstract

The automotive industry is undergoing a transformative shift driven by advancements in connectivity, intelligence, and electrification. While the rise of connected and autonomous vehicles (CAVs) and plug-in hybrid electric vehicles (PHEVs) presents new opportunities, it also introduces challenges including maintaining efficient inter-vehicle communication in dynamic traffic environments, coordinating energy flow among multiple propulsion sources under varying operating conditions, and ensuring the control effectiveness and practical applicability of hybrid powertrains. This research addresses these challenges by developing a reinforcement learning (RL) framework that integrates macro-level cooperative adaptive cruise control (CACC) and micro-level energy management strategies (EMSs) for multi-mode PHEVs, enabling intelligent and scalable energy optimization.
At the macro (traffic) level, CACC is critical for ensuring the stability and energy efficiency of CAV platoons. A fully-decentralized multi-agent reinforcement learning (MARL) framework is introduced to enhance communication efficiency by incorporating the quantified stochastic gradient descent (QSGD) and a binary differential consensus (BDC) method. Compared to existing state-of-the-art MARL approaches, BDC-MARL achieves a 5.8% improvement in energy efficiency, maintains an average velocity of 15.26 m/s, and ensures consistent inter- vehicle spacing (IVS) of 20.76 m. Validation using real-world OpenACC data demonstrates its ability to balance platoon stability with energy optimization effectively.
At the micro (vehicle) level, multi-mode PHEVs face the challenge of managing complex multi- input and multi-output (MIMO) energy flows, often leading to oversimplified and suboptimal solutions. To overcome this, a novel MARL-based framework is proposed by incorporating a hand-shaking strategy with a relevance ratio to improve coordination among energy control agents. By fine-tuning parameters such as learning rates and network configurations, this system achieves up to 4% energy savings compared to single-agent systems and up to 23.54% compared to traditional rule-based methods, validated in real-world driving conditions.
From a theoretical perspective, this research further refines RL methods in EMSs for PHEVs by addressing overestimation bias. An ensemble strategy with in-target minimization and an update-to-data (UTD) mechanism reduces the bias from 1.18 to 5.2 × 10\(^{−4}\) while achieving up to 15.59% energy savings compared to conventional methods. Through theoretical analysis, bias visualization, and extensive testing on software-in-the-loop (SiL) and hardware-in-the-loop (HiL) platforms under real-world driving conditions, the practical applicability in improving energy savings is validated.
By integrating macro and micro perspectives and aligning applications with theoretical analyses, this research provides efficient solutions for improving energy efficiency in CACC and PHEVs, contributing to the development of a sustainable and intelligent future in mobility.

Type of Work:

Thesis (Doctorates > Ph.D.)

Award Type:

Doctorates > Ph.D.

Supervisor(s):