Entropy estimation and optimization and their applications in Bayesian experimental design

Ao, Ziqiao (2024). Entropy estimation and optimization and their applications in Bayesian experimental design. University of Birmingham. Th.D.

Preview

Ao2024PhD.pdf
Text - Accepted Version
Available under License All rights reserved.
Download (6MB) | Preview

Abstract

This thesis delves into the realm of information theory, focusing on the crucial concept of entropy - a measure of uncertainty in datasets or signals. Initially conceptualized by Claude E. Shannon, entropy's applications have significantly expanded, influencing disciplines ranging from physics to biology to machine learning. Central to this thesis is the challenge of entropy estimation and optimization, particularly in high-dimensional spaces, and their applications in Bayesian Experimental Design (BED).

The thesis presents and addresses three primary research objectives. The first is a comprehensive survey of existing and emerging entropy estimation and optimization methodologies, with a special emphasis on their applications in BED. This exploration covers traditional methods like plug-in and k-NN estimators, as well as advanced techniques such as variational estimators and entropy gradient estimation methods. Moreover, the connection between entropy optimization and BED is thoroughly examined, revealing the potential of entropy optimization techniques to inspire innovative BED methodologies.

The second objective of the thesis is centered on reducing the bias inherent in entropy estimators, particularly in the context of high-dimensional entropy estimation challenges. To meet this goal, the thesis introduces an innovative transform-based method for high-dimensional entropy estimation. This method integrates a novel k-NN based estimator with a normalizing flow-based mapping technique, effectively achieving a significant reduction in estimation bias when compared to traditional methods. Additionally, the thesis provides comprehensive theoretical analyses to validate the consistency and efficiency of this proposed method, demonstrating its effectiveness in high-dimensional settings.

The third objective focuses on advancing BED through entropy gradient estimation. This objective is achieved by directly estimating the gradient of the design criterion, which includes an entropy term, with respect to design variables. Subsequently, stochastic gradient descent is applied to find the optimal design. Within this framework, the thesis introduces two novel methods for estimating the expected information gain (EIG) gradient: UEEG-MCMC and BEEG-AP. Each of these methods has its distinct advantages and limitations, which are meticulously examined and validated through a combination of theoretical analyses and empirical experiments. This comprehensive evaluation underscores the practicality and applicability of these methods in advancing the field of BED.

Finally, the thesis concludes with a discussion on its contributions and future research directions. It proposes two trajectories for further research: developing a global optimization method for BED that synergies local search capabilities of gradient-based methods with the global search efficiency of Bayesian Optimization, and extending entropy gradient estimation techniques to BED for implicit models.

Type of Work:

Thesis (Doctorates > Th.D.)

Award Type:

Doctorates > Th.D.

Supervisor(s):