Reinforcement Learning

Hannah Lehman successfully defended her Ph.D. dissertation on May 27th, 2025.  Hannah has been with VSCL since her freshman year in Spring 2017, for a total of 8.42 years with VSCL during which she implemented the Theory-Computation-Experiment paradigm. The title of her dissertation is: Hierarchical Auctions for the Coordination of Heterogeneous Agents using Machine Learning

Hannah’s dissertation investigates autonomous multiagent coordination.  Machine learning has long been discussed as a candidate for facilitating autonomous multiagent vehicle coordination. Many methods of autonomous multiagent coordination have been proposed, however few if any solutions consider realistic communication challenges. By using machine learning on multiple levels, and a self organizing hierarchical system, an autonomous, pseudo decentralized, heterogeneous, system can dynamically complete tasks without being fully connected. This approach, called Hierarchical Auctions for the Coordination of Heterogeneous Agents (HACHA) will be investigated and demonstrated on four simple, proof of concept simulations. Each simulation scenario is designed to demonstrate HACHA’s applicability to a different subset of multiagent problems and address specific requirements. Within HACHA, specific algorithm and data choices will be motivated real-world hardware constraints and informed by time complexity analysis of sub-algorithms. Results show that a parallel auction coordination framework can be used to organize multiple heterogeneous agents with different sensors, movement modalities, graph connectedness, and controllers to complete a task requiring multiple agents. The auction framework is independent of individual agents and has been utilized in this paper by a combination of reinforcement learning trained agents and optimally controlled agents to complete tasks. HACHA auction propagation methods are explored and recommended use case rules are developed based on theoretical and computational investigations and results. The HACHA auction choice is explored and compared to other popular auction methods over a variety of relevant network characteristics including dynamicism, sparsity, and number of tasks.

Hannah will be doing a short postdoc with VSCL and then starting full-time at Sandia National Laboratories, where she has now done four graduate internships, in July. Hannah’s is the 62^nd graduate degree advised by Dr. John Valasek and the 15^th Ph.D. student.

MD Sunbeam (B.S. Aerospace Engineering, University of Texas) successfully defended his Masters thesis: “Gaze-Regularized Imitation Learning”. 

Approaches for teaching learning agents via human demonstrations have been widely studied and successfully applied to multiple domains. However, the majority of imitation learning work utilizes only behavioral information from the demonstrator, i.e. which actions were taken, and ignores other useful information. In particular, eye gaze information can give valuable insight towards where the demonstrator is allocating visual attention, and holds the potential to improve agent performance and generalization. In this work, we propose Gaze Regularized Imitation Learning (GRIL), a novel context-aware, imitation learning architecture that learns concurrently from both human demonstrations and eye gaze to solve tasks where visual attention provides important context. We apply GRIL to a visual navigation task, in which an unmanned quadrotor is trained to search for and navigate to a target vehicle in a photo-realistic simulated environment. We show that GRIL outperforms several state-of-the-art gaze-based imitation learning algorithms, simultaneously learns to predict human visual attention, and generalizes to scenarios not present in the training data.

This work is sponsored by the Army Research Laboratory (ARL) through the Cycle of Learning Project. MD Sunbeam is employed as a researcher at the Human Research and Engineering Directorate (HRED), ARL.