Zachary Curtis successfully defended his M.S. thesis Adversarial Inverse Reinforcement Learning For Complex Vehicle Tasks on 4 September. Zach has been with VSCL for two years now, coming to TAMU from Utah Tech where he took classes from another VSCL alumnus, Kameron Eves. He has been heavily involved in autonomous vehicle control and getting the in-lab built rotorcraft, the Lunatic, flight-worthy.
Autonomous unmanned aerial systems must perform tasks that combine perception, decision making, and control while operating under dynamic constraints. Reinforcement learning can learn control policies through simulation, but its effectiveness depends on the reward function used to define successful behavior. For multicopter target tracking, manually designed rewards must balance target-centering accuracy, stable flight, control effort, and safety limits, making reward engineering difficult to scale to new vehicles and environments. This research investigates adversarial inverse reinforcement learning as a method for recovering task-level reward functions from expert demonstrations and reducing dependence on handcrafted reward design.
The method is evaluated on a simulated multicopter visual target-tracking task in which a vehicle commands altitude, forward velocity, lateral velocity, and yaw rate through a lower-level nonzero setpoint controller. Expert trajectories are used to train an adversarial reward model and a reinforcement learning policy. Training behavior is examined across discriminator update frequencies, policy learning settings, entropy regularization, warm starts from behavior cloning, and different expert demonstration sets. The learned policy is compared with behavior cloning and expert baselines, and the recovered reward is transferred from a quadcopter simulation to a hexacopter simulation to assess whether the inferred task objective remains useful under changed vehicle dynamics.
Zach’s research is supported by the Army Research Laboratory on the project Robust Threat Detection for Ground Combat Vehicles with Multi-Domain Surveillance in Hostile Environments. Zach is the 69th graduate degree advised by Dr. John Valasek and has full
Approaches for teaching learning agents via human demonstrations have been widely studied and successfully applied to multiple domains. However, the majority of imitation learning work utilizes only behavioral information from the demonstrator, i.e. which actions were taken, and ignores other useful information. In particular, eye gaze information can give valuable insight towards where the demonstrator is allocating visual attention, and holds the potential to improve agent performance and generalization. In this work, we propose Gaze Regularized Imitation Learning (GRIL), a novel context-aware, imitation learning architecture that learns concurrently from both human demonstrations and eye gaze to solve tasks where visual attention provides important context. We apply GRIL to a visual navigation task, in which an unmanned quadrotor is trained to search for and navigate to a target vehicle in a photo-realistic simulated environment. We show that GRIL outperforms several state-of-the-art gaze-based imitation learning algorithms, simultaneously learns to predict human visual attention, and generalizes to scenarios not present in the training data.
) successfully defended his Ph.D. dissertation titled “Multiple-Timescale Adaptive Control for Uncertain Nonlinear Dynamical Systems”. Kameron’s dissertation investigated combining nonlinear multiple time-scale controllers that VSCL has been researching for the last 15 years, with adaptive controllers which VSCL has been researching for more than 20 years. Multiple-timescale control has been shown to have difficulty with uncertain systems and adaptive control has been shown to have difficulty with multiple-timescale systems. His dissertation describes a novel control methodology called [K]Control of Adaptive Multiple-timescale Systems (KAMS). KAMS seeks to address systems that simultaneously exhibit uncertain and multiple-timescale behaviors. Unlike traditional multiple-timescale control literature, KAMS uses adaptive control to stabilize the subsystems. The reference models and adapting parameters used in adaptive control significantly complicate the stability analysis. KAMS is a flexible theory and framework and the stability proofs apply to a wide array of adaptive algorithms and multiple-timescale fusion techniques. Additionally, formal and numerical validation of how KAMS can relax the minimum phase assumption for a multitude of common adaptive control methods. KAMS is demonstrated and evaluated on examples consisting of stabilization and attitude control of a quadrotor Unmanned Air System; fuel-efficient orbital transfer maneuvers; and preventing inlet unstart on hypersonic aircraft.