Context
This paper was my thesis for the bachelor AI (officialy called 'Kunstmatige Intelligentie') at the Utrecht University. I collaborated with Giovanni Varricchione, who at the time was a PHD student at the Utrecht University.
Context
This paper was my thesis for the bachelor AI (officialy called 'Kunstmatige Intelligentie') at the Utrecht University. I collaborated with Giovanni Varricchione, who at the time was a PHD student at the Utrecht University.
Summary
In this paper I investigated how well a technique that improves sample efficiency for Reinforcement Learning algorithms called Reward Machines performed in a new domain. Reward Machines allow the specification of a goal into multiple steps in the form of a finite state machine. When an agent receives reward from a timestep it not only learns from that situation but also from all the other hypothetical cases were it would have been at another step towards reaching its goal. For example, a robot designed to deliver mail around an office might accidentally stumble into the delivery point during training. Using Reward Machines it could then learn: ‘If I had already had some mail with me I would now receive a large reward’ which would speed up the training process during its later stages.
Summary
In this paper I investigated how well a technique that improves sample efficiency for Reinforcement Learning algorithms called Reward Machines performed in a new domain. Reward Machines allow the specification of a goal into multiple steps in the form of a finite state machine. When an agent receives reward from a timestep it not only learns from that situation but also from all the other hypothetical cases were it would have been at another step towards reaching its goal. For example, a robot designed to deliver mail around an office might accidentally stumble into the delivery point during training. Using Reward Machines it could then learn: ‘If I had already had some mail with me I would now receive a large reward’ which would speed up the training process during its later stages.
In this thesis I applied reward machines together with a RL technique called DDPG to a new domain. This domain combined two aspects that when combined are difficult for RL agents: continuous control and navigation. Continuous control involves real valued inputs and outputs leading to a significantly harder learning problem as the agent will likely never be in exactly the same situation twice and must thus learn to adapt. Navigation can be a difficult task in RL because a reward signal might only be received when some long-term goal is achieved (i.e. reaching the goal), which is problematic for continuous control because that difficult setting requires frequent reward signals. It would thus be valuable to see if Reward Machines could help overcome the challenges of this setting.
The domain I designed consisted of a worm like agent called a swimmer and a circular ‘death zone’, all contained in a 2D environment made to act like water to the swimmer. The swimmers task was to complete laps around the death zone without ever entering it. For the reward machine, I added the subgoals of entering every quadrant around the death zone in order. Ultimately, I found that using Reward Machines was crucial to the success in this environment as without it the agent never learns how to get any reward at all.