Proximal-Policy-Optimization

Proximal-Policy-Optimization, often abbreviated as PPO, is a prominent Reinforcement-Learning algorithm introduced by OpenAI in 2017. It has since become one of the most popular choices for training Neural-Networks in complex environments due to its relative simplicity and high performance. The algorithm was designed to overcome the limitations of Trust-Region-Policy-Optimization (TRPO) by providing a similar level of stability while being much easier to implement and computationally cheaper.

At the heart of Proximal-Policy-Optimization is the Clipped-Surrogate-Objective. Standard Policy-Gradient-Methods are often sensitive to the choice of step size; a single large update can significantly change the Policy, leading to a collapse in performance. Proximal-Policy-Optimization solves this by penalizing changes to the Policy that move it too far from the previous version. By clipping the probability ratio between the new policy and the old policy within a small interval (typically [0.8, 1.2]), the algorithm ensures that the Optimization process remains within a safe 'trust region'.

This algorithm is typically implemented within an Actor-Critic framework. In this setup, the Actor learns the best actions to take in various states, while the Critic estimates the Value-Function to provide feedback on the quality of those actions. This architecture, combined with the clipped objective, allows Proximal-Policy-Optimization to achieve state-of-the-art results in diverse fields such as Robotics, Autonomous-Vehicles, and Atari-Games. For a deeper technical dive, refer to the original research paper: Proximal Policy Optimization Algorithms.

• Deep-Q-Networks
• Actor-Critic-Models
• Advantage-Actor-Critic
• Policy-Gradient