An Abstraction-based Technique for Safe Reinforcement Learning

	An Abstraction-based Technique for Safe Reinforcement Learning (EPSRC NIA Project)
Home Projects Publications Outreach Supervision Abstractions for Safe RL FMAI Seminar	Overview Autonomous agents learning to act in unknown environments have been attracting research interest due to their wider implications for AI, as well as for their applications, including robotics, network optimisation, and resource allocation. Currently, one of the most successful approaches is reinforcement learning (RL). However, to learn how to act, agents are required to explore the environment, which in safety-critical scenarios means that they might take dangerous actions, possibly harming themselves or even putting human lives at risk. Consequently, reinforcement learning is still rarely used in real-world applications, where multiple safety-critical constraints need to be satisfied simultaneously. To alleviate this problem, RL algorithms are being combined with formal verification techniques to ensure safety in learning. Indeed, formal methods are nowadays routinely applied to the specification, design, and verification of complex systems, as they allow to obtain proof-like certification of their correct and safe behaviour, which is meant to be intelligible to system engineers and human users alike. These desirable features have fostered the adoption of formal methods for the verification of general AI systems, which has variously been called safe, verifiable, trustworthy AI. Still, the application of formal methods to AI systems raises significant new challenges, also due to the “black-box” nature of most machine learning algorithms currently used. Specific to the application of formal methods to RL, we identify two main shortcomings with present approaches, which will be tackled in this project: Systems with multiple learning agents are comparatively less explored, and therefore less understood, than single-agent settings, partly because of the high-dimensionality of their state-space and their non-stationarity. Yet, multi-agent systems are key for applications, such as platooning for autonomous vehicles and robot swarms. Also related to the previous point, most of current verification methodologies do not scale well as the complexity of the application increases. This state explosion problem is particularly acute for RL scenarios, where agents might have to chose among a huge number of action/state transitions (e.g., robot navigation). To tackle both problems, we put forward an abstraction-based approach to verification, which is meant to reduce the state space, while preserving all its safety-related features, thus leading to guaranteed and scalable safe behaviours. The safe RL methodology thus developed is meant to have a positive societal impact on the trust of the general public towards deployed AI solutions, and to facilitate their adoption within society at large.

An Abstraction-based Technique for Safe Reinforcement Learning (EPSRC NIA Project)

Overview

Autonomous agents learning to act in unknown environments have been attracting research interest due to their wider implications for AI, as well as for their applications, including robotics, network optimisation, and resource allocation.
Currently, one of the most successful approaches is reinforcement learning (RL). However, to learn how to act, agents are required to explore the environment, which in safety-critical scenarios means that they might take dangerous actions, possibly harming themselves or even putting human lives at risk.
Consequently, reinforcement learning is still rarely used in real-world applications, where multiple safety-critical constraints need to be satisfied simultaneously.

To alleviate this problem, RL algorithms are being combined with formal verification techniques to ensure safety in learning.
Indeed, formal methods are nowadays routinely applied to the specification, design, and verification of complex systems, as they allow to obtain proof-like certification of their correct and safe behaviour, which is meant to be intelligible to system engineers and human users alike.
These desirable features have fostered the adoption of formal methods for the verification of general AI systems, which has variously been called safe, verifiable, trustworthy AI.
Still, the application of formal methods to AI systems raises significant new challenges, also due to the “black-box” nature of most machine learning algorithms currently used.

Specific to the application of formal methods to RL, we identify two main shortcomings with present approaches, which will be tackled in this project:

Systems with multiple learning agents are comparatively less explored, and therefore less understood, than single-agent settings, partly because of the high-dimensionality of their state-space and their non-stationarity.
Yet, multi-agent systems are key for applications, such as platooning for autonomous vehicles and robot swarms.

Also related to the previous point, most of current verification methodologies do not scale well as the complexity of the application increases.
This state explosion problem is particularly acute for RL scenarios, where agents might have to chose among a huge number of action/state transitions (e.g., robot navigation).

To tackle both problems, we put forward an abstraction-based approach to verification, which is meant to reduce the state space, while preserving all its safety-related features, thus leading to guaranteed and scalable safe behaviours.
The safe RL methodology thus developed is meant to have a positive societal impact on the trust of the general public towards deployed AI solutions, and to facilitate their adoption within society at large.