Game-Theoretic Multi-Agent Reinforcement Learning
2024 · Game theory, opponent-learning awareness, and cooperation in multi-agent reinforcement learning.
Future AI systems will be deployed in highly dynamic and complex environments such as road traffic, search-and-rescue operations, smart power grids, stock markets, and the Internet of Things. An effective and ethical AI agent must be able to 1) handle a temporally evolving environment and 2) take into account the presence of other intelligent agents. In this essay, I elucidate how issue 1) is addressed by reinforcement learning (RL) theory and 2) is addressed by game theory, to then fuse the two to arrive at multi-agent reinforcement learning algorithms. In particular, I explain a sophisticated technique called Learning with Opponent-Learning Awareness (LOLA), which makes an agent aware of the learning process of other agents rather than treating them as a static part of the environment, and demonstrate that it achieves social welfare in simple, partially competitive settings. My work serves as a rapid introduction to the relevant concepts for aspiring researchers digging into multi-agent RL, as well as a showcase for how small but clever algorithmic tweaks can greatly impact the emergent behaviour of AI systems in the non-cooperative case.