Human-level Video Game Control Through Deep Reinforcement Learning

Abstract

The fact that Alpha Go beat top human go player last year is considered as the milestone of artificial intelligence. Building artificial adaptive intelligence has long been the dream of many scientists. The game of Go, which has been regarded as the game that represents the unique genius of Human, seems impossible for computer or programs conducting exhaustive search to beat human due to its tremendous state space. However, the unprecedented development of machine learning has created new hope of adaptive artificial intelligence. Alpha Go utilized a quite new branch of machine learning algorithm that combines deep convolution network and reinforcement learning. Reinforcement learning was inspired by some psychological and neuroscientific research on animals and concerned with training software agents to gain the most reward by take optimum action under environment. Reinforcement learning agents already have some successful applications in temporal difference learning and robotic control. However, to train generalized software agents, the reinforcement learning system must derive effective representations of the environments from high-dimensional sensory inputs. Consequently, the applications of reinforcement learning are limited in some specific domains where features can be handcrafted. Recent development in deep learning have resulted in significant breakthroughs in computer vision, speech recognition, natural language processing etc. Deep neural network demonstrates fantastic speed and efficiency in abstracting features from high-dimensional data especially deep convolution neural network. Deep reinforcement learning is a combination of deep convolution neural network and reinforcement learning. The task of environment representations is conducted by a deep convolution network instead of handcrafting, which utilizes the superb feature abstraction ability of deep convolution network. DeepMind present the first successful deep reinforcement learning model called deep Q-learning to train control policies directly from highdimensional sensory inputs. The unexceptionable results of using deep Q-learning to play Atari game demonstrate an effective new method for online policy learning. Deep Q-learning indicates its overwhelming advantages over other traditional reinforcement learning methods in applications with high dimensional environment space. It is believed that deep reinforcement learning has many promising applications and opportunities for innovation at various region engaging decision making. This project focus on deep reinforcement learning and investigate its usage in human-level control in terms of feasibility, generality and latent challenges. Two kinds of algorithms, namely value approximation approach and policy gradient approach are implemented and studied but mainly focus on value approximation approach. Several specific algorithms of these two different approaches are implemented and compared on some traditional benchmarks of reinforcement learning and new application field like video game control and simulated robotic control. In this project, the performance of deep reinforcement learning algorithms on video game, which includes flappy bird, Plane fighter and other Atari games, and some robotic control problems. The results demonstrate the admirable performance of deep reinforcement learning over traditional reinforcement learning algorithm when high dimensional sensory input is used. However, some challenges like delayed reward, overestimated approximated value and instability remain to be solved. Deep reinforcement learning has already proved its power in the region of game AI and adaptive robotic control. This is believed to be the first step to general artificial intelligence. The advancement of research in deep reinforcement learning might reveal a brand-new age of technology.