Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription (2020)
There is now significant historical data available on decision making in organizations, consisting of the decision problem, what decisions were made, and how desirable the outcomes were. Using this data, it is possible to learn a surrogate model, and with that model, evolve a decision strategy that optimizes the outcomes. This paper introduces a general such approach, called Evolutionary Surrogate-Assisted Prescription, or ESP. The surrogate is, for example, a random forest or a neural network trained with gradient descent, and the strategy is a neural network that is evolved to maximize the predictions of the surrogate model. ESP is further extended in this paper to sequential decision-making tasks, which makes it possible to evaluate the framework in reinforcement learning (RL) benchmarks. Because the majority of evaluations are done on the surrogate, ESP is more sample efficient, has lower variance, and lower regret than standard RL approaches. Surprisingly, its solutions are also better because both the surrogate and the strategy network regularize the decision-making behavior. ESP thus forms a promising foundation to decision optimization in real-world problems.

Best-Paper Award in the GECH track.

View:
PDF
Citation:
In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2020), 2020.
Bibtex:

Presentation:
Video
Olivier Francon Collaborator olivier francon [at] cognizant com
Santiago Gonzalez Ph.D. Alumni slgonzalez [at] utexas edu
Babak Hodjat Collaborator babak [at] cognizant com
Elliot Meyerson Ph.D. Alumni ekm [at] cs utexas edu
Risto Miikkulainen Faculty risto [at] cs utexas edu
Xin Qiu Collaborator xin qiu [at] cognizant com
Hormoz Shahrzad Masters Alumni hormoz [at] cognizant com