Incident: Two groups of one algorithm, called different

Module 15 · Incident file · About 35 minutes

What happened

On 19 September 2017 six researchers from McGill University and Microsoft Maluuba, Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup and David Meger, posted "Deep Reinforcement Learning that Matters" (arXiv 1709.06560). It was published at the AAAI Conference on Artificial Intelligence in 2018; the version cited here is v3, revised 30 January 2019.

The authors tried to reproduce published results for policy-gradient methods on standard control benchmarks. Their abstract: "reproducing results for state-of-the-art deep RL methods is seldom straightforward", and "it is difficult to determine whether improvements over the prior state-of-the-art are meaningful."

The sharpest experiment needed no second method at all. They ran TRPO on HalfCheetah-v1 ten times with one hyperparameter configuration, "only varying the random seed across all 10 trials", and split the runs into two sets of five. A two-sample t-test across training gave t = -9.0916, p = 0.0016. In their words, "the variance between runs is enough to create statistically different distributions just from varying random seeds."

What it cost is in their conclusion: without full details and implementations, "wasted effort on reproducing state-of-the-art works will plague the community and slow down progress."