Incident: A frozen layer that was not frozen
Module 4 · Incident file · About 35 minutes · Needs: Lesson 1 (batch normalisation,model.train()andmodel.eval()) and Lesson 4 (PyTorch modules)
What happened
On 17 April 2018 Vasilis Vryniotis published "The Batch Normalization layer of Keras is broken" on the Datumbox blog (Datumbox, 17 April 2018). He had used Keras heavily and contributed to it, and he had been "bitten by the odd behavior of the Batch Normalization layer" many times.
The setting was transfer learning: take a network trained on a large dataset, freeze its lower layers, and fine-tune the top on your own small dataset. Keras marks a frozen layer with trainable = False.
He measured the damage with a deliberate test. On Keras 2.1.5 he fine-tuned ResNet50 on 200 CIFAR-10 images and evaluated it on the same images. In training mode it scored 100% accuracy; in prediction mode, on identical data, 86%. In a larger CIFAR-10 fine-tune the model reached 87.44% without his patch and 92.36% with it, "almost 5 points higher". In real applications he had seen "even more profound effects (from 100% down to 50% accuracy)".
His pull request to change the layer was not accepted. In a reply dated 5 August 2020 he wrote that "TF2 actually changed the semantics of the BN to operate exactly as proposed on my PR, so this problem is resolved in TF2 keras."