Incident: The same pictures, resized two ways

Module 13 · Incident file · About 35 minutes

What happened

In April 2021 Gaurav Parmar and Jun-Yan Zhu (Carnegie Mellon University) and Richard Zhang (Adobe Research) posted "On Aliased Resizing and Surprising Subtleties in GAN Evaluation" (arxiv.org/abs/2104.11222, v1 22 April 2021, v3 21 January 2022; CVPR 2022). It is a published account of a measurement failure across a whole field.

Image generators were compared by FID, a distance between features of real and generated images, computed after both are resized to the feature network's input. The authors found that "differences in 'low-level' preprocessing, specifically image resizing and compression, can induce large variations". Their Figure 1 downsamples a circle by a factor of 8: PIL widens its filter with the factor, while "implementations using OpenCV, TensorFlow and PyTorch libraries with default flags, contain severe aliasing artifacts".

The cost was in the numbers. The same real face images, resized with PIL bicubic and with PyTorch bilinear, scored FID 4.36 against each other, when the right answer is 0. Generated images resized differently from the reference scored 7.45 instead of 2.98. Yet "variations across recent methods are typically within 1 FID". Rankings, and even "a 'best' model checkpoint", could change with the resize call.