“We got a thousand times improvement [in training performance per chip] over the last 10 years, and a lot of it has been due to number representation,” Bill Dally, chief scientist and senior vice president of research at Nvidia said at the recent IEEE Symposium on Computer Arithmetic.