Programming

There's a whole lot of math in that article, and to be honest I'm not 100% clear on what they're trying to say. I think they're talking about how standard quantization isn't making full or "accurate" use of the number space, although if you read through to the end they note that there's really nothing terribly wrong with the standard way. But here's my take.

To simplify, I'll use an integer range 0-4 instead of 0-255, but the principle's the same.

So, say that we're quantizing from a floating-point range of 0.0-1.0 to an integer range of 0-4. Doing things the usual way, that'd be the orange numbers on the bottom of this diagram:

But you can see that this doesn't line up cleanly with a nice, regular division of the range into 5 equal blocks. If we just multiply by 4 (our integer max) and then round off to the nearest integer, the original value ranges represented by our integer 0 and 4 are half the size of the others. E.g. 0.0 to 0.49 become 0, but 0.5 to 1.49 become 1.

Now, if we want to represent true 0.0 and 1.0 simply with our integer range (0-4), we should use 0 to exactly equal 0.0, and 4 to exactly equal 1.0. And converting from integer to float by calculating [value / 4] (or [value / 255] in reality) will get us exactly that. I wouldn't change that.

What I would suggest changing--to anyone who cares enough about it--is how the numbers are converted from float to integer. If you look at the diagram again, you can see that there is one orange integer per evenly-divided block, even if they don't line up with the centers of the blocks. If we take a float (0.0-1.0) representation and multiply it by our integer max plus one, we'll get the blue numbers at the top of the diagram. Then all we have to do is round down to the nearest integer and cap the result at 4 (or 255, or whatever our integer max is), and we'll have a nice, even distribution in our conversion.

I could have misunderstood, but it seems like all of the errors and uncertainty discussed in the article come from how the multiplied floats are truncated into integers. I think that this method eliminates that cleanly.