Stream: show and tell

Topic: Article: 0.1 + 0.2

Richard Feldman (May 02 2024 at 23:55):

I wrote an article about how 0.1 + 0.2 works in Roc - any feedback welcome!

https://docs.google.com/document/d/11JzH-Xjc2WpRT0yp8w4PXwUtq9KJcL_EQ4-PG9f_JI8/edit?usp=sharing

Richard Feldman (May 03 2024 at 00:03):

not specifically in response to https://twitter.com/oznova_/status/1784841631143006367 but timely :laughing:

Karl (May 03 2024 at 00:29):

Not critical but the article doesn't show how to get F32/F64 numbers in Roc. There's an implication for doing it it via top level function annotations and the type names are provided but I think having it written out somewhere would be nice.

Brendan Hansknecht (May 03 2024 at 00:56):

20 before decimal point,
17 after decimal point

Should say 18 after the decimal point

Brendan Hansknecht (May 03 2024 at 00:57):

And I guess we can technically store some numbers with 21 before, but that is quite limited.

Brendan Hansknecht (May 03 2024 at 01:02):

I think it would also be good to mention precision loss due to multiplication. It hits fixed point decimals significantly more than floating point decimals.

Brendan Hansknecht (May 03 2024 at 01:03):

E.G. 0.000000001*0.0000000001 is 0 with roc's fixed point dec, but should lose no precision in the C# floating point dec. It should just update the exponent.

Brendan Hansknecht (May 03 2024 at 01:04):

And this generally isn't an issue, but with repeated multiplication of decimals, it is essentially guaranteed that some values will fall of the end. I don't think we currently have any sort of rounding for this. So values over time will trend smaller cause this is a form of truncation.

Brendan Hansknecht (May 03 2024 at 01:08):

This may be a clearer example of the error I am talking about: 0.123456789 * 0.00000000001 * 100000000000

Results in 0.1234567

Richard Feldman (May 03 2024 at 02:01):

Brendan Hansknecht said:

20 before decimal point,
17 after decimal point

Should say 18 after the decimal point

oh yeah I guess we get all 18 as long as we only have 20 before :thumbs_up:

Richard Feldman (May 03 2024 at 02:02):

thanks, those are good points to clarify!

Richard Feldman (May 03 2024 at 02:03):

after writing this I got curious about how our Dec performance compares to C#'s System.Decimal but I really need to switch to preparing for this live-coding demo on May 7 and not start another project :laughing:

Richard Feldman (May 26 2024 at 03:26):

here's a prettier draft with the previous round of feedback incorporated: https://rtfeldman.com/0.1-plus-0.2 - I'm looking to share it around Tuesday, so any feedback welcome between now and then!

Isaac Van Doren (May 26 2024 at 05:08):

Nice! It was interesting to hear that Go evaluates constants differently

Jasper Woudenberg (May 26 2024 at 07:43):

Loved the article! I liked the introduction to the Dec type, but it also works quite well as documentation for base-2 floating point behavior, I think! Got two feedbacky thoughts:

The article mentions calculations with money as a usecase for Dec, but I'm curious if we'd recommend that outside of the context of this blogpost demonstrating Dec? I think I might still prefer to represent money as the amount of cents, stored in an Int, to make amounts with partial cents unrepresentable.

Jasper Woudenberg (May 26 2024 at 07:50):

Second thought: Given the performance benefits, do we expect base-2 floating points to be the default in most production code? If so, is it going to trip people up that the default decimal type used in the REPL will be different to the default decimal type in most production code?

Concrete scenario: Someone new to the intricacies of base-2 float points sees a test failure that, when reduced, comes down to 0.1 + 0.2 != 0.3. When they try in the REPL, that works. Back in the test, it doesn't. Do we have a path for folks to resolve that confusion by themselves?

Sven van Caem (May 26 2024 at 10:25):

I would guess that since floating point numbers are explicitly opt-in, the only way for a codebase to end up using them is if said hypothetical person opted into them themselves, or someone on their team did. In the former case, it would mean that somewhere in their program is a type annotation or number literal suffix that might be discovered when looking for differences between the broken code and the working code in the REPL. In the latter case, it would just be a matter of asking one's coworkers.

Richard Feldman (May 26 2024 at 11:45):

Jasper Woudenberg said:

Given the performance benefits, do we expect base-2 floating points to be the default in most production code?

I expect it to vary by use case. For graphical things, I expect lots of F32s because that's what GPUs prefer. For money things, I expect Dec - I don't know why anyone would opt into floats for that! :sweat_smile:

Richard Feldman (May 26 2024 at 11:46):

so I guess I don't think there will be one uniform answer to that :big_smile:

Richard Feldman (May 26 2024 at 11:47):

Jasper Woudenberg said:

I think I might still prefer to represent money as the amount of cents, stored in an Int, to make amounts with partial cents unrepresentable.

so I based this on looking into how people represent money in other languages - I thought this was a good discussion (not any one answer, necessarily, but the discussion itself)

Richard Feldman (May 26 2024 at 11:48):

it seems like people tend to view more digits after the decimal point as a benefit when dealing with money, and then truncate only when printing

Richard Feldman (May 26 2024 at 11:48):

for example, one person in that discussion said "storing accounting values as integer multiples of 1/10000th of unit is very common"

Jasper Woudenberg (May 26 2024 at 13:02):

Interesting stuff! I've been working with a payment system for the past year or two and the need for fractional cents hasn't shown up there and that influenced me a bit, but I can totally imagine that someone writing, say, software for a stock market would have different needs. Thanks for the link!

Brendan Hansknecht (May 26 2024 at 13:57):

I would honestly expect Dec to be more common in most production systems for a few reasons:

1. It is the language default so many will use it without thinking about it.
2. It has a number of nice accuracy related benefits that people will get used to
3. Most production code is business logic where correctness is more important than perf
4. Assuming you are mostly adding/subtracting, Dec is faster than floats.

Matthieu Pizenberg (May 26 2024 at 17:01):

Really good read! I have a few remarks. Coming from the scientific world, I think I’m missing one major argument for when to use floating point representations. When you deal with multiplication and division of multiple values, you can easily reach the limits of fixed points representations. Basically you have no choice when doing math. But if most of your use cases only involve addition / substraction, and very few multiplications/divisions then no need for floating point. In that sense it’s nice that Decimal is even faster at addition/substraction for when it’s the actual preferred use case.

Matthieu Pizenberg (May 26 2024 at 17:02):

In a sense you showed this when doing the 0.00...1 square multiplication example.

Richard Feldman (May 26 2024 at 17:21):

ahh interesting! Do you happen to have a nice concrete example of that I could mention?

Richard Feldman (May 26 2024 at 17:21):

like a particular type of scientific calculation that runs into this

Matthieu Pizenberg (May 26 2024 at 17:27):

Anything that involves optimization problem means you want to solve a function that tends to 0 so you need "almost arbitrary" precision close to 0
Anything that involves iterative algorithms with matrix multiplications. Matrix decompositions etc. There is even a concept called the "condition number" https://en.wikipedia.org/wiki/Condition_number that is usually a measure of how "usefull for computation precision" is a matrix.

Matthieu Pizenberg (May 26 2024 at 17:38):

Maybe one concrete use case for potential newcomers would be students trying to solve their physics tasks.
In physics, there are a lot of constants that have very big/small exponents. For example:

• gravity : G=6.67430 * 10^(−11)
• charge of an electron : e=1.602176634 * 10^(−19)
• etc.
  -

Matthieu Pizenberg (May 26 2024 at 17:39):

And you typically multiply / divide them a lot to compute the answer you are looking for.

Richard Feldman (May 26 2024 at 19:20):

that makes sense!

Richard Feldman (May 26 2024 at 19:21):

and I assume you wouldn't care about base-2 vs base-10 in those use cases

Matthieu Pizenberg (May 27 2024 at 07:19):

I don’t think I would, except if there is a huge performance tradeoff and I’m doing something that may be long to compute.

Zeljko Nesic (May 28 2024 at 00:40):

I have just put down an intro to chaos theory that was pioneered by meteorologist who had a slight rounding error that have lead to wildly different results.

I feel that you could have a bit of Pop-Sci bit where you show the importance of the concept of exact numbers.

Richard Feldman (May 28 2024 at 11:11):

this is now published at https://rtfeldman.com/0.1-plus-0.2 - thanks everyone for commenting!

Richard Feldman (May 28 2024 at 12:09):

I submitted it to HN; it's under "new" if anyone wants to upvote it (they penalize upvotes that come from direct links)

Richard Feldman (May 28 2024 at 12:10):

also feel free to share it around anywhere else you like!

Folkert de Vries (May 29 2024 at 08:18):

someone commented that

It's possible to have a fixed-point, base-2 representation, but we won't discuss that combination because it's so rarely used in practice.

is not actually true if you are lowlevel enough. Apparently it comes up all the time in media codecs and such

https://hachyderm.io/@harold@mastodon.gamedev.place/112522962115617393

Richard Feldman (May 29 2024 at 08:34):

whoa, TIL!

Brendan Hansknecht (May 29 2024 at 19:41):

Hmmm.... I guess that makes sense. Any quantized float would become a fixed point base 2 representation.

Brendan Hansknecht (May 29 2024 at 19:41):

So by that definition, also used all the time in ML when trying to get more perf on cpu.

Brendan Hansknecht (May 29 2024 at 19:42):

Basically, you have to know the the offset and multiplier, but otherwise, you do U8 math (so the float is represented as 0 to 255).

Brendan Hansknecht (May 29 2024 at 19:43):

Personally, at least from what I have seen in ML, people wouldn't label quantization as using a fixed point binary format. Instead, they would say they are mapping a range of floats to the integer space and doing integer math

Brendan Hansknecht (May 29 2024 at 19:43):

Though I guess those both could be seen as different words for the same thing. Just perspective

timotree (May 30 2024 at 01:10):

Of course, "arbitrary-size" numbers can't actually represent arbitrary numbers because all but a hilariously small proportion of numbers in mathematics are too big to fit in any computer's memory. Even 1 divided by 3 can't be represented without precision loss in base-10 or base-2 format, no matter how much memory you use in the attempt.

What about "exact rational numbers" where you represent rational numbers using bigint numerator and denominator? I think those warrant discussion as a safe default numeric type

timotree (May 30 2024 at 01:15):

Pyret, a language which like Roc aims to be very beginner friendly uses rational numbers by default

Ben Plotke (May 31 2024 at 19:01):

timotree said:

What about "exact rational numbers" where you represent rational numbers using bigint numerator and denominator? I think those warrant discussion as a safe default numeric type

I was wondering the same thing. But looking into it, it looks like there's a lot of performance tradeoffs, such as:

• Multiplying involves two multiplies instead of one, then possibly a reduction step
• Adding or subtracting involves finding a least common denominator, two multiplies, before finally the add
• Denominators getting large can either be solved by rounding, meaning our representation is no longer exact, or arbitrary length ints, which would result in the performance hit of losing cache locality.

IEEE floats have some really rough edges, so I really like that Dec can provide nicer (though still imperfect) properties for only a small performance hit.

Brendan Hansknecht (May 31 2024 at 19:23):

Yeah, I think the only big issue with Dec is that it is fixed point instead of floating. For most real world numbers it is fine, but math done in the wrong order could lose a lot of precision off the end.

Brendan Hansknecht (May 31 2024 at 19:23):

That said, I think fixed makes a lot of sense in terms of perf and simplicity

Last updated: Jul 06 2025 at 12:14 UTC