Stream: beginners

Topic: Best way to handle very big "data" libraries

Hannes (Jun 05 2024 at 01:19):

I'd like to build some localisation packages for Roc using the unicode CLDR data, I've made a start here by using a codegen script to generate 11MB of Roc code :sweat_smile: Unsurprisingly, roc check doesn't like running on this codebase. I assume the right way to do this would be to load the JSON data as a string and using roc-json to parse it in the package?

Hannes (Jun 05 2024 at 01:22):

The reason I didn't do that initially was because I thought the Roc compiler wouldn't be able to do any dead code elimination if the data was imported as a string, leading to a giant binary size if the package was imported in an app. Maybe with compile-time constant evaluation, it would be possible to not include all the loaded data in the final binary?

Also, let me know if I'm completely wrong about any of this, I'm making a bunch of assumptions without having done any research :sweat_smile:

Brendan Hansknecht (Jun 05 2024 at 01:23):

Theoretically correct with compile time eval....at the same time, the eval might be similar expensive to the result of the code gen

Luke Boswell (Jun 05 2024 at 01:24):

Have you seen the approach I took with roc-unicode? nvm

Richard Feldman (Jun 05 2024 at 01:41):

Hannes said:

using a codegen script to generate 11MB of Roc code :sweat_smile: Unsurprisingly, roc check doesn't like running on this codebase.

I'm actually really interested in this!

Richard Feldman (Jun 05 2024 at 01:41):

that is, I'm interested in roc check liking it - or at least being ok with it :big_smile:

Richard Feldman (Jun 05 2024 at 01:41):

what happens when you run roc check on it?

Hannes (Jun 05 2024 at 01:43):

I remember hearing on an episode of software unscripted that you were interested in roc check being fast :)

Hannes (Jun 05 2024 at 01:45):

When I run roc check it maxes out half of my cpu cores, immediately uses 1.4GB of ram and then slowly uses less and less ram, I've never left it for more than 3 minutes though, I'll try and find out if it ever does finish :sweat_smile:

Update: the ram usage seems to fall slowly down to 500MB and then jumps back to 1.5GB, it's done that three or so times in the last 6 minutes, no idea if this is helpful or not :shrug:

Hannes (Jun 05 2024 at 02:07):

After 20 minutes I got this error:

$ roc check src/main.roc

thread 'main' has overflowed its stack
fatal runtime error: stack overflow

Richard Feldman (Jun 05 2024 at 02:47):

ok cool! Would you mind opening an issue with steps to reproduce?

Richard Feldman (Jun 05 2024 at 02:48):

I don't think we've ever thrown this much at it before, but I want roc check to be able to handle it!

Hannes (Jun 22 2024 at 12:44):

Finally got round to creating a reprex and opening an issue: https://github.com/roc-lang/roc/issues/6830 :)

Last updated: Jul 06 2025 at 12:14 UTC