Asbjørn Olling (Nov 22 2023 at 13:19): Asbjørn Olling (Nov 22 2023 at 13:19):Hi all!
I'm toying around with getting a roc setup that I like.
I'm an enjoyer of Nix, and was very happy to see that roc is a flake. That makes it super easy to compile the compiler. Neat!
...but I can't figure out how I'm supposed to compile roc files with nix.
Since the dependencies seem to be listed by http URL inside the source code, my best bet so far is to parse the files using plain nix, then download the files to the correct cache directory using fetchurl, and then use that as an input to my derivation. But that doesn't solve getting the hash (which nix needs as well).
Nix doesn't support BLAKE3, which seems to be the format that microsoft github exposes?
Languages like rust have a lockfile (Cargo.lock), which specifies the sha256 sum of each dependency. That enables projects like crane, which can automatically fetch all dependencies using plain nix, by parsing the lockfile.
This isn't necessarily the right solution for Roc - but I really appreciate being able to build roc code from source using nix.
Surely I'm not the first person to wonder about this. It seems there are some nix-lovers in this community :)
How do people usually build roc code with nix?
John Murray (Nov 22 2023 at 13:40):AFAIK you can't easily do so right now since roc will fetch the tar balls at compile time.
I had an idea to make a nix build helper that would do a basic find replace of package urls with a nix store path from a fetchurl call
Need to explore the intervals a bit to see if we could someone one a check pass to get all referenced packages. Worst case it could be a manual mapping to get started
Asbjørn Olling (Nov 22 2023 at 14:04):John Murray said:
AFAIK you can't easily do so right now since roc will fetch the tar balls at compile time.
Well, only if the tarballs aren't already present in the cache directory.
John Murray said:
I had an idea to make a nix build helper that would do a basic find replace of package urls with a nix store path from a fetchurl call
Ah that's an interesting idea. Kind of like the patchShebangs hook
But you probably would need a huge table of platform urls to nix derivations to make that work. At least if you want to stay inside nix.
...or I guess you could make a non-nix thing, that generates nix code for you. Kind of similar to the <something>2nix packages like node2nix
Maybe dream2nix is a nice way of building that?
John Murray said:
Worst case it could be a manual mapping to get started
This would definitely work. I figure I can also get the specific platforms I use to work by writing nix derivations that dumps them in the cache dir. It would just be so nice to have something that can build any roc package.
Richard Feldman (Nov 22 2023 at 14:15):so, some details on how Roc's downloads work:
roc verifies that the BLAKE3 hash matches what was downloaded. (We use BLAKE3 because it seems to be the fastest.)node_modules) Anton (Nov 22 2023 at 14:21):A quick fix can be to use the path platform syntax instead of the URL, like we do here and when developing you should use the prebuilt-platform flag; roc dev hello.roc --prebuilt-platform to prevent the platform from being rebuilt every time.
John Murray (Nov 22 2023 at 15:24):@Anton yea that approach seems like it would work well if we used nix store paths there
Is there a way for roc build to just output all the tars/just download them?
We could make a nix code gen tool like node2nix that
I need to dive more into how the rust builders don't need to do a code gen step but i think the above is on ok start for now
Richard Feldman (Nov 22 2023 at 15:38):I'm interested in exploring this, especially from the perspective of looking for something that would work for both Nix as well as other use cases.
For example, I know (vaguely) that Debian doesn't allow downloads during builds, I'm also aware of people who work in companies that have firewalls and need to mirror things locally.
So I wonder about a design where when the build is resolving urls, there's some way to specify "pass the URL to this thing and it'll go get the sources and put them in the right place for the build to find them"
Richard Feldman (Nov 22 2023 at 15:39):roc build would still do the BLAKE3 verification itself regardless, so I don't think it would be a problem if Nix didn't support that. Also if Nix wanted to do its own separate sha256 verification, I don't see why that would be a problem; it's not like roc would even know about it :big_smile:
Asbjørn Olling (Nov 22 2023 at 15:59):John Murray said:
I need to dive more into how the rust builders don't need to do a code gen step but i think the above is on ok start for now
As I understand it, crane works without a codegen step because it can do everything it needs to using pure nix code, without having to do any effects (like downloading a url in order to get the sha256 hash).
builtins.readFile lets you read the content of a file into a nix variable.
(as long as it is immediately present in the source, and not the output of another derivation)
Cargo.lock contains everything you need to supply arguments to fetchurl (a https url and a sha256hash).
Cargo.lock also contains the entire dependency tree, so it doesn't need to do this recursively. It just needs the one Cargo.lock file.
John Murray (Nov 22 2023 at 16:04):Ahh so it doesnt need import from derivation since the cargo lock has the same info as a fetch url call. Thats neat
Richard Feldman (Nov 22 2023 at 16:09):I'm open to ideas for how specifically to have it work, but I think the main design considerations would be:
roc doing the hash verification against the hash that's in the URL in the source file should rule out a ton of problems here, but I still want to think it through Richard Feldman (Nov 22 2023 at 16:09):I'm open to ideas for how specifically to have it work, but I think the main design considerations would be:
roc doing the hash verification against the hash that's in the URL in the source file should rule out a ton of problems here, but I still want to think it through Asbjørn Olling (Nov 22 2023 at 16:20):If nix supported blake3, then I figure we could do something similar to crane for roc - at least for one level of dependency. Sadly they only support md5, sha1, sha256, and sha512.
(or inversely, if roc supported any of the nix-supported hash formats)
I'm not sure how roc dependencies works. Would we also need to fetch the tarballs of a dependency's dependency, or is that included in the top-level tarball?
I don't really understand the "alternative package source" approach.
I think the core issue is that nix needs to know the sha256 hash of every external package and platform, before nix invokes the roc compiler.
So if the alterntive package source is something that roc calls, I'm not sure it addresses the problem.
John Murray (Nov 22 2023 at 16:33):This is a very rough overview of how i think the generated nix code would look
{pkgs}: let
roc-pkgs = {
basic-cli = {
url = "https://github.com/roc-lang/basic-cli/releases/download/0.6.0/QOQW08n38nHHrVVkJNiPIjzjvbR3iMjXeFY5w1aT46w.tar.br";
sha256 = "<some sha>";
};
# others...
};
importedRocPkgs = {
# map over roc-pkgs and call `import fetchTarball(url, sha256)` to get the nix store path
};
buildRocPackge = {...}: pkgs.mkDerivation {
preBuildStep = ''
# find replace all references to all packages to the nix store paths from importedRocPkgs
'';
# left as an exereicse to the reader....
};
in
buildRocPackge {
name = "foo";
version = "0.1.0";
src = ./; # un-modified roc src code
}
so the user would never need to update their roc code, just run something like roc2nix <file/folder to get the above
Richard Feldman (Nov 22 2023 at 16:44):hm, I may be misunderstanding the motivation here! So let me take a step back and try to understand the problem better.
Let's say I'm on nixOS, I run the roc executable, and it downloads and unzips some source files into a cache in my home directory (possibly repeating this process if those packages depend on other URL packages), and then the build uses those source files as normal.
So what's undesirable about that process? What would be the benefits of having a different process?
John Murray (Nov 22 2023 at 16:51):Theres two different modes I end up in when im using nixos
Dev mode where im working on some code. In cases like this i use cargo, npm, roc etc for quick iterations and better build times
"Prod" mode where I need a nix derviation to import into either my nix system config or other dev shells for other projects. For example if vscode was written in roc I would love to just put roc-code in my list of common programs in my nix config here. Then when i do system rebuilds it would either compile it for me or fetch from a nix binary cache
So in dev mode using roc as is works great!
Prod mode is the issue we are trying to address. Since roc does network calls during its build the nix mkDerivation sandbox will yell since it wont allow arbitrary network calls. The roc2nix approach would generate all the needed packages so the nix sandbox knows they are allowed
John Murray (Nov 22 2023 at 16:52):Its sort of like when you need to package up your project up in a docker file for a deployment. But in this case its a nix derivation and not docker
Richard Feldman (Nov 22 2023 at 18:59):I see - is it important that in prod mode it's still building from source? Or is it ok to just have a final binary that was built in dev mode
Asbjørn Olling (Nov 22 2023 at 19:39):+1 @John Murray
Developing roc projects inside a nix-managed environment already works great.
I think running roc dev, roc check, and friends will always be preferable to nix build and nix flake check.
I think the interesting bit is making a derivation that builds a roc program.
I think roc builds are probably properly reproducible without nix; but nix derivations can be used in other nix derivations, like nixos hosts, nix shells, or for building other programs that depend on my roc program.
I can think of three ways of doing this:
My gut feel right now is that roc2nix is the comfiest way to go.
It also sounds like a fun program to write!
I don't think it's strictly necessary to find/replace the package urls in the source code.
It doesn't seem important whether roc finds the package in /nix/store or in the cache dir of the sandboxed environment.
Since roc compiled roc binaries are (always?) fully static, the deps only need to be present at build-time.
I think it would be really neat if it were possible to add a dependency just inside the roc code, without having to rewrite the nix code that builds the roc package.
Rust projects have this ability because all the info nix needs (hashes and urls) are directly available in Cargo.lock.
Richard Feldman (Nov 22 2023 at 19:51):roc never introduces dynamic dependencies, but it's always possible for the platform to have them - so roc can't guarantee that the output binary is static
Richard Feldman (Nov 22 2023 at 19:54):yeah we do have the same info except the hashes are BLAKE3...I'd rather not change that to something slower (like sha256) or significantly less collision resistant, and I don't think a "pick your own hashing algorithm" feature would work well in practice because package authors would just end up with a ton of issues like "hey please offer a separate URL for this one algorithm because that's how I want to consume it in my use case" etc. :big_smile:
Asbjørn Olling (Nov 22 2023 at 20:07):Richard Feldman said:
I see - is it important that in prod mode it's still building from source? Or is it ok to just have a final binary that was built in dev mode
Warning: a lot of personal preference in this post
I prefer building from source.
I'd rather run a tool like roc2nix when I commit, than I would have my nix file refer to ./build/program or something. Binary blobs are ominous.
I like using nix as a complete CI tool. It's easy to run pipelines locally.
It's also cool that I can compile all my personal projects with nix build.
Having to run roc build && nix build instead is not that bad, and with something like roc2nix it'd be kind of equivalent anyway: roc2nix && nix build.
But with roc2nix, I would only have to run it on dependency changes, and I can check the resulting nix code into version control- (unlike the compiled binary).
...also now I kind of have an itch to write roc2nix (in roc, of course). :sweat_smile: No promises...
Asbjørn Olling (Nov 22 2023 at 20:09):Btw thanks for taking the time to look into stuff like this. :people_hugging:
Asbjørn Olling (Nov 22 2023 at 20:15):and 100% agree about changing the hashing algorithm. Even if roc supported multiple hashing algorithms, it's silly to hope for all roc code to use the hashing format that's compatible with my setup. It seems sane to go with The Right Thing, and use the one best hashing algorithm.
John Murray (Nov 23 2023 at 05:16):Spent some time working on a prototype for a roc2nix approach where the user specifies all the urls and shas. It almost works https://github.com/JRMurr/roc2nix
currently im seeing an error in the call to roc build of the patched source code of
thread 'main' panicked at 'There were still outstanding Arc references to module_ids', crates/compiler/load_internal/src/file.rs:1564:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrac
i probably need to do some more nix wrapping on the roc compiler in the derivation or something but has anyone seen that error before?
Luke Boswell (Nov 23 2023 at 05:35):Is it related to https://github.com/roc-lang/roc/issues/5951 ?
John Murray (Nov 23 2023 at 05:38):ahh yea thats it, thanks!
John Murray (Nov 23 2023 at 06:24):Got something working https://github.com/JRMurr/roc2nix/blob/main/lib/mkRocDerivation.nix
high level is user gives a list of {url = "...", sha256="..."} then this nix code will download the packages, decompress, and extract them to a folder in the nix store. then we replace all references to the urls with the nix store paths (only in the build not in the source code in the repo)
I got some errors on build if i didnt pass --linker=legacy. I assume roc was trying to mess with some objects in the nix store it didnt have permissions to update
Last updated: Jul 06 2025 at 12:14 UTC