Stream: compiler development

Topic: platform build instruction

Luke Boswell (Oct 21 2023 at 00:59):

I think we have a design for the platform to provide an instruction to roc so that it can build itself (all the object files etc) with the new module changes. However, until then we have hardcoded how to build zig platforms, and I wonder if we can find a quick and easy way to modify that to unlock some more functionality?

I was thinking about how we don't need that app module name any longer and wondered if we could in a similar vein, use the string in the platform name as the instruction to roc cli?

platform "zig build all" # use this as an intermediate solution until module changes??
    requires {} { main : _ }

This would mean I could write a zig platform using build.zig today that generated the pre-built platform objects for linux/mac/windows etc and I am hoping --bundle can then package to make a URL file.

Brendan Hansknecht (Oct 21 2023 at 01:12):

If you want something quick and super simple, I would just have roc check for a build.zig like we do with cargo.toml

Richard Feldman (Oct 21 2023 at 01:31):

I actually want to go in the other direction - I want to have roc no longer know about other programming languages

Richard Feldman (Oct 21 2023 at 01:32):

that was always intended to be a temporary let's-get-up-ad-running thing

Richard Feldman (Oct 21 2023 at 01:32):

so instead I'd prefer if the build.zig became the way to build

Brendan Hansknecht (Nov 21 2023 at 02:09):

I think I may want to work on this cause it is one of those old issues that is now starting to annoy me quite a bit.

fundamentally, we don't really want roc to control the full build process. We want it to really just point to a directory that has all the precompiled stuff we need. This is a package, just in an already extracted folder.

My thought on changes. I see two main options. One that is more convenient from a user perspective and another that is cleaner overall.

Option 1, more convenient for users:

1. Roc no longer compiles a platform with some bespoke intern magical command.
2. Roc will still be allowed to build local platforms for convenience, but it will always just be verbatim calling out to a platform specified command.
3. The platform also specifies a relative local folder for its output.
4. This output folder most look exactly like an extracted roc package.
5. Roc will call the specified command verbatim and then use that folder as if it was an extracted downloaded package.

Option 2, cleaner but less convenient:

1. Roc will never build a platform, ever.
2. The platform build must be run as a separate command.
3. This will output a local folder that looks just like an extracted downloaded package.
4. A roc platform can specify a local folder as it's platform path. If it does, that path will be used the same way a downloaded package would be used.

Option 2 is cleaner and simpler in the compiler, but it means that new users who want to run examples in the examples folder will need to manually compile the platform. So more readme docs and things won't just work with roc example/....roc

Either of these options remove all of our special platform building code and give platforms direct control over their build script. We still would keep the legacy linker and it hacky library includes. (note: we could maybe let the platforms specify extra linker flags that are required for the legacy linker to work)

Thoughts?

Brendan Hansknecht (Nov 21 2023 at 02:14):

With option 2, we probably also want to improve error messages to mention that a user needs to build the local platforms. Then give a readme in each local platform and probably a build.sh

Luke Boswell (Nov 21 2023 at 02:21):

I think Richard mentioned he thought a build.roc script might be a good default way to do this.

I think it is useful for roc to call this script, maybe we change --prebuilt-platform to be --rebuild-platform or something which just runs that roc script first to re-build the platform and then builds the app.

Is there a reason we need a special folder?

Can we have all the build artifects alongside platform/main.roc like we currently do? I imagine most build scripts could manage this.

Brendan Hansknecht (Nov 21 2023 at 02:29):

I'm against a build.roc script, it is extra complexity of no gain. No need for an extra platform. We have bash and other compilers build systems. They can manage this.

As for the special folder. I'm tired of everything being dumped in the platform folder. Build directories and caches are standard because they avoid filling source folders with arbitrary files that need to be gitignored. I want to enable output to be in zig_out for a zig platform. I think everything being dumped in the platform folder is legacy convenience and it should go away.

Richard Feldman (Nov 21 2023 at 02:36):

I think option 2 is the way to go

Richard Feldman (Nov 21 2023 at 02:38):

part of the reason for that is that if we try option 2 and it turns out to be a major pain point, then we can always reconsider incorporating additional functionality into roc

Richard Feldman (Nov 21 2023 at 02:38):

but if we try the other way around, we'll never know if we could have gone with the simpler design

Richard Feldman (Nov 21 2023 at 02:38):

another reason is that I think it makes it clearer for end users where errors and/or build slowness are coming from

Richard Feldman (Nov 21 2023 at 02:39):

one more reason is that it can be faster end to end; if you want to rebuild the platform every single time, then the extra step is slightly annoying (although not much if you do like cargo build && roc run at the command line)

Richard Feldman (Nov 21 2023 at 02:39):

but if you only want to rebuild the local platform once and then run the app on it a bunch of times, then it's definitely faster end to end to have them split up

Richard Feldman (Nov 21 2023 at 02:40):

and not have roc even attempt to run anything, like it does today

Richard Feldman (Nov 21 2023 at 02:41):

Brendan Hansknecht said:

I'm tired of everything being dumped in the platform folder. Build directories and caches are standard because they avoid filling source folders with arbitrary files that need to be gitignored. I want to enable output to be in zig_out for a zig platform. I think everything being dumped in the platform folder is legacy convenience and it should go away.

oh yeah 100% agree, that was never supposed to be the long-term design :big_smile:

Richard Feldman (Nov 21 2023 at 02:41):

and we have an --output flag now, so it would be natural to respect that when doing roc build on a platform

Richard Feldman (Nov 21 2023 at 02:42):

this is really exciting! I'm pumped to see this happen :smiley:

Richard Feldman (Nov 21 2023 at 02:42):

the status quo design is a hack from like...2019?

Richard Feldman (Nov 21 2023 at 02:44):

also I agree that the examples/ folder should use basic-cli instead of locally built platforms, with the exception of the one set of examples dedicated to showing how building platforms locally works (and those should have instructions in the README for how in order to build those platforms, you need to use cargo or zig or perhaps even make for C ones)

Brendan Hansknecht (Nov 21 2023 at 02:45):

Cool I think I am fully on the same page. Sounds nice!

Brendan Hansknecht (Nov 21 2023 at 04:20):

For local package dependencies, would it be preferable to just take a path to a folder or to have the platform main.roc specify the folder location? I currently assume path to folder.

Also, do we want a file:// prefix? I assume not, but just curious.

Richard Feldman (Nov 21 2023 at 04:22):

I think for local package dependencies it should be a path to main.roc

Richard Feldman (Nov 21 2023 at 04:22):

no file:// prefix

Richard Feldman (Nov 21 2023 at 04:22):

or rather, a path to whatever the .roc file is (whether it's named main.roc or not) that's the actual platform module in question

Brendan Hansknecht (Nov 21 2023 at 04:24):

Ok. so then we would need the platform to specify where it stores the output linux_x64.o and friends if we want to avoid them still all needing to be in the platform folder. That or we hardcode some other subfolder.

Brendan Hansknecht (Nov 21 2023 at 04:25):

So I guess this would still require a change like:

platform "zig_out/"
    requires {} { main : _ }

Richard Feldman (Nov 21 2023 at 04:27):

:thinking: actually, what if we used caching for this?

Richard Feldman (Nov 21 2023 at 04:27):

like we want to be taking the BLAKE3 hash of source files anyway, and then storing intermediate compiled values in a ~/.cache/roc (or similar) directory

Richard Feldman (Nov 21 2023 at 04:27):

wouldn't that be the natural place to put these?

Brendan Hansknecht (Nov 21 2023 at 04:28):

Not if zig or cmake or etc is controlling the build

Richard Feldman (Nov 21 2023 at 04:28):

ah true

Brendan Hansknecht (Nov 21 2023 at 04:28):

Also, I was assuming for local build we would just ignore the hash part

Richard Feldman (Nov 21 2023 at 04:29):

ok then yeah I think we should specify it in the platform module header

Richard Feldman (Nov 21 2023 at 04:29):

if desired, we could go straight to the design I think we want to eventually have, which is that the platform module header specifies on a per-target basis where to find its precompiled files

Richard Feldman (Nov 21 2023 at 04:29):

which specifies not only output directory, but also filename, and then also (perhaps most importantly) which targets it actually supports

Richard Feldman (Nov 21 2023 at 04:30):

which would address current problems like basic-cli doesn't support wasm, but there's no way to know that - let alone to get a helpful compiler error if you try to build it with --target wasm32

Brendan Hansknecht (Nov 21 2023 at 04:31):

Yeah, I guess it depends if we want to make that explicit or implicit. My thought was that it would specify an artifact folder. We would then check that folder for our expected files like linux_x86.o. If our expected file doesn't exist, we would print an error about needing to read the platform readme and compile it for linux x86.

Richard Feldman (Nov 21 2023 at 04:31):

concretely, I'm talking about something like

platform …
    targets {
        linux-x64 => "zig_out/linux-x64.rh",
        wasm32 => "zig_out/wasm32.rh",
    }

Richard Feldman (Nov 21 2023 at 04:32):

I think it's nicer for this to be explicit

Brendan Hansknecht (Nov 21 2023 at 04:32):

Fair.

Brendan Hansknecht (Nov 21 2023 at 04:32):

though what if the platform supports both legacy and surgical linker

Brendan Hansknecht (Nov 21 2023 at 04:33):

Like legacy as fallback or to get better debug info.

Brendan Hansknecht (Nov 21 2023 at 04:35):

I guess we could also just make that explicit:
linux-x64-legacy or something.

Richard Feldman (Nov 21 2023 at 04:36):

hm, what would the distinction be there? :thinking:

Richard Feldman (Nov 21 2023 at 04:36):

like you mean they'd build two different .rh files?

Richard Feldman (Nov 21 2023 at 04:37):

for the same target

Brendan Hansknecht (Nov 21 2023 at 04:37):

one .rh and one .a or .o

Brendan Hansknecht (Nov 21 2023 at 04:37):

That is what you will see in basic cli for linux_x64 currently for example.

Richard Feldman (Nov 21 2023 at 04:37):

hm but why would they need to build different ones?

Richard Feldman (Nov 21 2023 at 04:37):

because it would have something that breaks the surgical linker but works for the legacy linker?

Brendan Hansknecht (Nov 21 2023 at 04:39):

two main reasons currently:

1. just in case some specific use of the platform breaks the surgical linker
2. to get debug info for the platform and roc app (surgical linker doesn't support that currently).

Long term, everything should be fine to only be the surgical linker version, but I don't think were fully there yet.

Richard Feldman (Nov 21 2023 at 04:40):

hm I'd say let's not introduce the complexity and revisit if it actually comes up in practice

Richard Feldman (Nov 21 2023 at 04:41):

like if it's a problem they can (for example) always build for legacy linker and tell people to use that

Richard Feldman (Nov 21 2023 at 04:41):

but I really would like 2023 to be the last year we have --linker=legacy if possible :big_smile:

Brendan Hansknecht (Nov 21 2023 at 04:41):

It does. I use --linker=legacy all the time with gdb when debugging platform glue issues and similar.

Richard Feldman (Nov 21 2023 at 04:42):

oh sure, but if you're the platform author you can just build for that locally right?

Richard Feldman (Nov 21 2023 at 04:42):

I'd think the inconvenience of only having one would mainly apply to people getting the platform from a URL

Brendan Hansknecht (Nov 21 2023 at 04:42):

I guess, just have to update a single line from linux-x64 => "zig_out/linux-x64.rh" to linux-x64 => "zig_out/linux-x64.o"

Brendan Hansknecht (Nov 21 2023 at 04:44):

That or compile the roc app to a lib and just avoid the roc linking step all together instead letting the host compiler deal with that linking as well.

Richard Feldman (Nov 21 2023 at 04:45):

yeah, either seems fine!

Brendan Hansknecht (Nov 21 2023 at 04:46):

Ok. Sounds good.

Richard Feldman (Nov 21 2023 at 04:46):

as an aside, we should really spec out what adding debug info would look like - seems like maybe something where if we got it started, could be a cool first contribution to that part of the code base for someone looking to get involved!

Brendan Hansknecht (Nov 21 2023 at 04:47):

For sure. If debug info is concatenative, it may be possible that the surgical linker just needs to also copy it over and then update some offsets. No idea.

Richard Feldman (Nov 21 2023 at 04:48):

that would be sweet if so!

Brendan Hansknecht (Nov 21 2023 at 04:50):

One more question on the targets syntax, should it support host or current-system or whatever we want to name it?

I assume that would be useful in platforms that don't know how to cross compile. Instead they just build for the host system when built locally.

Brendan Hansknecht (Nov 21 2023 at 04:55):

Oh, another question I just realized, should packages now be expected to also follow the structure specified in the targets syntax? Should a zig platform potentially extract to be zig-out/linux-x64.o? That way we can still read the main.roc file to understand where to find these files? No concerns of the platform following our exact naming conventions.

Actually as I am thinking about this, I guess that roc build --bundle (probably should change to roc bundle cause it won't build anymore) can read the platform/main.roc and just copy all of the platform specified files to exactly where we want them in the final bundle.

Richard Feldman (Nov 21 2023 at 04:59):

yeah that makes sense!

Richard Feldman (Nov 21 2023 at 04:59):

(regarding roc bundle)

Richard Feldman (Nov 21 2023 at 04:59):

I think the keys to the mapping should follow our --target syntax, but the values can be whatever the author wants

Richard Feldman (Nov 21 2023 at 05:00):

like I could do linux-x64 => zig_out/x86_64-musl-linux.rh if I wanted to

Brendan Hansknecht (Nov 21 2023 at 05:01):

Ok. So I guess the scope of this work is:

1) Update all platforms in the roc repo to have their own build scripts and readme (later update basic-cli and basic-webserver)
2) Add the target syntax to enable platforms to specify where they store artifacts

    targets {
        linux-x64 => "zig_out/linux-x64.rh",
        wasm32 => "zig_out/wasm32.rh",
    }

3) update roc build with a local platform to just grab the specified file instead of rebuilding the platform.
4) update roc build --bundle to grab the specified file for each target instead of just assuming they will be in the platform directly as linux_x64.o, etc. It can build the exact expected directly in ~/.cache/roc if it doesn't already.
5) extract roc build --bundle into a standalone roc bundle
6) add nicer error messages about unsupported targets for platforms and unbuilt targets for platforms.

Richard Feldman (Nov 21 2023 at 05:03):

Brendan Hansknecht said:

One more question on the targets syntax, should it support host or current-system or whatever we want to name it?

I assume that would be useful in platforms that don't know how to cross compile. Instead they just build for the host system when built locally.

hm, I'm not sure if it would matter in practice, because:

• if I'm an application author, I get to cross-compile regardless, because all that matters is whether the target I specified has a precompiled host
• if I'm a platform author, it's my responsibility to generate a .rh file for each of those targets one way or another, and from roc's perspective, it doesn't really matter whether I need to use multiple operating systems to generate them

Richard Feldman (Nov 21 2023 at 05:04):

that plan sounds great! :heart_eyes:

Brendan Hansknecht (Nov 21 2023 at 05:05):

Aside, we may eventually want a roc bundle --merge command that would enable a platform author to generate bundles for each os (if they can't cross compile) then merge all of the os specific bundles into one generic os bundle.

Richard Feldman (Nov 21 2023 at 05:09):

interesting!

Brendan Hansknecht (Nov 21 2023 at 05:11):

just sounds more convenient than wrangling all of the .rh and .rm and .o etc files and dumping them into the same directory on a single machine. Instead, I just run github ci on all of my target systems. They all generate a bundle. I have a follow up step that runs merge and publishes.

Richard Feldman (Nov 21 2023 at 05:13):

hm, but a bundle is just a compressed tarball plus hashing the final thing

Richard Feldman (Nov 21 2023 at 05:14):

so I'm not sure if that saves any effort compare to just gathering all the binaries and running bundle afterwards :big_smile:

Richard Feldman (Nov 21 2023 at 05:14):

as in, gathering bundles and gathering binaries seems like the same amount of effort

Brendan Hansknecht (Nov 21 2023 at 05:15):

Hmm, yeah probably. I guess it just is a little more folder structure and maybe two files instead of one. Yeah, probably pretty minor work. nvm

Brendan Hansknecht (Nov 21 2023 at 05:20):

Filed tracking issue #6037

Brendan Hansknecht (Nov 21 2023 at 06:34):

what are thoughts on using zig as the compiler for the example c platform such that it gets super simple cross compilation?

Though, in general, I guess that example platforms don't really need to be able to cross compile. They can just be compiled for the current host machine cause they will never be distributed.

Brian Carroll (Nov 21 2023 at 07:36):

There's some benefit in having a C platform with a Make or CMake build, because it shows Roc can be embedded into your current project that uses that build system.

Brendan Hansknecht (Nov 21 2023 at 15:02):

That's fair. Though I'm not sure I want to go through the hassle of cmake instead of just having a build.sh. though cmake also deals with windows, so maybe that is better.

Last updated: Jul 06 2025 at 12:14 UTC