Brendan Hansknecht (Feb 02 2025 at 01:14): Brendan Hansknecht (Feb 02 2025 at 01:14):What is our plan for actually building and linking platforms? I feel like that story is still lacking quite a bit today.
In general, what do we think the starting point should be and what do we think the end target should be?
With the interpretter as our default debug interface, linking performance matters a lot less. We theoretically don't need the surgical linkers anymore. That said, the surgical linkers have another huge win in terms of dependency management. They enable us to avoid managing dependencies during linking. As a note, we can get the same thing without the surgical linker by compiling roc to a shared library. The executable will be precompiled and roc would just be a shared library that it loads.
I just want to make sure we start off with the right footing here. I feel like this is an area that has potential to really hinder roc. The surgical linker is not robust. Well it does generally work, making it robust on all platforms is not clear (it should work in theory, but it might be tons of segfaults and work, also, fixing it may really hurt perf).
Brendan Hansknecht (Feb 02 2025 at 01:15):Especially with the focus of the zig compiler being on correctness first, I'm wondering if there is a way we could avoid the surgical linker while still making a nice experience.
Luke Boswell (Feb 02 2025 at 01:31):we can get the same thing without the surgical linker by compiling roc to a shared library. The executable will be precompiled and roc would just be a shared library that it loads.
:plus:
Brendan Hansknecht (Feb 02 2025 at 01:32):Note, the main cost of doing that is that you now can't distribute a single file, you now have do distribute two files (./host and ./libapp.so)
Luke Boswell (Feb 02 2025 at 01:35):Could these be limited to within the platform package though, and roc somehow link the two together to produce a single executable after roc build?
Brendan Hansknecht (Feb 02 2025 at 01:36):and roc somehow link the two together to produce a single executable after
roc build
That would be the surgical linker
Luke Boswell (Feb 02 2025 at 01:37):There's a lot of nice things about the surgical linker :smile:
Brendan Hansknecht (Feb 02 2025 at 01:37):There may be some sort of simple runtime way to unpack and deal with this, but not really sure.
Luke Boswell (Feb 02 2025 at 01:39):I guess the simplest thing is
Brendan Hansknecht (Feb 02 2025 at 01:40):I'm not worried about the interpretter for this, pretty sure I know how to make that work (basically platform -> shim shared library that loads and tags data -> roc compiler loaded as a shared library to launch the interpreter).
Brendan Hansknecht (Feb 02 2025 at 01:40):generate an object using gen-llvm and link with a prebuilt-host that is a static library
The big question is, if we do this, how do we deal with the platform dependencies?
Luke Boswell (Feb 02 2025 at 01:45):I think the whole platform/app abstraction gives us a really nice opportunity for platform authors to provide a domain specific dev experience.
Basically, I think we flip things around from the traditional roc is a compiler and drives things, to roc the compiler is a library used by the host. I'm assuming here that bundling just the parts needed for interpreting roc (parsing, typechecking etc) is very slim.
A platform author like basic-cli could include the "roc interpreter" into it's host and bundle a separate executable in the package that conforms to some specific cli interface -- so that it can be instructed to parse an app and startup an interpreter.
So running roc app.roc -- the roc cli simply finds the basic-cli package files in cache and then says to the basic-cli interpreter binary "hey here's an app, go do interpreter things with it".
Luke Boswell (Feb 02 2025 at 01:47):The platform author then has a nice abstraction to work with, that will parse the roc code and provides a nice abstraction of the roc program, I imagine our effect interpreter like state machine or something.
Luke Boswell (Feb 02 2025 at 01:48):The platform author can then build whatever "interpreter" or "debugging" like experience around that makes sense in their world.
Brendan Hansknecht (Feb 02 2025 at 01:48):So like how lua jit gets embedded into game engines? Basically you load lua as a c library and then tell to to go run a function from a file with specific args.
Brendan Hansknecht (Feb 02 2025 at 01:49):This works and is a nice embedded experience, but I'm not sure how we mesh it with the llvm optimized experience. Theoretically we want those to both be seamless from the app author perspective.
Luke Boswell (Feb 02 2025 at 01:49):Maybe, I don't know anything about lua.
I just feel like there is a use case here that gives us nice interpreter, debugging, tooling, etc and simplifies our platform things too.
Brendan Hansknecht (Feb 02 2025 at 01:50):but I'm not sure how we mesh it with the llvm optimized experience
This is the main sticking point of that flow.
Luke Boswell (Feb 02 2025 at 01:52):llvm optimized experience
What does this mean? how does it relate to a dev loop or interpreter?
Brendan Hansknecht (Feb 02 2025 at 01:53):A platform has to work both with the interpreter and the llvm flow. What you have describe above only works with the interpreter and would not work with the llvm flow
Luke Boswell (Feb 02 2025 at 01:55):Brendan Hansknecht said:
generate an object using gen-llvm and link with a prebuilt-host that is a static library
The big question is, if we do this, how do we deal with the platform dependencies?
You're referring to this right?
Is there something different about this new world compared to current situation?
Brendan Hansknecht (Feb 02 2025 at 01:56):You're referring to this right?
yes
Is there something different about this new world compared to current situation?
Not specifically, but the current solution of link.rs is terrible.
Luke Boswell (Feb 02 2025 at 02:08):I'm wondering if there is a way we could avoid the surgical linker while still making a nice experience.
Random idea... could we have some kind of stubbed out facade that a platform author uses. And that "interfaces" with an interepreted version of the roc app somehow?
I made a diagram... just because I'm already looking at diagrams
Brendan Hansknecht (Feb 02 2025 at 02:09):could we have some kind of stubbed out facade that a platform author uses
This?
Brendan Hansknecht said:
I'm not worried about the interpretter for this, pretty sure I know how to make that work (basically platform -> shim shared library that loads and tags data -> roc compiler loaded as a shared library to launch the interpreter).
Luke Boswell (Feb 02 2025 at 02:19):We've gone through the process of mostly mapping out the desired workflows in that massive preprocess-host PR.
It'd be good to look at those out, and try to make some simplified diagrams which shows the different ways we can use roc (dev, embedded, optimised, prebuilt host, interptreted ...)
Luke Boswell (Feb 02 2025 at 02:20):Though, depending on where this conversation goes... the linking strategy might be very different
Richard Feldman (Feb 02 2025 at 03:23):so here are some things that are important to me:
roc build with no arguments and have it output a single executable file which can then be distributed. This rules out designs where we do things like have the platform build an executable that loads a dylib and then roc only knows how to compile to dylibs.roc build output a dylib for some platforms (because they opted into that, not because it's the only option), and another plausible way to do it is to have some way for hosts to call a Roc interpreter at runtime. Luke Boswell (Feb 02 2025 at 04:30):Some notes...
Platform has a string for roc to build lld libhost.a app.o -l raylib -l c
Roc cli bundles a linker “lld”, and the platform gives roc a list of dynamic libraries roc should link with. We can require the platform provide an explanation of how to go get dynamic dependency to provide nice error messages.
The platform should specify different build options in platform.roc header - NO_LINK, LINK (deps..), different options per target, use WASM_LD etc
interpreter.o and link that with the host.Required to use a fast dev loop with any platform. Linked once and ran multiple times. We need a away to tell the interpreter where the .roc files are to interpret, we hardcode the path to the app files when we build -- so one copy for each project.
Host passes into roc a function that is used to load files… roc_load : OsStr -> List U8.
libroc) – platform embeds libroc interpreter and uses it to provide an interpreted/debugging experienceWe will probably need something like this to provide the REPL, online playground… experience
This is the ultimate fallback where platforms can drive everything.
Brendan Hansknecht (Feb 02 2025 at 04:42):Just for context for others, this note list come from a call between Luke, Richard and I.
Jasper Woudenberg (Feb 02 2025 at 08:44):Luke Boswell said:
Basically, I think we flip things around from the traditional roc is a compiler and drives things, to roc the compiler is a library used by the host. I'm assuming here that bundling just the parts needed for interpreting roc (parsing, typechecking etc) is very slim.
Having this as a linking option would be super exciting to me! For Jay, the static site generator platform I'm working on, the host is essentially a dev-server, and this approach would allow that dev-server to be started even if the app doesn't compile yet. I could then show Roc compilation errors in the browser-preview of the static site being worked on instead of the terminal.
Another potential usecase: an integrated full-stack platform where some of the Roc code of the app ends up in a binary used to run the server, and some ends up in a webassembly file served by the server as part of a client. The host would contain code calling libroc to build the full artifact that can be deployed. Similar to Jay the host might also want to offer some type of dev-server that serves a development-version of the app and shows Roc compilation errors in that browser page, similar to what Rails does.
Richard Feldman (Feb 02 2025 at 12:27):I talked with Andrew some more about this and realized that the (probably most common) use case of having roc build emit a standalone executable requires a bit more from the platform author than just having them provide a static library.
specifically, we need to:
main and is built with PIE) instead of an executablecrt0, crt1, crti, and crtn) that their host will be using .roc file) the order in which to link the (bundled) static and dynamic libraries, not just the dynamic ones as we'd previously discused Richard Feldman (Feb 02 2025 at 12:31):our current link.rs does this automatically - first it finds the C runtime libraries
Richard Feldman (Feb 02 2025 at 12:31):and then adds them into the link path:
Richard Feldman (Feb 02 2025 at 12:35):the reason we need to do it like this is:
lld to create an executable, these static c runtime libraries must be part of the final link; they can't be done in advance unless you're making a full executable (e.g. platform authors can't "pre-link" them like they could with surgical linking)roc compiler to save platform authors from needing to do this would limit what hosts are possible (because everyone would get the same c runtime for a particular target and it couldn't be customized even if they needed a later version) Richard Feldman (Feb 02 2025 at 12:37):there are some fortunate parts about this:
lld to garbage-collect sections after doing all this, so the final output binary can be smaller than today Brendan Hansknecht (Feb 02 2025 at 16:39):Yeah, that sounds about right. The big question is does that make platform dev too painful? Like how hard would it be for us to add all of these things to basic cli?
Richard Feldman (Feb 02 2025 at 16:46):I don't think it'll be a big problem. We already want to move host building into the platform's own build system, so we can just add "find those files and add them to the bundle" as part of build.rs/build.zig
Richard Feldman (Feb 02 2025 at 16:47):so once we get that recipe set up, should be fine I think!
Richard Feldman (Feb 02 2025 at 16:59):actually there's nothing blocking us from trying that out right now
Richard Feldman (Feb 02 2025 at 17:01):the steps would be:
Brendan Hansknecht (Feb 02 2025 at 17:33):Yeah, I think the big pain is that these extra files are generally automagically found by rust and given to the linker. Even if a platform could copy the exact link command for their system by stealing from the verbose linking output, you still have to know how to find all those files in general (but we can do the work to write up the code to find these common files)
Richard Feldman (Feb 02 2025 at 17:43):agreed!
Oskar Hahn (Feb 02 2025 at 17:44):Will it be possible to do this in other languages? For example go?
Brendan Hansknecht (Feb 02 2025 at 18:32):Hmm.... I bet for a lot of compiler toolchains that are high level, they won't make this easy to do (if possible at all).....hmm
Brendan Hansknecht (Feb 02 2025 at 18:32):We definitely need to validate this before making final decisions. The problems of being an embedded language but trying to drive things as if we aren't embedded.
Richard Feldman (Feb 02 2025 at 18:32):how do we do that today?
Richard Feldman (Feb 02 2025 at 18:33):I assume we don't require the surgical linker
Brendan Hansknecht (Feb 02 2025 at 18:33):Does go support the legacy linker today? I'm actually not sure.
Brendan Hansknecht (Feb 02 2025 at 18:33):I think it may just be the surgical linker or be completely driven by golang....we should dig into this
Brendan Hansknecht (Feb 02 2025 at 18:33):Maybe @Luke Boswell knows
Brendan Hansknecht (Feb 02 2025 at 18:43):Oh, I see -buildmode=c-archive in luke's script, so seems doable, but that might subvert go's main...so not sure how it is all wired together.
Brendan Hansknecht (Feb 02 2025 at 18:47):Yeah, does subvert go's main, but I think it can all be made to work. I think for a lot of platform languages, we will be doing the c hack (or more likely zig hack cause zig is more portable). So in many high level languages, they will need to compile their app to a static library without a main. Then they will need to put the main in zig or c or something low level. Finally, that can collect all of the extra special files we depend on.
So subverts the other languages main, but is probably fine for the most part.
Brendan Hansknecht (Feb 02 2025 at 18:48):Though I'm pretty sure this subverted main was part of the reason rust basic-cli breaks with musl builds. musl doesn't have automatic init functions. So if main is subverted any sort of init is just skipped. In the case of rust, this leads to rust not having access to any cli args unless they are manaully piped in.
Brendan Hansknecht (Feb 02 2025 at 18:49):So I do think this system can be made to work with any language by making them a library and giving them a c/zig main function, but it definitely is a bit inconvenient on the platform dev side (luckily, it should just be setup once per language and then mostly just work)
Brendan Hansknecht (Feb 02 2025 at 18:50):So in summary:
Will it be possible to do this in other languages? For example go?
Yes, but it may require subverting main and having a c/zig shim
Last updated: Jul 06 2025 at 12:14 UTC