Brendan Hansknecht (Jan 14 2024 at 01:57): Brendan Hansknecht (Jan 14 2024 at 01:57):I think that we should add an init function to roc code.
I want to keep this proposal with a very small scope. This proposal is just to deal with some linking pains that current roc causes. Any sort of static global initialization and such is out of scope. In the future, that is a possibility, but it is ignored for now.
The fundamental reason for this change is that roc_alloc, roc_panic, etc exist. In other words, it is because roc depends on calling into the host. There are also all of the roc_fx_* functions, but those will go away when we switch to effect interpreters.
When it comes to shared libraries, it is not normal for a shared library to call into a host. As such, when compiling, hosts assume this is impossible and DCE a lot of things. To work around the problem in basic cli, we have this monstrosity. Recently with @Luke Boswell, I was trying to do the same in the roc compiler for some glue changes. We end up in a painful situation where we get weird compiler crashes.
example crash
The roc compiler is harder to shoehorn in these symbols because it is a much larger and more complex code base. Not to mention, Rust is a lot more finicky around these features when trying to emit a static binary like via musl.
All in all, this is essentially not a supported feature and I wouldn't be surprised if it totally fails with certain security settings or on specific operating systems or rust updates.
To fix this pain, I think that roc should never call into the host except via function pointers. My suggestion is that we expose a roc_init function that takes all of these pointers as input and sets some globals on the roc side. Roc applications will no longer directly depend on the host functions. Instead it will be though function pointers. (and in the future through returning commands to the effect interpreter).
My only real concern with this idea is that I am not sure if/how it works with wasm.
Brendan Hansknecht (Jan 14 2024 at 02:04):I need to double check, but I am pretty sure that stripping a host binary also breaks the ability of a shared lib to depend on functions in the host.
Richard Feldman (Jan 14 2024 at 02:07):what about the idea of passing an allocator struct around, like Zig does?
Richard Feldman (Jan 14 2024 at 02:07):what I like about that is that it makes Roc even simpler and more portable
Richard Feldman (Jan 14 2024 at 02:08):you just pass in the struct of function pointers when you call the Roc function from the host, no linking or init or anything like that involved
Richard Feldman (Jan 14 2024 at 02:08):(and then we compile to passing it around behind the scenes)
Brendan Hansknecht (Jan 14 2024 at 02:08):Sure, that is fine too. It is theoretically slightly more cost, but shouldn't matter in practices.
Richard Feldman (Jan 14 2024 at 02:09):yeah I think Zig has proven it's fine from a perf perspective :big_smile:
Richard Feldman (Jan 14 2024 at 02:09):they very directly have done that experiment!
Brendan Hansknecht (Jan 14 2024 at 02:09):Well today, it would be an allocator + every effect struct.
Brendan Hansknecht (Jan 14 2024 at 02:10):But in general effects should be much slower than the cost of the function call. And it is just a pointer that always consumes a register passed through all roc functions.
Richard Feldman (Jan 14 2024 at 02:10):true, although (A) that'll change after effect interpreters and (B) if it's being passed around as a pointer to a struct anyway, probably doesn't matter haha
Richard Feldman (Jan 14 2024 at 02:11):but I do think that's offset by Roc's embedding story getting nicer
Richard Feldman (Jan 14 2024 at 02:11):also would mean we wouldn't have to do the stub stuff anymore, right?
Brendan Hansknecht (Jan 14 2024 at 02:12):stub which piece specifically?
Brendan Hansknecht (Jan 14 2024 at 02:12):the gen-stub-lib?
Brendan Hansknecht (Jan 14 2024 at 02:24):Oh, this also makes arena per async request trivial. I double want this now.
Richard Feldman (Jan 14 2024 at 02:31):oh wow I didn't even think of that!
Richard Feldman (Jan 14 2024 at 02:31):yeah that would greatly simplify using arenas with roc
Richard Feldman (Jan 14 2024 at 02:31):and actually probably improve their performance
Brendan Hansknecht (Jan 14 2024 at 02:34):As a note, as I am looking into this more, I think we would need to add a void * context parameter to roc_alloc and friends. The allocator struct we pass into roc would also contain the void *context.
Brendan Hansknecht (Jan 14 2024 at 02:34):This is to enable rust to for example, pass the arena in.
Brendan Hansknecht (Jan 14 2024 at 02:34):Cause a c abi friendly function pointer can't close over any data.
Brendan Hansknecht (Jan 14 2024 at 02:35):Otherwise, the rust would still be stuck with needing to figure out how to use globals to deal with distinguishing async threads to be able to use the right arena when calling roc_alloc
Brendan Hansknecht (Jan 14 2024 at 02:36):Cause you can't pass a pointer to rust closure that use the arena as roc_alloc
Brendan Hansknecht (Jan 14 2024 at 02:38):but anyway, that is the standard way to build this kind of api in c, so no big deal.
Brendan Hansknecht (Jan 14 2024 at 02:38):useful context/reading: http://blog.sagetheprogrammer.com/neat-rust-tricks-passing-rust-closures-to-c (though we wouldn't need destroy cause we always return to the owning rust context and roc can't keep the allocator alive extra long)
Brendan Hansknecht (Jan 14 2024 at 06:38):created #6382
Brendan Hansknecht (Jan 14 2024 at 06:39):That will be a lot of pipelining, but a really cool project once implemented.
Richard Feldman (Jan 14 2024 at 11:31):It needs to handle roc_dbg too, right? Not just allocation :big_smile:
Brian Carroll (Jan 14 2024 at 13:05):I don't see any concerns here for Wasm. If there are any specific questions let me know.
Romain Lepert (Jan 14 2024 at 14:03):Richard Feldman said:
true, although (A) that'll change after effect interpreters and (B) if it's being passed around as a pointer to a struct anyway, probably doesn't matter haha
Can I ask what are effect interpreters and where these have been discussed / presented ?
Brendan Hansknecht (Jan 14 2024 at 15:44):Brian Carroll said:
I don't see any concerns here for Wasm. If there are any specific questions let me know.
My only concern/unknown, can a js host pass a function pointer into roc wasm code?
Though now that I think of it, there is normally a layer of indirection. So it would be a zig wasm host passing a function pointer into roc, that must work, right?
Brian Carroll (Jan 14 2024 at 17:32):Ah I see. That will work fine.
Brian Carroll (Jan 14 2024 at 17:32):The Wasm module defines a set of named JS imports with expected type signatures, and they all get a function index. That function index is what you use to call it either directly, or in this case, indirectly. The index is how Wasm actually implements a "function pointer".
Brian Carroll (Jan 14 2024 at 17:33):So it works fine with JS functions.
Brian Carroll (Jan 14 2024 at 17:33):But also as you mentioned, in our JS hosts we often add a Zig layer in between the JS and the Roc.
Brendan Hansknecht (Jan 14 2024 at 18:02):Thanks for the clarification
Oskar Hahn (May 01 2024 at 09:31):@Brian Carroll I am not sure, if this will work with wasm. If you have a zig layer, then there will be no problem. But it would be nice, if you could build a Wasm module with Roc without zig.
When I understand it correctly, you have to do this in wasm with a Table. But you can not store JavaScript functions in a table. Only functions exported by Wasm can be stored in a table.
You can pass a data structure as described here to Wasm, when you replace the function with the Table indexes. But I don't see, how you can fill the Table with your functions?
Brian Carroll (May 01 2024 at 09:48):OK there are a few points to make here
1) When you "create a function pointer" in languages like C or Zig or Rust, and compile it to Wasm, the compiler "puts the function index into a table". And then if the C/Zig code "calls the function though a pointer" it compiles to the call_indirect instruction which takes the table index of that function. This is just how function pointers are implemented in the Wasm instruction set. There is no concept of "tables" C or Zig or JS or any other high-level language. It is a low-level WebAssembly concept.
2) If your C/Zig program declares an extern function, that will compile to an "import" in the Wasm module. And imports do have function indices, just like any other Wasm function. So you can put them into the table. If you want to express that in C/Zig you just take the address of that extern function like &my_imported_function.
3) Roc is _deliberately_ not capable of expressing low level concepts like function pointers because it is a higher level language. You are going to need some lower level language that compiles to Wasm because otherwise you just cannot express the concept of a function pointer.
it would be nice, if you could build a Wasm module with Roc without zig.
Doing it without Zig, but with C or Rust or something instead, is totally possible and "just a matter of doing the work".
But doing it without a systems-level language... I'm not sure. You definitely can't do function pointers. Maybe you could write a memory allocator in JS. It would definitely be hard. It might not be possible without breaking some core principle of Roc somewhere.
Oskar Hahn (May 01 2024 at 11:00):How hard it is to write an memory allocator in JS might depend on your use case. I found it quite easy to write an arena memory allocator in JS.
The proposal was created, so Roc does not have to call into the host, which fixes linking problems with shared libraries. The first idea was to solve this with an init function. The later idea was to pass the allocator around. This is a nice idea. It makes it possible to build arena allocators without threadlocal. But it comes at the cost, that it is no longer possible to build wasm modules without a system-level language.
Could you consider, if there are other ways to solve the problems, where it is still possible to compile Roc to a Wasm module without a host language?
To solve the arena problem, you could just pass around a Context. This would just be a pointer to something.
For the initial problem, Roc could provide two solutions. Either the current way, where the host exports the functions, or with an init function, that passes in function pointers. Of cause, this only works, if the compiler or linker can detect, if the host exports the functions. I don't know, if this is possible.
But maybe there are other ways to solve this. I think, there are a lot of interesting things you could do with Roc if you compile it to Wasm that would get harder.
Last updated: Jun 16 2026 at 16:19 UTC