Stream: compiler development

Topic: roc build

Brendan Hansknecht (Oct 26 2025 at 17:00):

Question on roc build with the interpreter. It basically should be the same as roc run except that it will

1. build the roc module env and serialize it via llvm to a global constant in an object file
2. use a different version of the interpreter shim that loads from this global instead of from shared memory
3. link all of those together into a standalone binary

Does this sound correct?

Richard Feldman (Oct 26 2025 at 17:14):

yeah that sounds right to me!

Brendan Hansknecht (Oct 26 2025 at 18:48):

For our ipc, I see parent_base_address. Trying to makes sure I understand this correctly. Instead of serializing in a pointer free format, we serialize with pointers, but then shift them between the two different address spaces. Is that what is happening here?

Trying to understand the ramifications of serializing to a slice of bytes instead of to a shared mem allocator.

Brendan Hansknecht (Oct 26 2025 at 18:48):

Something just feels a bit off to me

Notification Bot (Oct 26 2025 at 18:52):

4 messages were moved here from #contributing > Worklog (Draft PRs and coordination) by Richard Feldman.

Richard Feldman (Oct 26 2025 at 18:55):

so the basic procedure is:

• we have a fixed number of pointers in the ModuleEnv data structure (some of which live in nested data structures like SafeList and SafeMultiList) - e.g. maybe it's like 20, I'm not sure what the exact number is, but it doesn't vary by what's in the module, it's always the same exact number no matter what
• to serialize, we go through each of those pointers and subtract the module env's base address. Once that's done, they are no longer memory addresses, but rather memory offsets
• to deserialize on the other side, we start with our freshly-created (but not yet initialized) ModuleEnv's base address, and go through each of those memory offsets and add that base address - thereby converting them back into valid pointers (except in the deserialized address space instead)

Richard Feldman (Oct 26 2025 at 18:55):

does that clarify? Happy to answer any follow-up questions about it!

Brendan Hansknecht (Oct 26 2025 at 19:04):

That overall makes sense. I still don't quite understand why we serialize: parent_base_addr: u64, over the IPC. What is the parent in this case? the entire module+header?

Richard Feldman (Oct 26 2025 at 19:08):

I forget offhand, but I think it's the base address of the ModuleEnv

Brendan Hansknecht (Oct 26 2025 at 19:45):

Oh cool, got it working!

Brendan Hansknecht (Oct 26 2025 at 19:45):

That was surprisingly easy

Brendan Hansknecht (Oct 26 2025 at 19:45):

That said, right now I have a hacked rocRun that builds an exe

Brendan Hansknecht (Oct 26 2025 at 19:45):

Need to make it a separated updated rocBuild path.

Brendan Hansknecht (Oct 26 2025 at 19:46):

❯ /Users/bren077s/Library/Caches/roc/f0bdb35f6bc3514a3a4574c23848dd0b/temp/roc-tmp-1FrVoAORWsZgVE1nyj6dkXtYhYHVDVuB/roc_run_2271902749
Generated numbers: a = 60, b = 99

=== Testing addInts (entry_idx = 0) ===
roc build with serialized
module_env offset: 536
parent_base_addr: 4575952896
child_base_addr: 4334715136
offset: -241237760
Expected add result: 159
Roc computed add: 159
SUCCESS: addInts results match!

=== Testing multiplyInts (entry_idx = 1) ===
Expected multiply result: 5940
Roc computed multiply: 5940
SUCCESS: multiplyInts results match!

=== FINAL RESULT ===
ALL TESTS PASSED: Both entrypoints work correctly!

Luke Boswell (Oct 26 2025 at 21:43):

Niiice :grinning_face_with_smiling_eyes:

Brendan Hansknecht (Oct 27 2025 at 00:08):

For roc build, instead of ever touching shared memory, I should be able to:

1. create the module env in standard gpa/arena.
2. create a new arena (or block of bytes) for what will be serialized.
3. write the header at the start of the area
4. serialize the module env into the bytes

Right?

So it will be a little bit of rearchitecting to do that, but nothing too bad. Probably the biggest hassle is that I want to factor it well instead of duplicating the code as a slight variant.

Brendan Hansknecht (Oct 27 2025 at 00:08):

Nothing hard, just cleanup/organization.

Richard Feldman (Oct 27 2025 at 00:21):

sounds right, yeah!

Brendan Hansknecht (Oct 27 2025 at 00:49):

Also, I am a little confused that our current module even shared memory thing. It doesn't actually call moduleenv.serialize. I guess it kinda just adhoc deals with everything instead to avoid an extra copy to serialize.

Brendan Hansknecht (Oct 27 2025 at 01:35):

Also, hmm. I guess an arena is not guaranteed to be contiguous given it is a linked list of blocks. So either I need to still used the shm allocator.

I guess using the shm allocator is actually the minimal amount of copying (though not sure if it would include some extra unneeded data being serialized).

Cause shm allocator is:

1. writing to the allocator
2. write to disk via llvm.

Vs gpa/arena would be:

1. write to allocator
2. serialize to flat array of bytes
3. write to disk via llvm.

Brendan Hansknecht (Oct 27 2025 at 01:36):

So assuming the shm allocator isn't accidentally including extra bytes or otherwise significantly costly to create. It likely is the nicest way to handle all of this.

Brendan Hansknecht (Nov 03 2025 at 19:53):

Just an update: I still just have the prototype working on a branch. It is a hacked roc run that builds and runs. Hoping to get back to his later this week or early next.

Since I am planning to anchor to shared mem, I think it will make the refactoring to land pretty small. Mostly need to enable running the roc app compilation step at two different potential locations. For build it has to run before all the linking. For run. It should run after (or eventually in parallel)

Luke Boswell (Nov 04 2025 at 00:58):

So hyped for this. Will be really nice to unlock wasm based platforms.

Last updated: Nov 08 2025 at 12:13 UTC