Stream: beginners

Topic: Passing a lambda to a platform function - does it cross ABI?

Karakatiza (May 26 2026 at 19:30):

Hi! I am poking at lower-level stuff for the first time, so apologies if this is obvious.

I'm working with a Zig-based nightly 2026-04-10 on a platform experiment (Rust host) that wraps a data-processing engine over an extern "C" ABI. The host does the heavy lifting - running processing loops over streams of records - and I'd like user Roc code to define the per-record transformations (let's ignore batching for now). So the natural API I reached for was a platform method that takes a lambda, something like:

  Builder := [].{
      map_records! : Stream, (Row -> Row) => Stream
  }

and then in an app you'd write Builder.map_records!(stream, |row| { ...transform... }), and my host would call that lambda once per record.

When I tried it, the lambda seems to just... disappear. The host method gets called fine, but it only receives the non-lambda arguments - the (Row -> Row) never shows up on the host side, and the lambda body doesn't seem to run at all. I even swapped |row| row for |row| transform(row) and the compiled output was byte-for-byte identical, so it looks like the lambda argument is dropped entirely at the boundary.
I also tried to pass a top-level function, but it seemingly got elided, too. As a control, passing a simple string works as expected, so this seems to affect any function-like object.

So my questions:

1. Is passing a lambda or a function directly to a platform function a supported thing? If yes, is there a particular way I need to declare it on the platform side (some special type, or a host_* form) so the host actually receives something callable?
2. If it's not meant to work that way, is the right pattern instead to have the app provide a function that the host calls into - the way basic-webserver's respond! works?The only workaround I can think of is having the app expose a single dispatcher keyed by an integer id, roughly:

  run_transform! : U64, Row => Row
  run_transform! = |id, row|
      when id is
          0 -> ...first transform...
          1 -> ...second transform...
          _ -> crash "unknown id"

and my host stores an id per operator, then calls back into run_transform! with that id whenever it needs to transform a record. The user-facing lambdas would get rewritten into branches of this one function under the hood.

Basically: can I pass lambdas or functions into the platform, or should I flip it around so the app provides a callback the host invokes? And if the second is the way, is there anything I should know about doing it efficiently, since this runs on a hot path (potentially hundreds of thousands of records per second)?

Thanks!

Richard Feldman (May 27 2026 at 02:46):

welcome! You're really diving straight into the deep end on this one...I like it! :smile:

Is passing a lambda or a function directly to a platform function a supported thing? If yes, is there a particular way I need to declare it on the platform side (some special type, or a host_* form) so the host actually receives something callable?

yeah, they have to be boxed right now - as in, Box(Row -> Row) instead of (Row -> Row)

Richard Feldman (May 27 2026 at 02:52):

some notes on this:

• the challenge with passing unboxed lambdas across the host boundary is that having a host ABI where the shape of your data structures varies based on what the application happens to close over (which you can't possibly know when you're designing your host ABI) creates a huge amount of problems. We went down a rabbit hole trying to come up with a nice design (which led to fun discoveries like "any record containing a lambda in any of its fields no longer has stable ordering, since record fields are sorted by alignment, and the alignment of a lambda can also change based on what the application happens to close over..." which we tried to work around by generating elaborate "getters and setters" for records and lambdas based on what the application closed over, which then led to problems of even just the names of those getters and setters being hard to generate, given anonymous records...)
• ultimately we decided "yeah let's just say that for now you can only pass heap-allocated closures to/from the host, and we can revisit stack-allocated closures across the host boundary in the future" - and Box is the way you opt into that.
• a lambda inside a Box has a stable size and alignment regardless of what it captures, because the capture is heap-allocated (which is the downside and why we make it opt-in to box closures)
• I think ideally a host would offer an unboxed API to the application author, so they don't see the boxing behind the scenes, and then just use the Box to send the lambda to/from the host
• boxed lambdas landed very recently in the new compiler, and although we have tests, I don't think anyone has actually tried to use them in a real platform yet, so feel free to leave a message here if you run into anything weird or have any questions about them! :smile:

Richard Feldman (May 27 2026 at 02:54):

as an aside, we should definitely give a compile-time error when you try to use something unsupported (like unboxed lambdas) in the host boundary, but we don't have any validation in place for that yet - sorry about that!

Richard Feldman (May 27 2026 at 02:55):

as far as your high-level question about approach, I think you have the right idea! in terms of performance, I'd expect boxed lambdas to be totally fine for the use case you just described - like the app says "here is my lambda to transform this data" and then the platform hands that off to the host, which runs it a gazillion times

Richard Feldman (May 27 2026 at 02:57):

another thing to note: until this PR lands (I expect it to land in the next few days), LLVM optimizations are currently disabled, so roc --optimize won't work and you won't get the performance that you'll get once that lands.

Richard Feldman (May 27 2026 at 02:57):

anyway thanks for the detailed write-up @Karakatiza and lmk if you have any other questions!

Luke Boswell (May 27 2026 at 04:38):

My plan to test out the recently landed Boxed lambdas was to start poking at https://github.com/lukewilliamboswell/roc-ray again ... but it's currently on the back burner as I've been focussed on Zig 16 upgrade and also helping Richard with things.

Luke Boswell (May 27 2026 at 04:39):

So I don't think anyone has a platform currently which demonstrates that functionality we can point at.

Karakatiza (May 27 2026 at 12:04):

Thank you for such a detailed reply! I had to compile Roc from the latest main, seems to work so far! Syntax Box((Str -> Str)) worked for me, single parentheses failed to compile. I assume the heap allocation is not too much of an overhead because of CPU caching, but we'll see. I'll share if I reach any meaningful result =)

As a part of this experiment I'm building a largish Roc platform + library - there are ~500 small helper functions plus a ~70-method "builder" pattern type. Working through different ways to structure it (Claude tried a lot of options), I kept hitting what looks like symbol count limits, and I'd love a sanity check on whether these are expected / known before I consider a proper bug report:

- A single := [].{} block seems to stop "seeing" its own imports once it has ~50+ functions — every FfiOption.x reference reports "no module FfiOption imported", even though import FfiOption is right there at the top.
- A platform seems to cap around ~350 total .{} methods across its modules before roc check panics ("reached unreachable code").
- An app seems to cap around ~200 methods pulled in (not referenced directly) across all imports before the same panic.

For now it seems I can declare all ~500 functions in a single library across multiple modules (couldn't simply within the platform), but I can only import a subset of the modules in the lib when writing the Roc app before the program stops compiling.

Are these known limits or bugs of symbol count per block / module / platform / app?

Karakatiza (May 27 2026 at 12:10):

You're really diving straight into the deep end on this one

For me it's like a warzone - you fly in before you can walk :sweat_smile:

Anton (May 27 2026 at 13:02):

Are these known limits or bugs of symbol count per block / module / platform / app?

I think they are all bugs

Richard Feldman (May 27 2026 at 14:18):

for sure! @Karakatiza any chance you could give a .roc file (or a few of them) which reproduce that?

Karakatiza (May 27 2026 at 20:31):

I want to get further along in my prototype to get to some working baseline, after that it should be easy to trigger these issues I encountered again, which I'd then share.

Richard Feldman (May 27 2026 at 20:59):

cool, thanks!

Luke Boswell (May 27 2026 at 21:21):

If you (or anyone) finds issues like this and makes a minimal repro and GH issue we usually can fix the bugs pretty fast.

Last updated: Jun 16 2026 at 16:19 UTC