Tanner Nielsen (Nov 29 2024 at 06:20): Tanner Nielsen (Nov 29 2024 at 06:20):I've been working on a custom Roc platform in Rust, relying heavily on example platforms. I think I have a decent high-level understanding for how the pieces fit together. I have a lot of ideas I'm excited to try, but unfortunately my struggles with glue generation have ground my progress to a halt. Hoping some of the lovely folks in here can help unblock me.
I've seen from other threads and my own experience that Rust glue generation is a bit broken. In my case, there were unnecessary Clone derivations and a few other minor issues preventing the generated glue code from compiling, which I was able to fix manually. But after getting past those errors, I get hit with this cryptic error at point when the platform tries to use the generated struct to read data from the application:
thread '<unnamed>' panicked at core/src/panicking.rs:221:5:
unsafe precondition(s) violated: slice::from_raw_parts requires the pointer to be aligned and non-null, and the total size of the slice not to exceed `isize::MAX`
I realize that without more context this error probably isn't actionable, so let me back up and show the Roc types I'm trying to generate glue for:
GenericEffect : [
Single (Bytes => Bytes),
Map GenericEffect (Bytes -> Bytes),
Chain GenericEffect GenericEffect,
Fork GenericEffect GenericEffect,
Parallel GenericEffect,
Branch GenericEffect GenericEffect (Bytes -> Bool),
Loop GenericEffect GenericEffect (Bytes -> Bool),
]
Bytes : List U8
I don't necessarily want to explain what I'm hoping to achieve with this custom platform at the moment, since I'm still in the exploratory phase (but I could if need be). What I'd like to know for starters is
Luke Boswell (Nov 29 2024 at 06:48):Do you need the platform to call those functions? Maybe if you box them so they're just opaque pointers (with fixed size) on the host side. I didn't think we supported functions crossing the boundary though yet, I thought we needed type erasure for that (which isnt implemented yet).
Tanner Nielsen (Nov 29 2024 at 14:24):Correct, the application builds up this structure full of functions and hands it off to the platform for execution. My platform makes the construction and composition of effects very restricted in order to make certain security guarantees and enable debug tooling.
I can try boxing the functions on the roc side when I have access to a computer.
In general does everything being passed from roc to the platform need a fixed size? So is List U8 also an issue?
Brendan Hansknecht (Nov 29 2024 at 18:50):Glue has a lot of bugs and limitations currently. Functions are an exceptionally sharp edge.
Brendan Hansknecht (Nov 29 2024 at 18:51):We do plan to restrict the platform API to fixed sized data types. Today, roc will create a size function for anything that is not fixed size, but it isn't aways enough to be useful.
Brendan Hansknecht (Nov 29 2024 at 18:52):I would not expect this platform to work in general. It looks a lot like effect interpreters which are stuck due to compiler bugs. It might end up working out, but I would expect to hit compiler bugs that need to be worked around.
Brendan Hansknecht (Nov 29 2024 at 18:54):Lists have a compile time known size and are totally fine to pass to the platform. A list is the size of 3 pointers.
Tanner Nielsen (Nov 29 2024 at 23:59):@Brendan Hansknecht regarding your comment about this looking like effect interpreters, does it make a difference to know that for this platform the effects being handed off to the platform is a one-time transaction? Like the application essentially is just a DSL for building up an effectful computation that the platform executes. There's no back and forth between the app and platform.
Luke Boswell (Nov 30 2024 at 00:01):I think the issue is that when you have recursion and captures you are in lambda set territory and those sometimes break.
Luke Boswell (Nov 30 2024 at 00:01):It may work, but when it doesn't we don't have many options available.
Luke Boswell (Nov 30 2024 at 00:02):Also, are you passing these functions across to the host (rust/zig) etc? or do they live purely in the roc side of things (between app and platform)?
Luke Boswell (Nov 30 2024 at 00:03):If you can provide more context I can probably provide more assistance. Even if it's just a fake platform.
Tanner Nielsen (Nov 30 2024 at 00:21):Sure, I can try to give more context. With this platform, the Roc API provides constructors for creating effects, but not executing any of them. The application can't even provide a main function, just a "main effect" value. Custom effects cannot be created by the app author, only selected from the platform library. The GenericEffect type I showed above isn't exposed to the app author either. Instead they are given a type-safe wrapper with functions for composing effects and/or doing ad-hoc mapping (pure functions only):
Effect a b := GenericEffect
map : Effect a b, (b -> c) -> Effect a c
chain : Effect a b, Effect b c -> Effect a c
fork : Effect a b, Effect a c -> Effect a (b, c)
parallelize : Effect a b -> Effect (List a) (List b)
branch : Effect a b, Effect a c, (a -> bool) -> Effect a (Either b c)
loop : Effect a a, Effect a b, (a -> bool) -> Effect a b
fixture : a -> Effect {} a where a implements Encoding
discard : Effect a * -> Effect a {}
There are some missing implements Encoding/Decoding annotations above, but this gives a rough idea of the application API. Effect a b internally manages the GenericEffect (shown earlier) by wrapping any provided functions that operate on a or b with encoders/decoders to keep the host's view of the computation generic (and enable some of the features described below)
Here's what "hello world" might look like:
app [main] { pf: platform "platform/main.roc" }
import pf.Stdout
import pf.Effect
main = Effect.fixture "Hello world!" |> Effect.chain Stdout.line
So to answer the question above, yes the host needs to be able to execute the functions defined in Roc, because the application (intentionally) isn't allowed to actually execute anything, other than functions to compose effects.
Tanner Nielsen (Nov 30 2024 at 00:26):Also Effect.fixture isn't truly an effect (probably needs a better name). It's just what I'm using for my POC code at the moment because it works nicely with chain
Tanner Nielsen (Nov 30 2024 at 00:32):The big idea is that by giving the platform a highly-structured view of the application, you can get a whole bunch of stuff for free, like automatic state chart documentation; debug instrumentation that allows you to pause, inspect, and modify data at runtime; and interesting security analysis capabilities (e.g., going beyond "this program has network and disk access" to "this program, despite disk and network access, cannot send your personal files to a remote server, because there's no path between the ReadFile effect and HttpPost effect")
Tanner Nielsen (Nov 30 2024 at 00:36):That last idea depends on functional purity and tightly controlling the composition and execution of effectful code (and Roc is the only language that I know of that has these critical features!)
Luke Boswell (Nov 30 2024 at 00:44):Ok, I'm not sure I can give you any definitive answers.
My approach with this kind of thing is to start with the very smallest e2e thing I can think of and build it up bit by bit. So maybe you could try just with:
GenericEffect : [
Single (Bytes => Bytes),
]
And see if you can get that working?
One thing I find that is helpful, is that you can generate the llvm ir with roc build --no-link --emit-llvm-ir path/to/app.roc. That will give you the definitive answer on what roc is generating. I often use this to confirm the FFI is correct.
Luke Boswell (Nov 30 2024 at 00:47):I was able to generate glue using RustGlue.roc and this platform.
platform ""
requires {} { main! : {} => _ }
exposes []
packages {}
imports []
provides [mainForHost!]
Bytes : List U8
GenericEffect : [
Single (Bytes => Bytes),
]
mainForHost! : {} => GenericEffect
mainForHost! = \{} -> main! {}
#[repr(C)]
#[derive(Debug)]
pub struct RocFunction_67 {
closure_data: Vec<u8>,
}
impl RocFunction_67 {
pub fn force_thunk(mut self, arg0: roc_std::RocList<u8>) -> roc_std::RocList<u8> {
extern "C" {
fn roc__mainForHost_0_caller(arg0: *const roc_std::RocList<u8>, closure_data: *mut u8, output: *mut roc_std::RocList<u8>);
}
let mut output = core::mem::MaybeUninit::uninit();
unsafe {
roc__mainForHost_0_caller(&arg0, self.closure_data.as_mut_ptr(), output.as_mut_ptr());
output.assume_init()
}
}
}roc_refcounted_noop_impl!(RocFunction_67);
#[derive(Clone, )]
#[repr(transparent)]
pub struct GenericEffect {
f0: RocFunction_67,
}
impl GenericEffect {
/// A tag named ``Single``, with the given payload.
pub fn Single(f0: RocFunction_67) -> Self {
Self {
f0
}
}
/// Since `GenericEffect` only has one tag (namely, `Single`),
/// convert it to `Single`'s payload.
pub fn into_Single(self) -> RocFunction_67 {
self.f0
}
/// Since `GenericEffect` only has one tag (namely, `Single`),
/// convert it to `Single`'s payload.
pub fn as_Single(&self) -> &RocFunction_67 {
&self.f0
}
}
impl core::fmt::Debug for GenericEffect {
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
f.debug_tuple("GenericEffect::Single")
.field(&self.f0)
.finish()
}
}
roc_refcounted_noop_impl!(GenericEffect);
This is looking positive...
Tanner Nielsen (Nov 30 2024 at 00:48):Yes, I should have been more clear that the glue generation does succeed, but the generated code doesn't compile
Luke Boswell (Nov 30 2024 at 00:49):Ohk... yeah the generated code is a mess rn. I usually just use it as a general guide
Luke Boswell (Nov 30 2024 at 00:49):Helps me figure out the order and alignment etc
Tanner Nielsen (Nov 30 2024 at 00:52):So admittedly I don't really understand what I need to look for to fix order/alignment issues :sweat_smile:. I saw the DescribeGlue.roc spec for spitting out the types/alignment, but I couldn't do much with that information. I'm not sure how much I need to understand about FFI in general
Tanner Nielsen (Nov 30 2024 at 00:54):Sorry for the super vague question :sweat_smile:
Luke Boswell (Nov 30 2024 at 00:54):I'm not an expert... but me and my buddy Claude stumble through it
Luke Boswell (Nov 30 2024 at 00:54):Having the LLVM IR is super helpful
Tanner Nielsen (Nov 30 2024 at 00:56):Is there something in particular I should look for in the 13k line LLVM IR output?
Luke Boswell (Nov 30 2024 at 00:56):Yeah, search for the entrypoint... like mainForHost etc
Luke Boswell (Nov 30 2024 at 00:59):For example in the JS DOM experimental platform I have the following...
platform ""
requires { Model } {
init : {} -> Model,
update : Model, List U8 -> Action.Action Model,
render : Model -> Html.Html Model,
}
exposes [Html, Action]
packages {}
imports []
provides [initForHost, updateForHost, renderForHost]
import Html
import Html
import Action
initForHost : I32 -> Box Model
initForHost = \_ -> Box.box (init {})
updateForHost : Box Model, List U8 -> Action.Action (Box Model)
updateForHost = \boxedModel, payload ->
Action.map (update (Box.unbox boxedModel) payload) Box.box
renderForHost : Box Model -> Html.Html Model
renderForHost = \boxedModel -> render (Box.unbox boxedModel)
So I can find these in the LLVM
define ptr @roc__initForHost_1_exposed(i32 %0) !dbg !99 {
entry:
%call = call fastcc ptr @_initForHost_99aa979e4a9cadd6dbe48ea878ec84acb7696eb93470c375f6893f1da46c3772(i32 %0), !dbg !100
ret ptr %call, !dbg !100
}
define i64 @roc__initForHost_1_exposed_size() !dbg !102 {
entry:
ret i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), !dbg !103
}
define { [1 x i32], i8 } @roc__updateForHost_1_exposed(ptr %0, ptr %1) !dbg !120 {
entry:
%result_value = alloca { [1 x i32], i8 }, align 8, !dbg !121
%load_arg = load %list.RocList, ptr %1, align 4, !dbg !121
call fastcc void @_updateForHost_392aebc0773ca1163ead8eb210e2c2aabca4fe4ded9f2b122a7dab30d082d98b(ptr %0, %list.RocList %load_arg, ptr %result_value), !dbg !121
%load_result = load { [1 x i32], i8 }, ptr %result_value, align 4, !dbg !121
ret { [1 x i32], i8 } %load_result, !dbg !121
}
define i64 @roc__updateForHost_1_exposed_size() !dbg !123 {
entry:
ret i64 ptrtoint (ptr getelementptr ({ [1 x i32], i8 }, ptr null, i32 1) to i64), !dbg !124
}
define ptr @roc__renderForHost_1_exposed(ptr %0) !dbg !183 {
entry:
%call = call fastcc ptr @_renderForHost_eabc27640eff330d625cb2f6435f5dccaec45dd590ad64015fdca105b70(ptr %0), !dbg !184
ret ptr %call, !dbg !184
}
define i64 @roc__renderForHost_1_exposed_size() !dbg !186 {
entry:
ret i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), !dbg !187
}
Which I've then translated to the following...
pub fn roc_init() -> RocBox<()> {
#[link(name = "app")]
extern "C" {
// initForHost : I32 -> Model
#[link_name = "roc__initForHost_1_exposed"]
fn caller(arg_not_used: i32) -> RocBox<()>;
#[link_name = "roc__initForHost_1_exposed_size"]
fn size() -> i64;
}
unsafe {
debug_assert_eq!(std::mem::size_of::<RocBox<()>>(), size() as usize);
caller(0)
}
}
pub fn roc_update(state: RocBox<()>, raw_event: &mut RocList<u8>) -> glue::RawAction {
#[link(name = "app")]
extern "C" {
// updateForHost : Box Model, List U8 -> Action.Action (Box Model)
#[link_name = "roc__updateForHost_1_exposed"]
fn caller(state: RocBox<()>, raw_event: &mut RocList<u8>) -> glue::RawAction;
#[link_name = "roc__updateForHost_1_exposed_size"]
fn size() -> i64;
}
unsafe {
debug_assert_eq!(std::mem::size_of::<glue::RawAction>(), size() as usize);
caller(state, raw_event)
}
}
pub fn roc_render(model: RocBox<()>) -> glue::Html {
#[link(name = "app")]
extern "C" {
// renderForHost : Box Model -> Html.Html Model
#[link_name = "roc__renderForHost_1_exposed"]
fn caller(model: RocBox<()>) -> glue::Html;
#[link_name = "roc__renderForHost_1_exposed_size"]
fn size() -> i64;
}
unsafe {
debug_assert_eq!(std::mem::size_of::<glue::Html>(), size() as usize);
caller(model)
}
}
Thank you... there's a lot of unfamiliar stuff to digest here, but I'll see if I can use this to make sense of my output and find any clues as to why my app is crashing :magnifying_glass:
Tanner Nielsen (Nov 30 2024 at 01:11):Before I get too far with that though, does this Roc type even make sense?
Effect a b := GenericEffect
Specifically how the type parameters aren't used. In Rust this is a problem and I'd need to use PhantomData or something, but the Roc compiler doesn't seem to care. Just want to check that this is expected.
Luke Boswell (Nov 30 2024 at 01:13):I think so, that would be identical to this right?
Effect a b := [
Single (Bytes => Bytes),
Map (Effect a b) (Bytes -> Bytes),
Chain (Effect a b) (Effect a b),
Fork (Effect a b) (Effect a b),
Parallel (Effect a b),
Branch (Effect a b) (Effect a b) (Bytes -> Bool),
Loop (Effect a b) (Effect a b) (Bytes -> Bool),
]
Not quite. I had to wrap GenericEffect because it's not possible to name all the type parameters of the inner Effects if I were to do it in a completely type safe way. For example:
Effect a b ??? := [
Single (a => b)
Map (Effect a b) (b -> c)
Chain (Effect a b) (Effect a c)
...
]
GenericEffect deals with this problem by wrapping all those functions with encoders/decoders to coerce everything to a single concrete type (Bytes/List U8 in my case, but I could just as easily have used Str)
Tanner Nielsen (Nov 30 2024 at 01:22):I guess I should have clarified up front that a represents the input to the effect, and b represents the output
Brendan Hansknecht (Nov 30 2024 at 03:45):Tanner Nielsen said:
Brendan Hansknecht regarding your comment about this looking like effect interpreters, does it make a difference to know that for this platform the effects being handed off to the platform is a one-time transaction? Like the application essentially is just a DSL for building up an effectful computation that the platform executes. There's no back and forth between the app and platform.
Is nothing returns a GenericEffect, you may be fine. Not sure.
Brendan Hansknecht (Nov 30 2024 at 03:47):Also, do you need to pass the lambdas to the host for the Generic effect? If you could make that all live in roc, it would make your interaction with glue complexity simpler.
Tanner Nielsen (Nov 30 2024 at 04:22):The main reason I wanted the lambdas to be executed by the host was so I could do parallelism via Rust threads. That's not a hard requirement of the platform though, and I guess a single-threaded proof of concept is better than nothing.
Brendan Hansknecht (Nov 30 2024 at 06:00):Makes sense
Last updated: Jul 05 2025 at 12:14 UTC