How does "roc__mainForHost_1_exposed_generic" work? · platform development

Hello experts. I'm trying to understand how to write a zig platform.
I'm able to get the zig platform example to work but I'm having troubles figuring out what is exported by roc and how to use it from zig. Basically what should I put in my main (zig) function if my main (roc) function is not just a string, but a more complex type?

To start, I copied the bare minimum of types and functions from the basic cli platform to be able to write a "hello world" type of program with effects and tasks. So basically I copied Effect, Task, InternalTask, and Stdout and commented out most of them.

My platform expects a type "Task {} [Exit I32]_" as it is a copy of the basic cli platform and my main.roc program is main = Stdout.line "test". And my host.zig is basically the same as in the switching platforms example.

After running roc build --lib and inspecting the main.so object I can see there are a few functions:

roc__mainForHost_0_caller
roc__mainForHost_0_result_size
roc__mainForHost_0_size
roc__mainForHost_1_exposed
roc__mainForHost_1_exposed_size
roc__mainForHost_1_exposed_generic

but how should these be used? what are their arguments and return types? Which one should I call to run the main function of my roc program?

The examples I have found all have different functions. (I have looked at the travisstaloch/roc-cli-platform-zig and lukewilliamboswell/roc-ray repos at github)

Luke Boswell (Apr 25 2024 at 09:24):

Luke Boswell (Apr 25 2024 at 09:27):

The experience can be challenging at the moment as things are undocumented and still in development, but still fun to learn and tinker with.

Luke Boswell (Apr 25 2024 at 09:28):

To answer your question more specifically you might find it helpful to generate the LLVM using something like roc build --emit-llvm-ir --no-link examples/gui-counter.roc

Luke Boswell (Apr 25 2024 at 09:29):

Then you can see the implementation of what Roc is generating and will link with the platform prebuilt binaries. So you can see what the platform is going to recieve.

Luke Boswell (Apr 25 2024 at 09:32):

I would recommend starting with much simpler interface between roc and the zig host... like maybe a RocStr in the platform switching example.

2uf56sqf at_duck (Apr 26 2024 at 06:08):

Hey @Luke Boswell , indeed it's a lot of fun. Thank you for taking the time to answer to me.
I have been looking at your code and the article you linked and they helped me a lot. Already took your advice and generated teh llvm-ir and that helped me get the arguments and return values (kind of... the best I can do is call almost everything *anyopaque).

The thing I'm having issues is: The zig example in platforms-switching seems to suggest that the RocStr object is like an allocator or a space in memory that should be passed to roc__maingForHost... to be "filled" with the result of the main (roc) function and then that result is printed to stdout. Do I need to write a parser for the type my platform is returning on the zig side? this doesn't seem to be the case with other projects.
Also, how did you decide which specific functions to call and how to use them? Do I need to make use of all of those functions or do they overlap in their functionality?

Hristo (Apr 26 2024 at 07:14):

To me, as a beginner, the following was the key insight from step 3 in Luke's article, which I think is relevant to the question here:

Also, *anyopaque sounds like a step in the right direction, yes. But I'd let the more experienced members confirm and/or expand upon further on the topic.

Luke Boswell (Apr 26 2024 at 07:15):

My understanding is limited, but I'll try and explain from my experience and previous zulip discussions.

In future, you will be able to expose multiple functions that a host can use to call into Roc. For now, I think we only support one "mainForHost" function that the host can use to call Roc.

The roc__mainForHost_1_exposed_generic takes two arguments, the first is a pointer to the return type, and the second is a pointer to the argument. It's generated by roc, so if in your platform/main.roc you use a different name or different type signature then the code that roc generates for that symbol may be named or behave differently.

For example a mainForHost: Str -> U8 you might expect to see something like a extern fn roc__mainForHost_1_exposed_generic(u8, *RocStr) void;

extern "C" {
    #[link_name = "roc__mainForHost_1_exposed_generic"]
    pub fn roc_main(output: *mut u8, roc_args: *mut roc_app::Args);
}

For now (I think the idea is for more in future) roc can only generate 1 entry-point, which is a way for the host to call into roc code. So where you want a platform to expose multiple functions to the host we do this by returning a record that contains.

e.g. for roc-ray we return a record (which will generate a struct) that contains those functions.

ProgramForHost : {
    init : Task (Box Model) [],
    update : Box Model -> Task (Box Model) [],
}

mainForHost : ProgramForHost
mainForHost = { init, update }

var model: *anyopaque = undefined;

extern fn roc__mainForHost_1_exposed_generic(*anyopaque) callconv(.C) void;
extern fn roc__mainForHost_1_exposed_size() callconv(.C) i64;

// Init Task
extern fn roc__mainForHost_0_caller(*anyopaque, *anyopaque, **anyopaque) callconv(.C) void;
extern fn roc__mainForHost_0_size() callconv(.C) i64;

// Update Fn
extern fn roc__mainForHost_1_caller(**anyopaque, *anyopaque, *anyopaque) callconv(.C) void;
extern fn roc__mainForHost_1_size() callconv(.C) i64;

// Update Task
extern fn roc__mainForHost_2_caller(*anyopaque, *anyopaque, **anyopaque) callconv(.C) void;
extern fn roc__mainForHost_2_size() callconv(.C) i64;

Now this is more complicated as the host needs a way to know the size of things to handle captures. This is basically at the limit of my knowledge on the topic, so I usually ask for assistance from @Folkert de Vries or @Brendan Hansknecht and others far more experience with low level programming.

There is at least two significant changes planned here that I am aware of, the first is support for effect interpreters (enabling platforms to run things in async), and the second is switching to passed in allocators.

Now the other part is how Roc calls back into the host. That is done by functions that are exposed by the host and roc will link to. These are defined in the Effect.roc which is a special module of hosted type and will be removed in future with the effect interpreters change.

So for example, if you see a signature like setWindowSize : I32, I32 -> Effect {} the host will need to export a symbol that looks something like this.

export fn roc_fx_setWindowSize(width: i32, height: i32) callconv(.C) void {
    raylib.SetWindowSize(width, height);
}

Or for a function like stdoutLine : Str -> Effect (Result {} Str) the host will export somethin like,

#[no_mangle]
pub extern "C" fn roc_fx_stdoutLine(line: &RocStr) -> RocResult<(), RocStr> { ... }

Ultimately the best solution for these types is to use roc glue which enables someone to write a plugin for the roc compiler (which is just a roc app), that receives a data structure of all the Roc types for a given application and can code gen the required types for that language.

If you have a Rust platform for example, you wouldn't hand roll the types for your Roc code, instead using the roc glue plugin it will generate a roc_app crate with all of these for you. While glue is definitely a WIP, I have used it a lot to generate code for example in basic-webserver.

The types like RocStr are not necessarily straightforward to use even though they may look simple. So this is why using the generated types will be a nicer experience than writing these from scratch each time.

pub const RocStr = extern struct {
    str_bytes: ?[*]u8,
    str_len: usize,
    str_capacity: usize,
}

@Brendan Hansknecht gave a nice overview of the List type and explained how Roc uses various optimisations to make these fast and efficient.

In future, I hope we can have a nice library of bindings for common languages that are widely tested and well supported. But for now, we have a WIP RustGlue.roc spec which is useful for getting started quickly.

Luke Boswell (Apr 26 2024 at 07:26):

Also, I want to mention that I have put a fair bit of work into the basic-ssg platform this week. My goal was to cleanly separate out all the parts into modules/crates so that it is much easier to follow how things are built, and also test some ideas before doing the same changes to basic-cli and basic-webserver.

The thing I like most there is how the rust "host" is split out into a crate instead of in the same folder as the roc "platform" code.

It's a nice separation, because I think it's much easier to follow how these are separately compiled into static libraries and then linked together into an archive or "prebuilt" binary. These prebuilt binaries are what is package into the URL .tar.br files for releases like with basic-cli and used by the legacy linker.

The surgical linker is a little different, in that the platform is actually given a stubbed version of a application object so it can be built into a fully completed executable binary. It's then processed a little, so that when the "real" application object is available from the application author, we can surgically swap out just the parts needed. Hence why surgical linking is so fast, but unfortunately only available for linux I think, as macos linking is hard and windows has a bug that causes everything to segfault.

Luke Boswell (Apr 26 2024 at 07:48):

I think the best way for now to know for sure what the types of all the parameters for the generated symbols for a specific platform is to generate the LLVM IR and inspect that. The generated functions are specific to that platform and change whenever the types change, i.e. changing the type of mainForHost by adding new arguments or a functions in the record etc.

Or if you use the exact same type for mainForHost as another platform, you can just copy and paste that implementation and be good to go, i.e. copy basic cli if a single Tasksuits.

Stream: platform development

Topic: How does "roc__mainForHost_1_exposed_generic" work?

2uf56sqf at_duck (Apr 25 2024 at 08:40):