Matt Harden (Dec 06 2024 at 19:43): Matt Harden (Dec 06 2024 at 19:43):I'm confused about what a platform needs to include. The basic-cli seems to include implementations for all of the builtins. But other demo platforms are much smaller. Are the builtins in basic-cli just there for convenience with interfacing between Roc and Zig? If so, where are the real implementations of the builtins?
Sam Mohr (Dec 06 2024 at 19:53):When you say builtins, are you referring to List and Dict and all, or stuff like File and Http?
Sam Mohr (Dec 06 2024 at 19:54):Roc's "builtins" are those types defined here: https://github.com/roc-lang/roc/tree/main/crates/compiler/builtins/roc
Sam Mohr (Dec 06 2024 at 19:54):And the basic-cli interaction with them is auto-generated as "roc glue"
Sam Mohr (Dec 06 2024 at 19:55):The other types like File and Http are defined in basic-cli specifically for now because it's easier to maintain them there, but will probably get pulled into separate packages at some point
Matt Harden (Dec 06 2024 at 19:56):Sam Mohr said:
When you say builtins, are you referring to
ListandDictand all, or stuff likeFileandHttp?
List, Dict. Str...
Matt Harden (Dec 06 2024 at 19:56):Thank you!
Sam Mohr (Dec 06 2024 at 19:57):In particular, the builtins like List and Result are included by the compiler without import in every Roc program
Sam Mohr (Dec 06 2024 at 19:57):Which is why you never see them explicitly imported
Matt Harden (Dec 06 2024 at 19:58):Follow up: how would one add support for a new platform language to roc glue?
[edit] I found it at https://github.com/roc-lang/roc/tree/main/crates/glue
Sam Mohr (Dec 06 2024 at 20:03):yep!
Sam Mohr (Dec 06 2024 at 20:03):I think @Sven van Caem was working on the glue recently, maybe he can help you figure out how to get started?
Matt Harden (Dec 06 2024 at 20:06):Thanks! I'm interested in (perhaps) making a better Go platform than https://github.com/roc-lang/examples/tree/main/examples/GoPlatform, so might give it a try to add support for Go to glue.
Matt Harden (Dec 06 2024 at 20:12):btw. I noticed GoPlatform is missing roc_panic, roc_memset, roc_getppid, roc_shm_open, and roc_mmap. Are those optional?
Sam Mohr (Dec 06 2024 at 20:12):I'm not sure.
Sam Mohr (Dec 06 2024 at 20:12):Does it run? haha
Sam Mohr (Dec 06 2024 at 20:13):Also, having Go glue could be helpful for our attempt to make stackful coroutines work, so this would be a useful endeavour!
Matt Harden (Dec 06 2024 at 20:13):It runs the provided example, but I don't know what might break if I try to use it.
Sven van Caem (Dec 06 2024 at 20:14):Sam Mohr said:
I think Sven van Caem was working on the glue recently, maybe he can help you figure out how to get started?
I might be able to answer questions! I'll have more time later though
Matt Harden (Dec 06 2024 at 20:15):Sven van Caem said:
Sam Mohr said:
I think Sven van Caem was working on the glue recently, maybe he can help you figure out how to get started?
I might be able to answer questions! I'll have more time later though
No rush! I'm pleasantly surprised to have gotten any answer at all so quickly. :-)
Sven van Caem (Dec 06 2024 at 20:15):Matt Harden said:
btw. I noticed GoPlatform is missing roc_panic, roc_memset, roc_getppid, roc_shm_open, and roc_mmap. Are those optional?
It seems to depend on the language e.g. a C app will run just fine without them as long as they're not used, but a Zig app won't compile without them
Sam Mohr (Dec 06 2024 at 20:15):Context on stackful coroutines, btw: https://roc.zulipchat.com/#narrow/channel/304641-ideas/topic/stackful.20coroutines.20in.20hosts/near/473001620
Luke Boswell (Dec 06 2024 at 20:17):https://github.com/lukewilliamboswell/roc-platform-template-go
This may be of interest.
Also checkout some of @Oskar Hahn work on different go platforms.
Matt Harden (Dec 06 2024 at 20:18):Luke Boswell said:
https://github.com/lukewilliamboswell/roc-platform-template-go
This may be of interest.
Also checkout some of Oskar Hahn work on different go platforms.
Oh! thank you, I missed those.
Luke Boswell (Dec 06 2024 at 20:20):Matt Harden said:
btw. I noticed GoPlatform is missing roc_panic, roc_memset, roc_getppid, roc_shm_open, and roc_mmap. Are those optional?
roc_panic is called by roc when there is a crash or a runtime error. It is needed.
The others are part of the shared memory infra that is used for roc test and expect . I believe the plan is to remove those. I made a start on a PR in that direction...
Luke Boswell (Dec 06 2024 at 20:21):https://github.com/roc-lang/roc/pull/7256
Matt Harden (Dec 06 2024 at 20:24):So RocStr and RocList in https://github.com/lukewilliamboswell/roc-platform-template-go/blob/main/host/roc/roc_std.go are hand spun, right?
Luke Boswell (Dec 06 2024 at 20:27):Yep
Matt Harden (Dec 06 2024 at 20:27):So the host directory is for the Roc to host language basic interface, and the platform directory is for the extended capabilities provided to Roc by the platform?
Luke Boswell (Dec 06 2024 at 20:29):A roc app doesn't exist without a platform. The platform includes a host which is written in another language like Go, and is the executable that starts first and then calls into roc.
Luke Boswell (Dec 06 2024 at 20:30):The platform files are all .roc and are basically just an API or interface between the roc part and the host part.
Matt Harden (Dec 06 2024 at 20:31):Luke Boswell said:
The platform files are all .roc and are basically just an API or interface between the roc part and the host part.
I see now; thanks.
Luke Boswell (Dec 06 2024 at 20:31):One day we might have a Go package for RocStr and RocList and the builtins so platform authors dont have to worry about the internals of how roc handles things. But for now we have been writing them by hand.
Matt Harden (Dec 06 2024 at 20:31):We would want to generate those with roc glue, right?
Luke Boswell (Dec 06 2024 at 20:33):I dont think so. Glue is more for generating types defined in the platform, like if you make a record or tag union you would use glue to generate interfacing code in go. It would probably use the Go package that already had impls for RocStr and RocList etc.
Luke Boswell (Dec 06 2024 at 20:35):So the builtins should live in a Go package/library somwhere. e.g. RocResult, RocList
And custom types specific to the platform are generated by glue. E.g. HttpResponse, File
Matt Harden (Dec 06 2024 at 20:36):Which builtin Roc types would we want to have hand implementations of in other languages? It seems like that could be a maintenance issue. Str and List are fairly complex already and the more different implementations we have the harder they will be to change without breaking things.
Matt Harden (Dec 06 2024 at 20:40):It seems like a world where glue creates a working but perhaps suboptimal impl of those types for you would be nice, as long as you keep the option of replacing them with better implementations manually.
Matt Harden (Dec 06 2024 at 20:42):glue/README.md agrees with what you're saying though.
Matt Harden (Dec 06 2024 at 20:48):I noticed that build.roc in https://github.com/lukewilliamboswell/roc-platform-template-go tells the go command to use zig cc, meaning there's a dependency on Zig. Is that necessary?
Brendan Hansknecht (Dec 07 2024 at 00:40):I really think for these types we should enable roc to expose a c abi interface for the platforms to use
Brendan Hansknecht (Dec 07 2024 at 00:40):Then the platform can just see them as an opaque bag of bytes
Brendan Hansknecht (Dec 07 2024 at 00:41):I came to this realization after making Dict. It is too complex to be worth implementing in platforms, yet it would still be very convenient to have in platforms.
Brendan Hansknecht (Dec 07 2024 at 00:42):That said, monomorphization makes this all a tad tricky.
Richard Feldman (Dec 07 2024 at 00:57):we should be able to expose zig versions that take size and alignment as arguments to all their operations, which the host needs anyway - and I think that basically takes care of monomorphization except for refcounting, yeah?
Brendan Hansknecht (Dec 07 2024 at 01:50):Yeah. Something along those lines (but doesn't work for dict which is written in roc)
Luke Boswell (Dec 07 2024 at 01:53):Brendan Hansknecht said:
I really think for these types we should enable roc to expose a c abi interface for the platforms to use
Can you give me a simple example?
Like List.len maybe?
Luke Boswell (Dec 07 2024 at 01:53):It'd be great to explore this direction further... particularly if it greatly simplifies the glue stuff Sven is working on
Luke Boswell (Dec 07 2024 at 01:54):Or at least simplifies the roc_std for zig/rust/go we've been talking about
Brendan Hansknecht (Dec 07 2024 at 01:57):It would be very similar to the zig builtins for roc list, but exposed as c abi. Then each language is ready of implementing roc list would just wrap our single bespoke c abi implementation
Brendan Hansknecht (Dec 07 2024 at 02:01):Really might be best to clean up the zig builtins and use that as the base. Though might be better to avoid the zig dependency and make this all in C.
Brendan Hansknecht (Dec 07 2024 at 02:02):Also, this still isn't enough cause our zig builtins don't understand refcounting. This is part of the reason I think it would be better for roc to generate these functions for each platform.
Luke Boswell (Dec 07 2024 at 02:03):Is there an opportunity to kill two birds, with the zig package we want to make?
Brendan Hansknecht (Dec 07 2024 at 02:03):Imagine that instead of exposing init and respond from basic webserver, it also exposed a whole suite of functions related to each of the primitives used in init and update.
Luke Boswell (Dec 07 2024 at 02:04):Oh interesting... you're imagining roc exposes more functions that are related for the primitive types exposed to the host. Nice
Luke Boswell (Dec 07 2024 at 02:06):Like in the roc-on-simple example..
Smf : {
header : Header,
tracks : List Track,
}
mainForHost! : {} => Smf
Here we might include a void roc__smf_refcount_inc() or something?
Brendan Hansknecht (Dec 07 2024 at 02:07):Exactly. Plus the primitives to interactive with a list of Track (including refcounting a Track).
Brendan Hansknecht (Dec 07 2024 at 02:07):This stuff is hard to build out in glue, but roc already knows how to do it
Brendan Hansknecht (Dec 07 2024 at 02:07):So it would be nice if roc could expose it to the platform
Brendan Hansknecht (Dec 07 2024 at 02:08):Then glue truly is just glue code and not implementation code
Brendan Hansknecht (Dec 07 2024 at 02:08):This is where I think we need to head
Richard Feldman (Dec 07 2024 at 05:22):yeah we should spec this out!
Richard Feldman (Dec 07 2024 at 05:23):definitely seems like the right approach based on what we've learned :big_smile:
Richard Feldman (Dec 07 2024 at 05:25):I think keeping the implementation in Zig should be fine - either way, the host just needs some extern definition, which can be hardcoded .h and .rs files that we can write tests against using zig alone, and which glue can just copy/paste into the output dir
Oskar Hahn (Dec 07 2024 at 08:28):Coming back to go glue code. I also tried it some time ago. My problem was, that cgo needs a .h file, with all types. This .h file needs also the signatures of the provided functions, like this one: https://github.com/lukewilliamboswell/roc-platform-template-go/blob/0e8d53b6087662ea231a164a059f1c2ba0a91c34/host/roc/app.h#L14
But this is currently impossible with glue, since the signature of the exported functions are not provided by the glue API.
The separate .h file is also the reason, why I don't think a roc-std-go package is possible. Your app.h file could not import/include the std.h file, provided by the package. So your app.h file could not use types like RocStr. Therefore I think the std types have to be generated for each app with glue.
I don't know if this would change, if roc would provide the types by API. I think it would still be necessary to generate the type definition and all std function definition with glue to use them inside the app.h.
Sven van Caem (Dec 07 2024 at 10:35):Brendan Hansknecht said:
Imagine that instead of exposing init and respond from basic webserver, it also exposed a whole suite of functions related to each of the primitives used in init and update.
Love this idea! Would really lighten the load for a glue spec author.
Matt Harden (Dec 07 2024 at 16:27):Oskar Hahn said:
Coming back to go glue code. I also tried it some time ago. My problem was, that cgo needs a .h file, with all types. This .h file needs also the signatures of the provided functions, like this one: https://github.com/lukewilliamboswell/roc-platform-template-go/blob/0e8d53b6087662ea231a164a059f1c2ba0a91c34/host/roc/app.h#L14
But this is currently impossible with glue, since the signature of the exported functions are not provided by the glue API.
The separate .h file is also the reason, why I don't think a roc-std-go package is possible. Your app.h file could not import/include the std.h file, provided by the package. So your app.h file could not use types like RocStr. Therefore I think the std types have to be generated for each app with glue.
I don't know if this would change, if roc would provide the types by API. I think it would still be necessary to generate the type definition and all std function definition with glue to use them inside the app.h.
If all Roc values were passed around as opaque pointers in the API you wouldn't need any types in the .h files.
Richard Feldman (Dec 07 2024 at 17:25):well it's pretty common that the host wants to do things with the values it gets from Roc, e.g. read the contents of a string or list
Richard Feldman (Dec 07 2024 at 17:25):so in those cases it needs something more than opaque pointers
Brendan Hansknecht (Dec 07 2024 at 17:42):Richard Feldman said:
I think keeping the implementation in Zig should be fine
I don't think so. Zig does not know how to recount things. Only roc does. It depends on roc passing it functions that tell it how to recount things. I think we need to leave this all to roc. Otherwise, glue becomes way harder.
Brendan Hansknecht (Dec 07 2024 at 17:46):Roc will surely already be generating these functions anyway, it needs them to do refcounting of things passed into roc.
Brendan Hansknecht (Dec 07 2024 at 17:54):I guess after recounting, the rest could be in zig theoretically.
This is where you get a trade from:
Extreme 1, like today, platform has to implement everything, super brittle
To:
Extreme n, platform just needs to know the size of the type and which methods it has (including methods to get and set the data in the object). Roc exposes all of those via cffi. The platform doesn't even need to know the layout of the type.
I think we should be way closer to extreme n. Maybe even at extreme n. Extreme n does have a minor cost in perf due to there being no inlining between platform and app, but it removes tons of what makes the API brittle and hard to work with.
Richard Feldman (Dec 07 2024 at 17:58):oh sorry I mainly meant zig as opposed to C
Richard Feldman (Dec 07 2024 at 17:59):it makes sense that Roc would expose functions to the host, and that some of those would be implemented in the Roc code base using Zig
Richard Feldman (Dec 07 2024 at 17:59):but I don't think we would need to change any internal implementations from Zig to C to facilitate any of that
Brendan Hansknecht (Dec 07 2024 at 17:59):Oh yeah, for sure not
Brendan Hansknecht (Dec 07 2024 at 18:00):Literally could make an exposed function for every builtin if we wanted. No need to implement anything in c for that.
Richard Feldman (Dec 07 2024 at 18:01):yeah, basically we could look through every type that's exposed to the host and expose all the appropriate functions for it
Richard Feldman (Dec 07 2024 at 18:01):and put them in their own sections so a linker can DCE any unused ones
Richard Feldman (Dec 07 2024 at 18:02):and in a surgical linking future we could even do it lazily
Richard Feldman (Dec 07 2024 at 18:02):actually I guess just preprocessing the host is enough :thinking:
Richard Feldman (Dec 07 2024 at 18:04):like see which functions the host is dynamically linking in, and go generate them all (or give an error if it's asking for something that's malformed or shouldn't be exposed to the host)
Richard Feldman (Dec 07 2024 at 18:04):that way we wouldn't waste time generating a ton of monomorphized functions that the host wouldn't even use in practice anyway
Richard Feldman (Dec 07 2024 at 18:06):I guess with --no-link you'd have to generate them all anyway, unless we gave --no-link an optional host to analyze but not link
Brendan Hansknecht (Dec 07 2024 at 18:08):Yeah, exactly this. And I don't think we need to expose the full standard library just enough primitives to use each type. Host languages tend to have enough tools to deal with most things (like mapping folding, looping, etc)
Richard Feldman (Dec 07 2024 at 21:00):yeah that makes sense to me! :+1:
Richard Feldman (Dec 07 2024 at 21:00):is there anything blocking that? I can't think of anything
Brendan Hansknecht (Dec 07 2024 at 22:35):Not that I can think of.
Brendan Hansknecht (Dec 07 2024 at 22:36):As I said earlier, it could be done manually today by adding a ton more functions exposed to the host. Should be very doable to automatically generate them instead.
Norbert Hajagos (Jan 09 2025 at 10:10):Was scanning zulip for some knowledge regarding the builtins when I came across this. Did this discussion produce a GH issue? Would be a shame to lose the conclusion.
Luke Boswell (Jan 09 2025 at 10:14):I don't think we have an issue. But Brendan has mentioned a couple of times that he plans to work on this
Luke Boswell (Jan 09 2025 at 10:20):
Norbert Hajagos (Jan 09 2025 at 11:04):Ah, thanks Luke, saw that thread when browsing, but it was in the additional projects section. It feels like @Brendan Hansknecht can do anything :big_smile:
Brendan Hansknecht (Jan 09 2025 at 16:06):Anything with the backend, maybe....I don't tend to touch anything type checker and up.
Last updated: Jul 06 2025 at 12:14 UTC