Brendan Hansknecht (Jul 03 2024 at 00:43): Brendan Hansknecht (Jul 03 2024 at 00:43):So I just had a realization. With Task as a builtin, we totally can enable a limited form of platform composability. For example, we could enable using sqlite with any platform at all. No need for the platform to support sqlite or any sqlite related primitives.
We essentially allow packages to have a special form of the essentially hosted module. This special form of the hosted module would load a shared library and run calls against it. It likely would need design work to be made nice, but fundamentally should work and be totally safe. Everything impure will still be wrapped in Task.
Now a user can package roc_sqlite which includes an sqlite wrapper shared library for all major platforms along with some roc code.
If we really want the best api here, it would be preferable for roc to automatically be able to deal with a certain class of type conversions. That way we could directly call into existing shared libraries instead of always requiring a shim.
The main concern I have with this is how does it work with async and the underlying state machine. In the perfect world, we would want to enable the host state machine to keep running while we are waiting on file io in a shared library. In practice, I think this may require accepting that it will always block the host state machine by default. Maybe there is something smart we can do here to at least make the idea of async callbacks work nice.
Brendan Hansknecht (Jul 03 2024 at 00:46):I guess related to blocking the state machine. There should be some sort of primitive to say run this in another thread cause it is blocking io. That would spawn a thread that would block while the rest of the state machine keeps running. So maybe we just need something like that.
Note, theoretically this could be done today if every platform added support for libffi. That said, I have no idea how that mapping would be made nice without support from the compiler.
Ryan Barth (Jul 03 2024 at 04:11):Is the idea specifically to wrap dynamic libraries?
I thought about this but another important thing I think this gives up is 1) the platform sandbox and 2) the platform having complete knowledge of the effects of the roc app.
It seems like you can either have sandboxing or arbitrary calls from packages to shared libraries, but not both.
Brendan Hansknecht (Jul 03 2024 at 04:53):Specifically dynamic libraries cause otherwise we have no way to properly deal with linking. Surgical linker wouldn't work with an arbitrary library to link in.
Brendan Hansknecht (Jul 03 2024 at 04:56):As for sandboxing, I guess it depends on the use and maybe platforms need some sort of way to opt into this feature (and maybe compilation in general).
For basic CLI there really is no loss. Give the user the power to call sqlite, blas, etc. that is just more power. It is the users app after all.
For basic webserver, a user may shoot themselves in the foot by adding blocking io wrong. I think that is fine, we just need to add a threading system to support running blocking tasks. So the user has a solution to that.
For a security or plugin focused platform. For example mods for a game. This feature should probably be turned off.
Jasper Woudenberg (Jul 03 2024 at 06:28):One risk I see in this is that it might create a split in the ecosystem. It'd become possible to create a 'base platform' that takes a Task and runs it, with all the effects provided by libraries. That would provide maximum composibility of effects, but effectively also ignore the platforms feature.
I imagine the 'base platform' approach would be an attractive option as well, both for platform/library authors because it'd be the most flexible way to publish their code, and also app authors because it's the approach most familiar from other general purpose programming languages.
Brendan Hansknecht (Jul 03 2024 at 06:30):Maybe. Though I would expect the restriction to shared libraries and some of the cumbersome nature of dynamic ffi would make that less likely.
Brendan Hansknecht (Jul 03 2024 at 06:30):Still definitely could happen.
Brendan Hansknecht (Jul 03 2024 at 06:31):Still definitely could happen.
Brendan Hansknecht (Jul 03 2024 at 06:31):Though if it were to become the default pattern, that would likely suggest that there is a major issue with Roc's assumptions about the value of platforms.
Brendan Hansknecht (Jul 03 2024 at 06:36):I think there will always be a long tail of potential extra functionality that someone wants from a platform. Currently, roc has no solution for this. I think at some point we need to figure out a way to address it.
I doubt forking the platform is the answer. As Roc's community grows, more people will just want to be Roc application authors that never touch platforms.
I think configurable platforms with many feature flags might be the answer, but that will likely only ever cover a short part of the long tail of wanted features. (and becomes an 2^n distribution problem if we want precompiled binaries instead of requiring users to build from source)
For many things, breaking down to primitives and creating a roc library with module params will be the solution (just chat with postgres and mysql directly over sockets). I just don't think it is a large enough solution. There are many C ffi friendly tools that don't make sense to break down into primitives. They also may not be large enough to be a feature flag on any platform. So they either just won't exist in roc or require something more flexible.
Brendan Hansknecht (Jul 03 2024 at 06:41):I definitely think this will be a case where we want explicit opt in. This should be behind some sort of feature flag for auditing reasons. As said above, a secure-cli script and a game modding api likely want to restrict all ffi through only the platform exposed tasks. No 3rd party tasks.
Jasper Woudenberg (Jul 03 2024 at 07:17):I appreciate someone might want to, say, write an application that outputs some stats of an sqlite database, and have no way to write that application in Roc of the basic-cli platform does not provide sqlite support. Substitute sqlite with any other less-known database technology and the argument stays the same. I don't have a better idea for solving this.
I think there will always be a long tail of potential extra functionality that someone wants from a platform. Currently, roc has no solution for this. I think at some point we need to figure out a way to address it.
I doubt forking the platform is the answer. As Roc's community grows, more people will just want to be Roc application authors that never touch platforms.
I think there might be two different audiences here. One is a group that wants to do something non-standard/untested on the platform, like use a database the platform was not designed to work with. This group is living on the edge, and I think forking the platform (and potentially contributing back if things work out) would not be out of their wheelhouse.
The people who don't want to touch platforms at all might be best served by using the IO primitives that come with the platform, because those are built to integrate with the platform well. For instance, one thought I have had is to build/contribute to a web platform with great observability/tracing support out of the box, automatically instrumenting outgoing HTTP requests, database queries, etc. For the less platform-savy users especially, I wouldn't want them to do something seeming reasonable like pulling in a database library, then be surprised when those queries aren't traced.
Though if it were to become the default pattern, that would likely suggest that there is a major issue with Roc's assumptions about the value of platforms.
My understanding of platforms and the assumptions behind them, is that they're inherently and intentionally adding some friction. They essentially introduce a constraint that all IO primitives be defined in the same place, which no other programming language I know has. I think the hypothesis is that this constraint will push folks to build nice curated experiences, the Elm's of other domains.
I can definitely see a path where given the choice folks choose to avoid the friction, and we don't get to test the hypothesis.
I definitely think this will be a case where we want explicit opt in. This should be behind some sort of feature flag for auditing reasons. As said above, a secure-cli script and a game modding api likely want to restrict all ffi through only the platform exposed tasks. No 3rd party tasks.
This might be a solution, though I see some trade-offs:
If it's the platform author that decides whether the flag is enabled or not for their platform, that could create a lot of pressure on the platform author to "unlock the door" so to speak, some of it ugly. This is what happened in Elm, where group pushed to allow performing arbitrary effects from pure Elm code and wouldn't take no for an answer.
If it's the app user that decides whether the flag is enabled or not, does that mean the feature is aimed more at expert users than novice users?
Ryan Barth (Jul 03 2024 at 07:30):Brendan Hansknecht said:
I think there will always be a long tail of potential extra functionality that someone wants from a platform. Currently, roc has no solution for this. I think at some point we need to figure out a way to address it
We may not need to cover everything though. Just looking at some of the better loved "batteries included" standard libs (eg. Python and Go) and reproducing their functionality in basic-cli may be enough.
Coming from the python ecosystem, people often will sacrifice using external libraries to keep their scripts portable and avoid the dependency management headache of python. You can get a lot done within the confines of those standard libs (basically all of ansible depends on that fact). The only thing that worries me is that those authors always have the escape hatch of reaching for whatever package they want in the future. Currently Roc's basic-cli users don't outside building their own platform.
Ryan Barth (Jul 03 2024 at 08:02):Jasper Woudenberg said:
This is what happened in Elm, where group pushed to allow performing arbitrary effects from pure Elm code and wouldn't take no for an answer.
There are 2 critical differences this time around however. 1) You have an officially blessed 100% capable escape hatch: go build your own platform. 2) Richard does not hold the keys to the castle for that escape hatch, unlike Evan.
Brendan Hansknecht (Jul 03 2024 at 08:02):I don't think python or go are good examples here. They have tons and tons of packages that do arbitrary io. Sure the standards are nice, but python is glued to cffi and go still has plenty.
Ryan Barth (Jul 03 2024 at 08:11):If it's the platform author that decides whether the flag is enabled or not for their platform, that could create a lot of pressure on the platform author to "unlock the door"
One of the reasons I was asking about standards and module params earlier is there are parts of this problem that are similar to the wasm platform. Wasm is sandboxed and cannot execute anything without the host runtime.
So there is not an "all or nothing" but more control. What if apps / platforms had the option to enable sections of some standard effectful interface ala wasi capabilities.
(edit: lost part of my quote there)
Ryan Barth (Jul 03 2024 at 08:17):A program like ftracecan wrap syscalls, couldn't roc as well as part of this dylib enabled platform?
Richard Feldman (Jul 03 2024 at 11:36):Ryan Barth said:
It seems like you can either have sandboxing or arbitrary calls from packages to shared libraries, but not both.
this is correct, and it's why I think the bar should be extremely high for doing something like this
Richard Feldman (Jul 03 2024 at 11:39):for example, I think we need at least 10 more real motivating examples than sqlite
Richard Feldman (Jul 03 2024 at 11:41):the actually-secure sandboxing benefit is one of the only things that genuinely cannot be found in any other language besides Roc
Richard Feldman (Jul 03 2024 at 11:41):(I guess unless you count languages that are JS, or only compile to JS or wasm)
Richard Feldman (Jul 03 2024 at 11:42):and the specific reason for that is that Roc doesn't support arbitrary dependencies introducing arbitrary C code
Richard Feldman (Jul 03 2024 at 11:43):Brendan Hansknecht said:
Though if it were to become the default pattern, that would likely suggest that there is a major issue with Roc's assumptions about the value of platforms.
I partially agree - in one sense, if it turns out that everyone prefers to use platforms with FFI-like capabilities, then yeah, maybe that means everyone doesn't care about the sandbox after all
Richard Feldman (Jul 03 2024 at 11:43):on the other hand, it could also mean that people value expedience over guarantees
Richard Feldman (Jul 03 2024 at 11:44):for example, today there is a huge ecosystem of packages written in 100% JavaScript, because JavaScript in the browser did not have an escape hatch
Richard Feldman (Jul 03 2024 at 11:45):if JS in the browser had enabled C FFI, probably a lot of people would have used it extensively instead of writing libraries in JS, and the result would have been that it was fundamentally unsafe to use JS in the browser because it could give you viruses, etc.
Richard Feldman (Jul 03 2024 at 11:48):so you could look at that world and say "well I guess the sandbox wasn't so valuable after all, since everyone reached for the C ecosystem as soon as it was available" but you could also look at that world and say "well they were too impatient, and really missed out on an opportunity to have something with real security guarantees; if they'd just waited for the ecosystem to mature, they would have ended up with something much more valuable"
Richard Feldman (Jul 03 2024 at 11:49):obviously the considerations are different with Roc vs JS in the browser :big_smile:
Richard Feldman (Jul 03 2024 at 11:49):but I do think we are trying to over-generalize what feels like a narrow problem right now
Richard Feldman (Jul 03 2024 at 11:50):for example, Postgres, MySQL, and Redis all run in a separate process and are not motivating examples for this sort of design at all
Richard Feldman (Jul 03 2024 at 11:50):among databases, it's currently just sqlite and that's it
Kiryl Dziamura (Jul 03 2024 at 11:53):I wonder if the following scenario is possible: one creates a base platform with a plugin system. then onyone can write a plugin in low-level language (and with a corresponding roc modules) and in the end you have to compose a platform of your dream manually with very minimal low-level code. as a result, the downsides are the same as for FFI packages from the box, right? but it's probably will be discouraged by community
Richard Feldman (Jul 03 2024 at 11:55):this is already possible in general, but the ergonomics aren't great
Richard Feldman (Jul 03 2024 at 11:56):for example, if I'm a platform author, I can provide something like this:
runDylib : { dylibPath : Path, fn : Str, args : List U8 } -> Task (List U8) DylibErr
then you'd need to use a wrapped dylib (e.g. a dylib wrapper around the sqlite dylib) which exposes all of its functions as accepting a RocList U8 as its one arg (and then decoding it somehow into the desired args) and then returning a RocList U8 (encoded somehow)
Richard Feldman (Jul 03 2024 at 11:58):and then the application can encode the args and decode the return value after calling it from the platform, which can take care of caching the opened dylib to prevent having to go back to the OS repeatedly, etc.
Richard Feldman (Jul 03 2024 at 11:58):this is already possible today, and does not require any new language features
Richard Feldman (Jul 03 2024 at 11:59):you could take this a step further by importing bytes (e.g. importing a List U8 into a .roc file) to make a shareable dylib
Richard Feldman (Jul 03 2024 at 11:59):you could include the bytes for all the different targets you want to support
Richard Feldman (Jul 03 2024 at 12:00):and then platforms could expose some functions for like "register dylib" which takes the raw bytes and creates a dylib out of them
Richard Feldman (Jul 03 2024 at 12:00):(or there may be a way to dlopen a pointer to bytes; I haven't checked if that exists but it's not strictly necessary for this to work since platforms can write the bytes to a file on the filesystem and then dlopen that)
Richard Feldman (Jul 03 2024 at 12:01):this is also already possible today, and doesn't require any new language features (and again, the ergonomics aren't great)
Richard Feldman (Jul 03 2024 at 12:01):this is why I think the exploration around ergonomics of all this is premature; if there is really a lot of demand for this sort of thing, nothing is blocking people from trying it out right now! :big_smile:
Kiryl Dziamura (Jul 03 2024 at 12:02):sounds like an escape hatch :grinning_face_with_smiling_eyes:
I imagine a lot of ergonomics complexity can be hidden
Richard Feldman (Jul 03 2024 at 12:12):yeah potentially
Richard Feldman (Jul 03 2024 at 12:13):personally I think we should focus on seeing how things go in practice with module params
Richard Feldman (Jul 03 2024 at 12:14):it’s always been clear that there are tons of motivating use cases for platform-agnostic packages that can do effects
Richard Feldman (Jul 03 2024 at 12:14):e.g. the entire category of “library that talks to a web service that has a REST or GraphQL API”
Richard Feldman (Jul 03 2024 at 12:15):of which there are incredibly many
Richard Feldman (Jul 03 2024 at 12:19):whereas in this case we’ve basically talked about “sqlite is a dylib rather than a separate process” and “BLAS/LAPACK are too big to reimplement in pure Roc, plus maybe the inline assembly they use is essential” - but that use case doesn’t want to useTask anyway
Richard Feldman (Jul 03 2024 at 12:20):so to me, “wait and see” feels like the correct way to proceed here :big_smile:
Brendan Hansknecht (Jul 03 2024 at 14:53):Yeah. I think I agreed with your analysis for the most part. I do think beginners asking about composing two platforms is pretty common. So there is definitely a bit more want for flexibility (especially if they don't want to get into platform dev), but currently this is a very niche feature. Sqlite and blas being the two main potential examples. With things like essentially a more configurable basic CLI being another pseudo example.
Brendan Hansknecht (Jul 03 2024 at 14:56):I need to make a libffi prototype just to see what the ergonomics really are like in practice. Maybe I can wire that into basic CLI. I think if you require a shim, the ergonomics of usage might be okish. I think allowing general flexibility without a shim is where compiler support almost certainly would be required.
Brendan Hansknecht (Jul 03 2024 at 14:57):Of course there is currently no place for the shim to exist. So even if you could do it. It means you couldn't make a library for calling sqlite without distributing and sqlite shim library as well which is separate from all the roc package ecosystem.
Brendan Hansknecht (Jul 04 2024 at 06:33):I feel like a monster right now, but I have a very basic ffi working. It includes the most type unsafe roc code I have ever written.
I have a function that takes any roc type, boxes it and returns a pointer as an opaque. I also have the reverse to go from a pointer as an opaque to a box of any type. So the ffi is all boxes with exactly 0 type safety, but it does work.
On the basic-cli ffi branch
From the little prototype, I actually learned a lot about the required types. From what I can tell there are 4 ways to make the types work here while having roc constraints and a platform in the middle:
RocObject : [ RocStr Str, RocList (List RocObject), RocU8 U8, etc ]. Very high overhead, especially with nested data. Have to map all elements of a list to a wrapped version of the element. Totally type safe. Sam Mohr (Jul 04 2024 at 06:45):Option 4 having low overhead sounds of course very enticing. This smells a lot like roc glue.
Sam Mohr (Jul 04 2024 at 06:48):If if we go with option 1 or 4, then platform devs probably wouldn't be put off too much by the perf cost. Options 2 or 3, on the other hand, would make me personally prefer to avoid the FFI styling and just use some Rust crate that provides the common code, and copy-paste/symlink the requisite FFI definitions instead
Brendan Hansknecht (Jul 04 2024 at 06:49):I mean fundamentally a similar problem. This is a larger problem than glue cause it is even more dynamic. FFI info is known at app compile time, but not at platform compile time. So having it go through the platform is forcing something dynamic to go through a static hole.
Sam Mohr (Jul 04 2024 at 06:49):In short, bad enough performance makes convenience unappealing
Brendan Hansknecht (Jul 04 2024 at 07:06):From what I can tell, 1 probably is a viable option, just terribly unsafe. It could attempt to be augmented with sideband types, but they would be user provided and not guaranteed to be correct. It would just give a way for the functions to assert they are called with the correct args.
I think 2 is just too high effort for anyone to want to use it. Like to wrap sqlite, you would have to setup idk, protobuf encode and decode for every single function....just not great.
I think 3 makes sense for a dynamic language where the data is already in a tagged form. I think the cost of tagging at runtime for anything nested is too expensive. So would be ok if you limit to non-nested primitives....which probably isn't enough. Cause that would mean no lists. I don't think it would work out for roc on practice.
For 4, I think it could be made roughly equivalent to writing effects in a platform, but a bit less safe. Though thinking about it more, linking doesn't verify function args are correctly matching between platform and app. So maybe as safe. Of course, it is a big cost ecosystem and compiler cost that roc doesn't want to consider currently.
Maybe there is a 5 and 6 option that I am missing, but currently, boxing records to make each call only one allocation sounds like the best bet for trying this out. I think I am gonna try to setup a super duper basic ffi plugin for sqlite. Just see what that looks like.
Brendan Hansknecht (Jul 04 2024 at 07:14):As a note, the boxing could theoretically be avoided, but it would require roc enabling a user to explicitly pass reference to values to the platform. Cause fundamentally, FFI needs a list of references to data (the data can be on the stack). I'll definitely have to mess around more to see if I can make that API nicer without any extea compiler features.
Brendan Hansknecht (Jul 05 2024 at 02:24):Working ffi running sqlite from basic cli.
Roc code: https://github.com/roc-lang/basic-cli/blob/ffi/examples/ffi/sqlite.roc
C++ shim: https://github.com/roc-lang/basic-cli/blob/ffi/examples/ffi/sqlite-shim.cpp
Brendan Hansknecht (Jul 05 2024 at 02:26):The ergonomics are actually approximately the same as adding an effect to a platform. Could definitely use some glue help. Also, it may use a bug in the roc type system to function. I don't think these should actually be legal effect signatures:
ffiCall : U64, Str, Box a -> Effect (Result (Box b) Str)
ffiCallNoReturn : U64, Str, Box a -> Effect (Result {} Str)
The ergonomics are actually approximately the same as adding an effect to a platform
As in, without glue generation support of some sort is 100% type unsafe and and easy way to shot yourself in the foot. But once that api is correct in the shared lib and the roc wrapping function, all the roc code after that is safe.
Brendan Hansknecht (Jul 05 2024 at 02:31):Also, I assume this would need some sort of record builder magic or something else smart, but I really that this is returning a List (List SqlVal). Really you would want to return something like List { id: I64, task: Str }. Not only is it terrible for performance due to all of the list allocations, but it also is a lot less convenient and requires handling overly generic types.
That said, given ffi has not type rules, I technically could just return a List { id: I64, task: Str }. The user of the library would just be able to shot themselves in the foot if they get the type wrong. Would return total garbage in that case.
Brendan Hansknecht (Jul 05 2024 at 16:58):So, this is totally unsafe. It requires the roc user to specify the correct output tuple for every sql query they write. That said, it is also super cool cause it avoids nested allocations and a huge perf cost:
Opening db: examples/ffi/todos.db
Preparing statement: SELECT id, task FROM todos WHERE status = :status
[[(Integer 1), (String "Prepare for AoC")]]
Now querying a second time without nested allocations!
[(1, "Prepare for AoC")]
Cleaning up
Done
The roc user is manual specifying the second output type is a List (I64, Str). If there sql query doesn't return exactly that type, they will get back garbage data.
Brendan Hansknecht (Jul 05 2024 at 17:01):Anyway, I feel like even this totally unsafe ffi feels like a super power. Basically makes it so that any platform that supports a few minimal ffi primitives can be extended to do anything. That said, it is horribly type unsafe and would be an easy way to crash or return totally garbage data to roc. (again, roughly the same ergonomics of working on a platform, but more exposed to the end users/library author).
Brendan Hansknecht (Jul 05 2024 at 17:08):Anyone have ideas on how to make it either:
a monomorphizes to in Box a. Being able to do typeOf someVar and getting back a tag would be pretty awesome here. Then could add some guard asserts at least. Brendan Hansknecht (Jul 05 2024 at 17:09):On that general note, do we think this is something worth adding to basic-cli in general? Let users play with it and extend basic-cli via ffi if they want. That said, maybe should have written the shim in rust to have access to roc_std.
Richard Feldman (Jul 05 2024 at 17:27):I think glue is likely the way to go for making it safer! It’s the exact shape of problem glue is built to solve :big_smile:
Richard Feldman (Jul 05 2024 at 17:28):I don’t think we should add it to basic-cli right now, but I think it’s a reasonable thing to discuss in the future
Richard Feldman (Jul 05 2024 at 17:29):I think if we put it in basic-cli, it might give the impression that “oh this is a thing every platform should include as a matter of course” - which I don’t think is the right impression to give right now :big_smile:
Richard Feldman (Jul 05 2024 at 17:29):but it’s great to see the proof of concept! :tada:
Brendan Hansknecht (Jul 05 2024 at 17:31):While glue helps, I think it hits an issue where this is more flexibly defined than the platform. Though maybe that is just cause I have a terrible overly flexible SQL API that doesn't actually make sense in roc.
Brendan Hansknecht (Jul 05 2024 at 17:32):But yeah, if we could run glue on an arbitrary FFI function call (or signature), that definitely would help make wrappers.
Brendan Hansknecht (Jul 05 2024 at 17:33):Richard Feldman said:
I think if we put it in basic-cli, it might give the impression that “oh this is a thing every platform should include as a matter of course” - which I don’t think is the right impression to give right now :big_smile:
Yeah, especially given it is super easy to add to any platform. It's like 3 effects.
Last updated: Jun 16 2026 at 16:19 UTC