Richard Feldman (Sep 09 2023 at 20:04): Richard Feldman (Sep 09 2023 at 20:04):having used roc glue for awhile now, I'd definitely prefer an experience where I literally run roc glue with no arguments and it regenerates my glue.
I think the way to get there is to have the platform module specify what .roc file to use for its glue. (I'd also like it to support a https URL for that file, like we do with packages - which would be annoying to write out on the command line, but which has felt fine once it's in a file.)
then we could change roc glue to have all of its arguments be optional, and change their order to be:
glue/)main.roc, and if that's an app module, generate glue for its platform - this is already behavior I want for unrelated reasons) Richard Feldman (Sep 09 2023 at 20:05):I also think this would make it nicer to contribute to platforms; there would be a universal default way to regenerate glue just like there is to build
Richard Feldman (Sep 09 2023 at 20:06):we could make it optional to specify a glue script in the platform module, although I could also see an argument for making it mandatory because it's so unrealistic to develop a nontrivial platform without glue
Luke Boswell (Jan 11 2024 at 01:53):Have we got an issue to track this proposed change? I assume this is accepted as it's a great idea
Also, is this still compatible with module params proposal?
Richard Feldman said:
have the platform module specify what .roc file to use for its glue
# from module params proposal, where could we put glue spec?
platform package [Stdout, Stdin, echo, read]
requires [main]
provides [mainForHost] to "prebuilt-hosts/"
packages [
Foo, Bar, Baz from "https://…",
Something as Smt from "https://…",
]
hosts [
echo : Str -> Task {} [],
read : Task Str [],
]
yeah I guess just put glue "/path/to/whatever.roc" at the end?
Luke Boswell (Jan 11 2024 at 02:18):Is the idea here that the package author might vendor a glue spec into the repository?
Luke Boswell (Jan 11 2024 at 02:19):Oh, nvm it could be a URL
Richard Feldman (Jan 11 2024 at 02:38):yeah the string would be the same as the string we use for packages - either path to local filesystem or URL with hash at the end
Luke Boswell (Jan 13 2024 at 21:22):I would really like roc glue to be able to accept a URL for the spec file rn. I'm working on the changes for basic-cli and I removed the glue files from the repository, made a script to manually fixup and bugs, and have it generated in CI. The issue is that I don't have a good way to reference the RustGlue.roc spec from within CI without vendoring this file into to the repository.
Brendan Hansknecht (Jan 13 2024 at 21:27):I say, go for it. Just add the feature.
Brendan Hansknecht (Jan 13 2024 at 21:27):Then if you want to be really thorough, ad RustGlue.roc as a published artifact to the repo.
Brendan Hansknecht (Jan 13 2024 at 21:29):As a short term fix, just change the CLI script for glue to recognize a url
Luke Boswell (Jan 13 2024 at 23:13):So I implemented this... well the download URL spec file part etc, and then realised it wont be as simple as I thought because we need to make the platform into a standalone package I think.
This the the current header of the spec file... it references both the platform and roc_std files locally.
app "rust-glue"
packages { pf: "../platform/main.roc" }
imports [
pf.Types.{ Types },
pf.Shape.{ Shape, RocFn },
pf.File.{ File },
pf.TypeId.{ TypeId },
"../static/Cargo.toml" as rocAppCargoToml : Str,
"../../roc_std/Cargo.toml" as rocStdCargoToml : Str,
"../../roc_std/src/lib.rs" as rocStdLib : Str,
"../../roc_std/src/roc_box.rs" as rocStdBox : Str,
"../../roc_std/src/roc_list.rs" as rocStdList : Str,
"../../roc_std/src/roc_dict.rs" as rocStdDict : Str,
"../../roc_std/src/roc_set.rs" as rocStdSet : Str,
"../../roc_std/src/roc_str.rs" as rocStdStr : Str,
"../../roc_std/src/storage.rs" as rocStdStorage : Str,
]
provides [makeGlue] to pf
Ah, makes sense. needs to be a package with files instead of just an individual file
Brendan Hansknecht (Jan 13 2024 at 23:21):Wait, can we compile it to a shared library and just send the shared library
Brendan Hansknecht (Jan 13 2024 at 23:22):Then make glue skip compiling and instead just load the plugin?
Luke Boswell (Jan 13 2024 at 23:22):Oh, the RustGlue.roc part?
Brendan Hansknecht (Jan 13 2024 at 23:22):Yeah. Make it libRustGlue.so and friends for other platforms.
Luke Boswell (Jan 13 2024 at 23:22):That looks like it might work
Luke Boswell (Jan 13 2024 at 23:23):How will we handle different architactures? compile for each?
Brendan Hansknecht (Jan 13 2024 at 23:24):Yeah, same as distributing platforms
Brendan Hansknecht (Jan 13 2024 at 23:25):Probably should do the same hash thing as well
Luke Boswell (Jan 13 2024 at 23:26):I wonder if we could implement a feature for the roc cli to build the --lib against each --target and then bundle the app and add a hash?
Luke Boswell (Jan 13 2024 at 23:27):Aside, this is taking a long time to run, not sure if its frozen or just going to take a looong time
10:23:53 ~/Documents/GitHub/roc/crates/glue/src main $ roc build --optimize --lib RustGlue.roc
Maybe one day, but I'm not sure we currently can link a shared lib for all architectures from the same system currently
Brendan Hansknecht (Jan 13 2024 at 23:28):Also, not surprising, that probably generates horrid code for llvm to optimize
Luke Boswell (Jan 13 2024 at 23:29):Luke Boswell said:
Aside, this is taking a long time to run, not sure if its frozen or just going to take a looong time
10:23:53 ~/Documents/GitHub/roc/crates/glue/src main $ roc build --optimize --lib RustGlue.roc
Oh it finished 0 errors and 0 warnings found in 288667 ms while successfully building: :sweat_smile:
Brendan Hansknecht (Jan 13 2024 at 23:29):Yeah, glue normally only does debug builds
Brendan Hansknecht (Jan 13 2024 at 23:30):And how glue is written, llvm probably inlines everything essentially and tries to do a lot of heavy optimizations on the string processing.
Brendan Hansknecht (Jan 13 2024 at 23:30):Would probably be a good case for looking at optimizing what llvm ir we generate.
Luke Boswell (Jan 13 2024 at 23:35):Do you think we would still want to support loading glue spec as a file? if we load it as a library then we don't need to rebuild it each time
Luke Boswell (Jan 13 2024 at 23:41):I think this means we can remove a lot of logic from glue as we no longer need to call roc to build anything, just load the symbol and pass in the input files
Brendan Hansknecht (Jan 13 2024 at 23:44):The original idea was that it would just be a single easy to distribute file. Due to includes, that isn't realistic.
I think we need to pick at this point. Either. commit to compiled plugins, or commit to essentially packages with multiple files that get compiled after download.
Brendan Hansknecht (Jan 13 2024 at 23:45):I think either is fine, but we should just pick one for simplicity.
Brendan Hansknecht (Jan 13 2024 at 23:45):That said, for local development, the plugin flow might be quite annoying. So have to think about that too.
Luke Boswell (Jan 13 2024 at 23:50):I'm convinced the library is the way to go.
If we can have CI generate each of the libraries with roc build --optimize --lib crates/glue/src/RustGlue.roc and add them to our nightlies then this will be a really nice workflow and remove a lot of logic from the compiler.
For tests, we can still build and cache the library locally as we have all the files -- and I think this will be faster as it isn't rebuilding each time.
Brendan Hansknecht (Jan 14 2024 at 00:12):@Richard Feldman curious if you have any thoughts. I know the original goal was definitely to have source with quick compiles, but could also just be a handful of shared libraries.
Brendan Hansknecht (Jan 14 2024 at 00:14):glue spec could instead be "https://github.com/roc-lang/rust-glue/releases/download/0.1.0/Icc3xJoIixF3hCcfXrDwLCu4wQHtNdPyoJkEbkgIElA.tar.br"
Brendan Hansknecht (Jan 14 2024 at 00:15):which just unwraps compiled libs
Richard Feldman (Jan 14 2024 at 00:15):I don't quite follow what's being proposed, sorry :sweat_smile:
can you summarize?
Brendan Hansknecht (Jan 14 2024 at 00:16):If we are to start storing glue files at a url, we can't just store RustGlue.roc. It has dependencies on other files.
Brendan Hansknecht (Jan 14 2024 at 00:16):Either, we need to change to storing a full package:
# All in package tar.bz
RustGlue.roc
roc_std/...
...
Or we need to start storing fully compiled shared libraries:
# All in tar.bz
libRustGlue.so
libRustGlue.dylib
...
This proposal would be to store and distribute the compiled shared libraries.
If you want to do local development of glue, you would need to manually compile to a shared library.
For all users, they would just point to a url package that would download the shared libaries.
Brendan Hansknecht (Jan 14 2024 at 00:20):I would assume that we would want to also do the hashing and what not that we do with package releases for the same security reasons.
Richard Feldman (Jan 14 2024 at 00:21):oh I see
Richard Feldman (Jan 14 2024 at 00:23):yeah I think optimizing for better platform author experience makes sense over glue author experience, because typically there will be many platform authors using the work of one glue author :big_smile:
Richard Feldman (Jan 14 2024 at 00:23):so distributing the shared library binaries makes sense to me
Brian Carroll (Jan 14 2024 at 15:56):I thought we were going to generate roc_std from glue. If that's the case, what other dependencies does it have?
Richard Feldman (Jan 14 2024 at 15:58):that's the plan, yeah - actually RustGlue.roc already does that!
Richard Feldman (Jan 14 2024 at 15:58):I don't think it needs any dependencies in particular
Brian Carroll (Jan 14 2024 at 16:02):I thought this proposal was based on the idea that it has dependencies but now you're saying it doesn't... I might have skipped over some details in the thread.
Brendan Hansknecht said:
If we are to start storing glue files at a url, we can't just store
RustGlue.roc. It has dependencies on other files.
Richard Feldman (Jan 14 2024 at 16:03):oh I think he just meant that it imports interface modules
Richard Feldman (Jan 14 2024 at 16:04):like RustGlue.roc itself does, as a matter of code organization, as opposed to having 100% of the glue logic being in RustGlue.roc itself
Brendan Hansknecht (Jan 14 2024 at 16:07):All or roc_std is a dependency. It is imported as a Str
Brendan Hansknecht (Jan 14 2024 at 16:07):So RustGlue.roc as a standalone file will fail to compile due to not being able to find the roc_std files.
Brian Carroll (Jan 14 2024 at 16:07):I don't see why the platform dev experience is better with shared libs than with source code. Seems strictly worse because you're limited to targets the glue developer compiled for.
Brian Carroll (Jan 14 2024 at 16:16):Also if you find a bug in glue you might want to hack a fix locally before submitting a PR upstream. Easier with source code.
Brendan Hansknecht (Jan 14 2024 at 18:07):I think the three gains for the plugin style are:
libRustGlue.dylib, but eh not a great gain)../../roc_std/src/lib.rs?). You don't really want many copies of the same files, but might be stuck with that and versioning pains if we package src.Really, I see 1 and 2 as pretty minor gains. I guess with a large enough glue in the future, maybe 1 will matter.
3 is the only piece of potential significant gain to me.
Notification Bot (Jan 14 2024 at 18:26):10 messages were moved from this topic to #compiler development > bundling modules that import blobs from paths by Richard Feldman.
Last updated: Jun 16 2026 at 16:19 UTC