One thing I really like about Elm is that there's a single well written elm/http package that lots of other packages have built off of to support various rest APIs. In Roc, if I want to write a package to support a rest API, do I need to maintain multiple versions of it for each platforms own http API?
our vision is that there is one package that exposes a Request
type, and then many platforms can interpret that request description in whatever way they want
I believe Richard called these capability modules?
They essentially can be a base for common APIs
Cool, that's an interesting idea!
Unsure if we should create new topics on Zulip or write in those that have a similar name to our question?
So I have been playing around with the rust examples and am also learning some more rust along the way since I am completely new to the language.
What I want to accomplish to get more familar with roc and writing platforms:
I want to write a little roc platform example that extends the cli
readLine
/ writeLine
example. But instead of shutting down after writing out your first and lastname I want it to repeat from the start and ask again! That seems like a reasonably small project.
But I don't want to just while loop / re-allocating the unsafe block in the cli lib.rs
main example.
I want to learn and understand how to keep the roc program alive and let it recive / emit "events" / "callbacks".
#[no_mangle]
pub extern "C" fn rust_main() -> i32 {
let size = unsafe { roc_main_size() } as usize;
let layout = Layout::array::<u8>(size).unwrap();
// I don't just want to while loop around this block and endlessly re-init the roc program...
unsafe {
// TODO allocate on the stack if it's under a certain size
let buffer = std::alloc::alloc(layout);
roc_main(buffer);
let result = call_the_closure(buffer);
std::alloc::dealloc(buffer, layout);
result
};
// Exit code
0
}
Could i get some help in how to accomplish that? :big_smile:
Currently I do not know how to keep a roc program alive... or how to send "messages" back and forth between roc and the platform.
Yeah, platforms can be a bit complex and are still in the process of being made nicer. I think the process of making them will probably be some of the biggest Roc library and abi changes in the future. There are also still a few missing pieces that make certain platform architectures painful or at least hard to make.
Anyway, to attempt to answer your questions.
1) Your goal shouldn't need any platform changes. You should just be able to use recursion in the roc app.
2) Calling a closure is unsafe, so you would need the repeated unsafe block if you want to change the platform as you mentioned. That being said, you should be able to keep the allocation and deallocation outside of the loop and just reuse the buffer.
3) Keeping a roc program alive with some basic state and event handling done in the host is currently really annoying to maybe impossible to do as expected (should hopefully be updated soonish). Currently the sdl example is blocked on issues related to that (exposing multiple functions to host and type boxing). The easier option currently is probably to define your context in Roc and recursively use it. The false-interpreter example does everything in Roc with a context for state.
Thank you for that detailed step by step @Brendan Hansknecht !
I added main
as the last line and it did indeed work:
main : Task {} *
main =
{} <- await (Stdout.line "What's your first name?")
firstName <- await Stdin.line
{} <- await (Stdout.line "What's your last name?")
lastName <- await Stdin.line
Stdout.line "Hi, \(firstName) \(lastName)!"
main
Altho I feel like this is not what I had in mind exactly, it certainly is a step in the right direction :smile:
There is no really any back and forth going on between Rust and Roc here. Would not work with a webserver platform for example. (if Roc blocks the thread in an infinite recursion "loop" and never lets the platform do anything anymore).
I think part of my confusion might come from being a JS developer. The browser does not shut down after a "main" function has completed.
I am so used to event based systems. Gonna take a look at that SDL github issue. Thank you.
Also, it is possible to have a webserver, it's just that currently it would have to be a webserver with only a single function exposed to the host. That function either would get called directly on each request, or it would return a list of closures to be called on specific requests. So definitely doable with current roc, but not a nice event loop structure where you expose a small handful of functions (eg: model, view, update).
There is a pr with a simple webserver. It probably still works, but hasn't been touched in a while. It opts for the first option of calling the same function for every request. It may also match what you are looking for.
I am hijacking this, hope this is okay.
So what I do not quite understand yet is how the link from my C API in the platform is made to the Roc code. Where is declared that I have to export roc__mainForHost_1_exposed_generic
which then is mainForHost
in the roc code? I do not see where that connection is made. And in addition: Am I allowed to export any symbol and it is automatically linked to roc? So for example if I have a fn get_four() -> i32 { 4 }
and I want to make that available in roc (of course via C calling convention, not via Rust calling convention), how do I export it to roc?
@Brendan Hansknecht but that "one function exposed to the host" is the "main" is it not?
The one function exposed to the host could be any one function with any set of arguments and return value. So it could be an event handler or something similar.
@Matthias Beyer we definitely need to start improving the docs around this especially for platform builders. Essentially the package-config.roc defines the c API. It specifies which functions are imported and exported from roc. Then roc, when compiling, expects those functions with some minor name changes. Exposed functions from roc are definitely the most complex because they may return closures.
All functions provided to roc that are specific to the platform are in the form roc_fx_...
There is also another set of functions that must always be provided that are used for memory management and panicking: roc_panic
, roc_alloc
roc_realloc
, rpc_dealloc
also 2 others that might go away in the Future: roc_memcpy
and roc_memset
Then, if roc is exposing only a simple function, the function is just roc__<name>_1_exposed
. If it is returning a closure, another set of extra information is exposed so the host can understand the closure.
So if I implement a function roc_fx_hereBeDragons
with a C ABI, I can call hereBeDragons
from roc? Is it that simple?
Have to add to the Package-Config.roc
as well. Like the function signiture
Also, roc forces it to be an effect so calling is normally done through tasks, not directly, but it is pretty similar.
Can you explain a bit about how tasks and effects work? I've been trying to expose my own platform functions to my Roc code too, but haven't had much success yet.
I will try, but I may not have the best answer. If I am off base, hopefully someone will jump in and correct me.
Essentially an Effect is a promise or a future. A wrapper that is capable of giving you a value. This is a type built into Roc. I think (though I could be wrong) that it is required as the return type from all platform functions. Roc deals with this translation internally, and the platform can still just return raw types.
A Task on the other hand is a common Roc module that is used to deal with the possibility of failure. A task is defined as Task ok err : Effect.Effect (Result ok err)
. It enables nice chain-ability when it comes to the potential of errors. For example, Task.await
, takes a task, runs a transformation on result if it is not an error and then returns a task that might or might not be an error.
Thank you, Brendan! That's very helpful.
Where can I find documentation for Roc Platforms? (and the Roc Platforms themselves)
it doesn't exist yet - my plan has been to write the tutorial for application authors first and then to move on to platform authors!
Ah, okay... how would you recommend I learn a platform's API? Maybe your "Edges of Cutting-Edge Languages talk?" Just figuring out where to go from looking at the initial Hello World examples (and this seems to be the next step).
basically yes. Most examples have platforms written in rust or zig, so pick your favorite out of the 2 and see how the platform connects to the app
Actually, what you just said just clicked lol... I'm looking to be an Application author.
for an app author the platform is just a bunch of roc code
usually there is a Task
module
and then there can be some extra stuff around that
@Jason Hobbs I also gave a talk on zig showtime with a demo on how to work with platforms
you can skip the roc intro and go straight to the demo
if you're curious that is, but the examples folders are also good
So funny.. I just finished watching that like 5 minutes ago. Great talk on it... has me curious about using Node.js as a platform for kicks
hm interesting thought, I hadn't considered that
and thanks :), it was my first public talk
Probably would not be a pleasant experience. I would assume cffi in node would be annoying. Also, with rocs current compilation model and setup, I think extra work would need to be done manually or via updates to the compiler.
maybe it could work if you use zig to import node as a c lib?
Is that a thing?
idk I'm just spit balling lol
:rolling_on_the_floor_laughing:
lol I guess I imagined the Platform-to-Application API as being more loosely connected. Makes sense for performance it isn't.
A roc application is essentially a static library that the platform depends on.
Is it the LLVM that makes it compatible across different languages? Or is there some other common protocol for the static libraries to work? (Might be an odd question - I don't have much experience linking libraries or using systems-level languages)
it's the C ABI
cffi is a standard protocol supported by most languages.
oh, okay.. nice.
If an application wants to use any pseudo-random number generation, will the application need to use a platform that exports a pRNG function?
(or a noise function, from like a clock or thermometer)
I'm just thinking of simple randomness ergonomics like Python's random.choice([a, b, c, ...])
, but maybe randomness in FP is its own can of worms?
in elm it's enough for the platform to provide a seed value
(After quickly reading about RNG in Elm...) Are you referring to Elm's step
function? Or would an effectful function like generate
be preferable?
Perhaps RNG-supportive Roc platforms should provide a seed generator function like getSeed : {} -> U128
, and then platform-agnostic randomness packages can provide ergonomic abstractions. (Ex: choice : List a, U128 -> a
)
Of course, platforms could also provide their own domain-specific abstractions. (Hmm, I can't think of a good example.)
it would have to be a {} -> Random.Random U128
function; the value returned by getSeed is different every time, so it needs to be somehow wrapped to keep the rest of the language pure
Is that to prevent inaccurate memoization? Is that wrapper somehow recognized as opaque?
purity (same input always gives the same output) is very core to roc, a bunch of things don't make sense without it
for instance we can currently freely reorder definitions in a let block. That's not true without purity
and yeah a wrapper like Seed : [ @Seed U128 ]
is opaque and has no runtime cost versus just the U128
the Random.Random
(or Random.Generator
) is meant to chain randomness
it's defined as Random a : [ @Random (Seed -> { value : a, newSeed : Seed }) ]
and on that type you can define map2
and after/andThen
ect
Gotcha! Thanks for that great explanation. Are those built-in Elm types that you expect to be emulated by many Roc platform authors?
this is similar to http I think where there could be a generic Random
package with the types and map2/andThen, and the platform provides the function to get a new seed
actually that would be getSeed : Task Seed *
in practice I think
and of course you could also have a constantSeed : U128 -> Seed
or something like that in the random package, mostly for testing
Perfect. I wonder what percentage of platforms will support RNG! I'm used to it as a language built-in, but I suppose that some application categories need it more than others (game engines vs. text editors?). That might be reflected in different platform authors seeing it as critical vs. bloat.
yes. Note that with the constantSeed
you could use any entropy. So e.g. if your application has users, you could use the username as the initial seed (need some hashing or something to turn it into a number, but it's doable)
so even if the platform does not provide it conveniently, if it gives you anything that is varying in a meaningful way (time of day, ...) you could use that to make an effectively random seed
(this is all not cryptographic of course)
yeah, actually the package that eventually became elm/random
was originally implemented entirely as a third-party package with no language integration or dependency on effects: https://github.com/mgold/elm-random-pcg
it was eventually moved into elm/random
because it became the recommended way to do things, and at that point it also picked up the generate
task
so in Roc it could just be implemented as a third-party library along the same lines as the original Elm implementation, which would naturally imply that it's platform-agnostic!
that Elm implementation is based on https://www.pcg-random.org/ (hence the name random-pcg
), although it had to do some tricks to work around how browsers don't support 64-bit integers natively (or didn't at the time), such as using two 32-bit integers instead
so an ideal Roc implementation would probably have a public API with a lot of similarities to the Elm one, but an internal implementation that might look more like a Roc port of https://github.com/rust-random/rand because of the 64-bit integer support
Well that sounds like a good ## Goals
section to me!
Let the RNG begin: https://github.com/JanCVanB/roc-random
Contributors welcome!
yoooooo, that's awesome! lmk if you have any questions about the Elm version - Max and I used to go to San Francisco Elm meetups back in the day and talk about it when he was developing it :smiley:
Thanks, right now I'm having fun contorting my brain to translate code with [personally-semi-advanced math concept]
from [personally-unlearned programming language]
to [objectively-unfinished programming language]
while utilizing [newly-proposed platform-agnosticity pattern]
to future-proof it for use with [nonexistent potential platforms]
through [nonexistent publishing system]
:sweat_smile:
I love the ambition! :smiley:
The nice part is that even without a packaging system, it can probably be copied into any random platform and used.
to be fair, that's how you end up being the person who says "I wrote the random generation library everybody uses in Roc" :grinning_face_with_smiling_eyes:
@JanCVanB this is great!
Not sure if you're aware of this already but if you're trying to figure out what some unfamiliar Elm code is doing, then Debug.log is your friend. For example you could clone the repo and then start inserting _ = Debug.log "some_var" some_var
into let
blocks all over the place, then run the tests and see what comes out on the console.
I (a newborn Rustling) am trying to create a Bevy https://docs.rs/bevy/latest/bevy/ "hello world" platform by adapting examples/hello-rust/platform
. While I could post the wide and wild variety of errors I'm getting, what files would you expect me to need to change to get Bevy to compile (just a minimal hello popup)? I'm currently changing platform/sec/lib.rs
and platform/Cargo.toml
.
that page you link has an example. In theory you could just have the platform not use any roc code at all, so changing the platform rust code to that example should do it? (swap main
for rust_main
)
So no Cargo.toml
changes necessary?
Last night I tried injecting Bevy main into Rust main function and got so many angry messages haha
oh, yes sure
you'd need to add bevy to the dependencies section of the Cargo.toml
typically you can find what to add on crates.io
: https://crates.io/crates/bevy
so bevy = "0.6.0"
in this case
Great - sounds like I'm on the right track :)
Bevy recommends "dynamic" in its dependency definition, but that causes errors about bevy missing "rlib"
Last I saw, without bevy dynamic, it was complaining about hundreds of missing files that started with underscores - can post details later
Thanks!
hmm you could try getting bevy to run just outside of a roc context
it's hard to know now if the problem is with roc or with rust or bevy
I got their hello world working normally, but when I integrated with Roc it was hard to say what failed
Here's where I'm stuck, if anyone can help :smile: https://github.com/rtfeldman/roc/commit/88c94bf3031feb7f9bad4008cd403bd0b2f42db5
(This is not intended as a WIP contribution, it's just pushed to the repo for convenient diff-ing)
Well, this seems solidly beyond my Rust abilities now:
ld: symbol(s) not found for architecture x86_64
error: linking with
cc failed: exit status: 1
= note: clang-7: error: invalid version number in '-mmacosx-version-min=12.0'
I seem to be experiencing https://github.com/rust-lang/rust/issues/91372 via https://github.com/rust-lang/rust/issues/91781, but the fixes they mention don't seem to work for me. :shrug:
@Richard Feldman could this be because we have some C code in the rust platforms? I always forget why that was needed, but maybe information gets lost in the process?
it's needed until we can switch over to surgical linking for everything
https://users.rust-lang.org/t/error-when-compiling-linking-with-o-files/49635/5?u=rtfeldman
@JanCVanB is there a commit somewhere I can check out so I can try it on my machine?
Yes! https://github.com/rtfeldman/roc/commit/88c94bf3031feb7f9bad4008cd403bd0b2f42db5
Thank you both :)
hm, I can't build bevy at all on my M1 (even outside Roc) because I'm running into https://github.com/bevyengine/bevy/issues/928#issuecomment-943882168 :sweat_smile:
@JanCVanB are you on an M1 or x64 mac?
x64
Their helloworld & breakout examples build for me
I tried this with the latest nightly of rust, which is supposed to have the fix for #91372, and that didn't work either
so I guess a reasonable question is: how important is it that it be bevy specifically? :sweat_smile:
I have an x64 mac I can try it on tomorrow
Haha, to me? Very. To the community? Probably not.
I haven't tried the fix yet, but might get some coding in tonight to try it :) thanks!
shot in the dark: clang-7: error: invalid version number in '-mmacosx-version-min=12.0'
- when you run clang --version
what does it say?
oh also what version of macOS are you on?
That's a good question - I'm afk right now, but that was my error when running with some recommended env var like MACOSX_DEPLOYMENT_TARGET=12.0 && cargo run examples/hello-rust/Hello.roc
I'm on 12.0.1
[nix-shell:~/code/clones/roc]$ clang --version
clang version 7.1.0 (tags/RELEASE_710/final)
Target: x86_64-apple-darwin
Thread model: posix
InstalledDir: /nix/store/zfh3npfhfjjgwi0dqpriklip5k15ppmj-clang-7.1.0/bin
I'll take this debugging into https://github.com/rtfeldman/roc/issues/2432
Unrelated to this debugging, Bevy's API looks super-resistant to first-level Roc-ification:
mut
parametersHowever, second-level Roc-ification seems promising:
Example for a 2D platformer (like Super Mario): The platform says that there must be a Player
that jump
s, but the Roc app must specify when the jump happens via keymapping, how the jump looks via animation curve, when the jump is allowed via state machine rules, and how the jump is interrupted via collision rules. :smiley:
Can a platform requires
multiple things?
I can make a one-requires
platform work, even if the def isn't named main
.
Now I'm trying to get this two-requires
platform to work:
platform "breakout"
requires {} { ballSpeed : Dec, paddleSpeed : Dec }
exposes []
packages {}
imports []
provides [ ballSpeedForHost, paddleSpeedForHost ]
effects fx.Effect {}
ballSpeedForHost : Dec
ballSpeedForHost = ballSpeed
paddleSpeedForHost : Dec
paddleSpeedForHost = paddleSpeed
but I get this error:
── MISSING REQUIRES ────────────────────────────────────────────────────────────
I am partway through parsing a header, but I got stuck here:
1│ platform "breakout"
2│ requires {} { ballSpeed : Str, paddleSpeed : Str }
^
I am expecting the requires keyword next, like
requires { main : Task I64 Str }
not currently, a quick fix is to accept a record with the fields that you want
That would be great, though since I didn't see a RocRecord
in roc_std
, I didn't know how to proceed with that approach.
you can expose multiple things from the package config
hmm wait no
well kinda
you can expose a record, and that turns into a rust struct in a predictable way
on the rust side you need to make sure the struct sorts its fields first by alignment, then by name of the field, and it's annotated with #[repr(C)]
Folkert de Vries said:
on the rust side you need to make sure the struct sorts its fields first by alignment, then by name of the field, and it's annotated with
#[repr(C)]
I just tried this, and it works!
I never expected to dive this much into platform development, but it's smooth!
@Folkert de Vries are nested Records expose-able?
I'm getting this: emitted runtime error function "The
8.IdentId(0) function could not be generated, likely due to a type error."
(though it may be from some unrelated mistake)
Also, any tips on passing a record field like List F32
to Rust? Or should I just experiment?
it should just work
and nested records also should just work
only tricky thing is the order of record fields on the C/Rust/... side
How would you write/order the matching Rust struct for this Record?
a : {
b : Str,
c : U8,
d : U32,
e : List U8,
f : List U32,
g : {
h : U32,
i : I32,
j : List U32,
k : List I32},
x : List { y : Str, z : Str } }
Allright, adding an alias for clarity
T :
{
h : U32,
i : I32,
j : List U32,
k : List I32,
}
U :
{
b : Str,
c : U8,
d : U32,
e : List U8,
f : List U32,
g : T,
x : List { y : Str, z : Str }
}
We can start with T
: the rule is sort first by alignment, then by field name.
(for this you need to know some stuff about alignment, I can elaborate if you're not sure how that works)
For this record, we're done immediately (lists store a pointer and a length (as a nat/usize) and hence have the alignment of a pointer which is equal to the alignment of a usize).
{
h : U32,
i : I32,
j : List U32,
k : List I32,
}
Compound structures have the alignment equal to the maximum of the alignment of their fields. That means we can sort the fields by alignmenthere:
{
c : U8,
d : U32,
b : Str,
e : List U8,
f : List U32,
g : T,
x : List { y : Str, z : Str }
}
because of the names you picked this is also already sorted by field name
I see, so small-to-large alignment order goes like this?
I8 == U8
I16 == U16
I32 == U32
I64 == U64 == Str == List * == Record *
I128 == U128
Record *
is not correct there. Record fields are stored on the stack (while list elements are stored on the heap)
that means that what types a record's fields have influence the record's alignment
so a record { x : u8 }
has an alignment of 1, but a record { x : U128 }
has an alignment of 16
Side question: Are Nat
s weird? Would V : { r: U32, s: U64, t: Nat }
require different struct definitions for different systems?
yes, I was thinking about that too when writing this out
in particular this is kinda weird if you mix targets, e.g. a rust project with a wasm frontend and normal x86 backend
then you need to order the fields depending on your target
which you can do, but it's not awesome
Maybe Nat
s shouldn't pass between platform and app
it's inevitable
e.g. a List *
also has a different alignment
because it contains nats and pointers
Is this alignment-based sorting algorithm...
a) an unbending constraint?
b) a performance optimization?
c) a placeholder?
Just curious whether pure-alphabetical sorting is possible/optimal.
it's a performance optimization
when we don't sort by alignment, we can get gaps between fields
e.g. { x : u32, y: i128 }
Maybe this sorting would benefit from a compiler flag? --alphabetical-structs
an i128
has an alignment of 16, meaning that its start address needs to be a multiple of 16
If a platform developer wants to pass strange values or be flexible somehow
assuming we start at 0, we put in the u32
field , taking up 4 bytes. But now we need to add 12 bytes of padding before we can put in the i128
field
the opposite order is not beneficial in this example, but in general sorting by alignment minimizes the amount of padding you need
Ah, that makes sense
e.g. C says: definition order is what we use, if that means we need to insert padding to get the alignment right then that is what we do
rust says: don't worry about it, we will lay it out in an optimal way
except if you use #[repr(C)]
With an algorithm toggle, platform developers would be responsible for optimizing performance via field name alphabetization, which might be a nice tradeoff
(basically to let developers use that C mode you just described)
well the order of record fields is not well-defined
I think the best long-term solution is to have a compiler command to generate the entire definition, including ordering - but in the short term, maybe we could make a quick compiler flag that prints out the sorted versions of all type aliases in the program?
like it just takes the AST of the type alias, sorts everything, and then runs the formatter on it to get a string
and then prints the string
so at least platform authors don't have to figure all that out in their heads until we have that long-term solution
I'd love that. To be clear, I don't have an immediate need for such complex records, but it's right around the corner (especially since only one def can be passed from app to platform right now, I think)
yeah and tag unions are even harder than records :sweat_smile:
@Folkert de Vries I successfully implemented a Roc-to-Rust record/struct with 3 Str
s and 2 U32
s in the (WIP) plotting platform, but now I'm having trouble simply adding a second F32
to the game platform.
https://github.com/JanCVanB/roc-bevies/pull/3/commits/768ad5d
Do you know why only the first f32
is being heard? Depending on which field comes first in the struct, the debug print looks like
Config { ballSpeed: 400.0, paddleSpeed: 0.0 }
or
Config { paddleSpeed: 400.0, ballSpeed: 0.0 }
.
:confused:
I'm probably just not seeing a typo or something, but at this point I can't see it.
The behavior is the same if I change them both to i16
s or u16
s.
Whoa, when I add another F32
between them, it hears the 1st as the 1st and the 2nd as the 3rd!
Roc source: config = { ballSpeed: 111, manualPadding: 222, paddleSpeed: 333 }
Rust debug: Config { ballSpeed: 111.0, manualPadding: 0.0, paddleSpeed: 222.0 }
This must be related to alignment, but I don't know how. Maybe... Roc is writing the record with 8-byte alignment (because it's represented with an 8-byte pointer) but then Rust isn't recognizing the padding between the 4-byte F32
s?
https://github.com/JanCVanB/roc-bevies/pull/3/commits/8eed55c
I'm seeing no problems when using
fn roc_config() -> Config {
extern "C" {
#[link_name = "roc__configForHost_1_exposed_generic"]
fn roc_config_internal(_: &mut Config);
}
let mut config = Config::default();
unsafe { roc_config_internal(&mut config) };
config
}
#[repr(C)]
#[derive(Default, Debug)]
struct Config {
pub ballSpeed: f32,
pub paddleSpeed: f32,
}
I'm using the _generic
version of configForHost here, which takes a pointer that it writes the value into
that is the safer way of doing this, because simply returning values (like -> Config
) seems to break in some cases across the C FFI boundary
Well that's nice that the extern and unsafe can both be absorbed into the roc_config
function - that's how I'll do every Roc call from now on.
This Rustling never would have found that _generic
solution, thank you very much
yeah I wonder if we should actually remove the non-generic version
since it causes a bunch of problems
on the other hand, it's not entirely clear why it causes problems
Okay, here's a new request... is passing a function as a record field possible?
Since only one def can be passed from an app to a platform, I see only two ways of passing both static primitive data and dynamic function logic:
I've tried the following, but the Rust compiler is complaining about methods & traits, so I'm out of my Rust depth :sweat_smile:
use std::boxed::Box;
use std::ops::Fn;
#[derive(Default)]
#[repr(C)]
pub struct Config {
pub ballSpeed: f32,
pub paddleSpeed: f32,
pub todo: Box<dyn Fn(i16) -> i16>,
}
the tui
example exposes a record with functions and values
the trick here is that the field in the record only contains the captured environment of the function (usually that is a unit value, because the exposed function is top-level)
we then expose an additional extern function to actually call the function
tui
is in Zig - does the same system apply to Rust?
yes both languages use the C api that we expose
Phew, complex platform APIs hurt my brain :sweat_smile:
extern fn roc__mainForHost_1_Update_caller(ConstModel, *const RocStr, [*]u8, MutModel) void;
that's the downside. Hopefully you get that right once and can then build a nicer api on the rust/zig side
also platform author ergonomics might be most underdeveloped part of Roc tooling right now :big_smile:
we have no way to help generate these things yet
but we can in the future!
Unrelated, I find this file hilarious:
platform "roc-plotters/bitmap-chart"
requires {} { config : Config }
exposes [ Config ]
packages {}
imports [ Config.{ Config } ]
provides [ configForHost ]
configForHost : Config
configForHost = config
The cherry on top is that the file name is Package-Config.roc
:laughter_tears:
config x 10 == 10x engineering
this reminds me of the greatest talk of all time https://www.youtube.com/watch?v=yL_-1d9OSdk
config config, config config config. Config
Richard Feldman said:
yeah and tag unions are even harder than records :sweat_smile:
Is this a solved problem? I just ran into a use case for a tag union :stuck_out_tongue:
what kind of tag union?
we have support for RocResult
, and an enum is actually easy
I suppose it's an enum: format : [ Bitmap, Svg ]
Is there an enum example in examples/
?
no but you can treat this as if you're sending a u8
over
this might also work
#[repr(u8)]
enum {
Bitmap = 0,
Svg = 1,
}
not sure if that plays nicely with extern functions
How about lists like List I64
or List List { label: Str, values: List I64 }
?
I see a List I64
passing between Zig & Roc in examples/quicksort/
, but I'm hoping to pass complex lists from Roc to Rust (without knowing their length at compile-time, which requires Vec
s?). Will that require any tricks that (a) aren't demonstrated in examples/quicksort/
or (b) look different in Rust than in Zig?
Thanks for all of this help on these API patterns :) I hope to produce some useful examples of these patterns, so that future devs can simply copy/paste the goods!
(Side note, I just fought with Rust generics for 2 hours and won!!! I feel like I just received my second Rustling badge, haha)
(perhaps I should have read a Rust guide/tutorial/book, but instead I'm just guessing the syntax until the compiler finally gives in! :sweat_smile:)
you can use RocList
from roc_std
although you have to carefully think about who owns the memory
@Philippe Hosey-Vinchon Do you want to help me make some fun Roc platforms? :) I'm pretty comfortable in Roc after a couple of months with it, but Rust is brand new for me! After fighting with Rust generics today, I can see that platform-ifying existing Rust libraries can have surprise traps, some of which are best defused by a more-experience Rustacean than myself :crab:
:loudspeaker: In fact, anyone is welcome to collaborate! Help me Roc-ify some useful+popular Rust libraries: :smiley:
Plots/charts/graphics: https://github.com/JanCVanB/roc-plotters
Games/simulations/UIs: https://github.com/JanCVanB/roc-bevies
Hi, new here. I read through a bunch of platform posts here, but after looking at the examples, I don't think I understand platforms at all. Should I, some time in the future, decide to write a simple cli tool in Roc, would there be a standard platform for this purpose? Would it only support nix or windows at one time? I'm confused why there are rust examples and zig examples, and why anyone writing an app would care what language the platform is written in?
Hi, Andrew! I'd recommend using examples/cli/
, as it's the only platform thusfar that supports general-purpose CLI apps.
In addition to the example CLI apps in that dir (form.roc
, countdown.roc
, echo.roc
), I have some more (mostly-redundant) examples in this abandoned repo.
Regarding Nix/Windows, what operating system are you on? I'm using macOS, and Nix is super convenient for hopping into a Roc-friendly environment.
Sorry, I meant *nix, as any random linux. I run Windows, but built Roc in WSL Ubuntu last night, and will be playing with it there.
The platforms that are in the examples seem(and this is fine for this purpose) custom built out with just enough functionality to support the example they work for. This is part of my confusion. Would you expect there to be a batteries included platform that would meet most needs of a particular type of app?
I recommend this thread: https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/A.20platform.20for.20production.20CLIs
(TL;DR: Not yet, but it's an inevitability that can be accelerated by demand & contributions)
Regarding Rust/Zig, Roc app developers shouldn't care what language their platform is written in.
I think about Roc platforms like this:
examples/
don't use any unique libraries, I expect that the languages chosen in there were mostly chosen arbitrarily, to demonstrate Roc's flexibility.I hope that makes platforms make more sense, but now that I've written it all out I realize this could just confuse the issue :sweat_smile:
JanCVanB said:
- Meanwhile, a Roc platform can use any of its language's features (Rust crates, Zig libraries, etc.), and a Roc app can use any Roc features (Roc libraries, etc.).
- Therefore, the only reasons for a Roc platform developer to choose Rust vs. Zig (vs. something else?) is the libraries they want to leverage and their personal preferences.
These still feels like you would be in a place where a Roc app developer could need to also write their underlying platform(or copy/paste from a previous similar app and add new lib Y). Am I misunderstanding?
in the future platforms will be available in the package registry (which doesn't exist yet!) so you can just say what platform you want and the compiler download and install it behind the scenes, just like any other package
but since we don't have a package registry yet, for now it's all copying and pasting onto the local filesystem :big_smile:
add new lib Y). Am I misunderstanding?
No, you're correct. If an app wants to do something that its platform doesn't support, then it will need to fork+modify the platform or find another platform. That's by design, so that app consumers/end-users can be absolutely certain what any app might do when they run it. For example, if an app's platform doesn't have networking or file-writing functionalities, you know it's (relatively) safe to run.
but the idea is for obtaining a platform to feel equivalent to obtaining a framework in another language: you just list it as a package dependency and it gets installed for you
of note, it's theoretically possible for a platform author to provide a C FFI (via normal function calls, just asking for the name of a dynamic library along with binary encoders and decoders for its arguments and return value) but of course any platform that does that can no longer offer any security guarantees, so it's a strictly opt-in thing :big_smile:
(that's not a special language feature or anything, it's just an observation that any platform author can in fact do that if they want to and it will work)
I know this isn't strictly a platform question, but I think it's related to @Richard Feldman 's comment: If I wanted to use that handy dandy Random lib, would I always expect to get just the source(and drop it right in my app folder?), or is a Roc compiled binary of some kind possible/intended?
For now, copy/paste it or use a Git submodule. In the indeterminate future, something like npm or pypi for Roc will exist.
However, I don't believe Roc libraries will ever be pre-compiled, as compilation only happens in conjunction with a specific platform. Roc libraries must perform a delicate dance of platform-agnosticity, so that any Roc app (on any Roc platform) can use them.
@JanCVanB if you are interested, I would be up for helping you with some of the platform stuff and rust type fun. I should theoretically have an ok understanding of the underlying types and rust.
Are there any stability guarantees regarding roc_std
?
not really, if we think a change is necessary we make it and then update all code in the compiler repo
so far RocList
is not used that much, RocStr
a bit more but even there it's just more convenient to quickly turn that into a String
/&str
or whatever the host language wants
there will be eventually, but a major part of the reason the repo is private now is to set expectations around the fact that Roc isn't ready for production use yet, so we may make breaking changes on short notice! :big_smile:
When passing a List from a platform to a Roc application, does the List always have to be reference counted? compiler/str/README.md
mentions that the platform can't assume anything about Lists provided by Roc application, but AFAICT doesn't say anything about Lists passed the other way around.
I'm asking because valgrind is reporting that I'm leaking memory which I tracked down to being caused by returning a unique List with a capacity to a Roc application.
All list are reference counted in general. If Roc returns a list and it has only 1 reference count left (which is the normal case), it is the host's job to free the memory.
If it had a higher than 1 reference count, it would be expected that the host lowers the reference count by one when dropping the list.
that misses capacity. But the capacity as it's currently used in roc_std
is an older idea that we'll likely not use any more
currently, RC grows from isize::MIN
upwards. At 0, the RC reaches "infinity" and an object with that refcount will live forever
a positive value in that location used to indicate a capacity. But we now want to store the capacity on the stack
so in effect that means that a positive value in the refcount position is invalid
Folkert de Vries said:
that misses capacity. But the capacity as it's currently used in
roc_std
is an older idea that we'll likely not use any more
Oh ok, good to know.
but we plan to transition that to basically exactly what Vec
does in rust
with capacity stored in the struct, not behind the pointer
Hi all, I'm trying to build Roc (update_zig_09 branch) on Guix but I'm facing unresolved LLVM symbols when linking roc_cli. LLVM_SYS_130_PREFIX is set, and llvm-config is in PATH. I have zero knowledge of Rust, any hint?
that branch doesn't quite work for most things related to lists/strings.
if you still want to try it: what errors are you getting exactly?
Looks like every LLVM symbol is undefined.
I went with the 09 branch because that's the zig version that happens to be packaged on Guix, but I guess I can downgrade.
you can set the zig version with ROC_ZIG=path/to/zig
if you don't want to touch the system version
So, should I give up the 09 branch and try with master? Will that get rid of the llvm unresolveds?
I think trying trunk
is the best approach for now. Even if you get this branch to compile, it won't really give you much
because most examples will just fail to compile
note that trunk uses llvm12
Ok, I'll try that over the weekend, thanks.
btw it does look like llvm cannot be found by the linker. it looks like gcc is used?!
so, using trunk may not solve that problem
Ups, good catch. Looks like I depend on clang instead of clang-toolchain in the package definition, that might well be the problem.
I compiled Roc in Ubuntu on Windows(WSL). Compiles seem to take a really long time. Is this expected? image.png
WSL is very slow if the code is stored on the windows side, but compiled from the linux side
@Brian Carroll knows more about how to fix this
I saw something related to that, and copied all the code into the linux side, but it didn't make any appreciable difference. I'll try compiling it in native windows this weekend. (I only compiled it from linux because I thought it'd be a smoother process. Hint: it was bumpy)
we don't currently support windows outputs, so that means that if you compile a program with the compiler, you need to then run it in WSL
also, nobody has actually tried compiling on windows I think. I guess it should work?
I remember trying and running into a lot of issues, but that was quite a while ago on a mini pc without a lot of storage. I don't remember why (maybe for llvm-as, maybe for something else), but I needed to compile llvm from source and it ate loads and loads of disk space.
I have a quick question, when specifying the path to a platform, does the path have to be absolute?
I'm asking because relative paths don't seem to work, but not sure.
app "Hello"
packages { pf: "../bin/examples/cli/platform" }
imports [ pf.Stdout ]
provides [ main ] to pf
nevermind, I was being silly lol
Folkert de Vries said:
Brian Carroll knows more about how to fix this
Just to add a bit more detail, the exact issue I had was that my clone of the repo, and therefore the Cargo target
folder, was on the Windows side in the NTFS filesystem. But I was running cargo test
and cargo build
in the Linux terminal, so Cargo was also using the ~/.cargo
folder from the Linux ext4 filesystem. So it must have been talking to both filesystems, maybe copying stuff between them, probably going through some sort of translation layer...
The fix that worked for me was to move everything to the Linux filesystem. There were no other steps to it, just that.
But the commands that Andrew ran in his screenshot don't look like they would have the same issue since all the files are on the Linux side, and he's running Roc rather than Cargo.
But I'm not sure what else to suggest since the only WSL-specific issue we know about is this one
I'll try running the same command on my machine and see what happens.
brian@acer:~/code/roc$ time target/debug/roc build examples/cli/form.roc
🔨 Rebuilding host... Done!
🎉 Built examples/cli/form in 2268 ms
real 0m2.314s
user 0m1.997s
sys 0m0.380s
brian@acer:~/code/roc$ time target/debug/roc build examples/cli/form.roc
🔨 Rebuilding host... Done!
🎉 Built examples/cli/form in 317 ms
real 0m0.327s
user 0m0.400s
sys 0m0.064s
brian@acer:~/code/roc$ time target/debug/roc build --precompiled-host examples/cli/form.roc
🎉 Built examples/cli/form in 201 ms
real 0m0.210s
user 0m0.299s
sys 0m0.031s
From the Windows "System > About" menu:
Device name acer
Processor 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 2.80 GHz
Installed RAM 8.00 GB (7.80 GB usable)
Device ID 8E0330F2-1463-4F8C-8E66-5A35508DA65E
Product ID 00325-96756-33883-AAOEM
System type 64-bit operating system, x64-based processor
Pen and touch Touch support with 10 touch points
Why does hello.roc link against all these libraries on MacOS ?
AudioUnit.framework, Cocoa.framework, CoreAudio.framework, CoreVideo.framework, IOKit.framework, Metal.framework, QuartzCore.framework, Security.framework
I think those libraries are needed for some programs, so for now they are included in all programs
Is that something the platform does?
Yeah, today Roc platforms can't specify which OS libraries they need, but there's a plan for that. Until then, link ALL the things!
i haven't finished the tutorial yet but is there an argc/argv interface in the common platforms ?
all of that linking will go away when we have surgical linking working on macOS - currently that only works on Linux, but the x86 macOS implementation is partially complete
the current linking solution on trunk
has a bunch of problems, but it was a lot faster to get up and running than the long-term implementation we're working towards :big_smile:
so wheneever any platform needs to link something new, we just add it in there
eventually that whole .rs
file will be unnecessary and get deleted
Aaron said:
i haven't finished the tutorial yet but is there an argc/argv interface in the common platforms ?
I don't think any platform in the examples currently exposes this, though it shouldn't be too hard to do. This could be a candidate for a standard platform.
(Digression: I wonder if argc/argv should be more standardized in the language, or always handled in the host, because right now if the host exposes them they must be wrapped in an Effect - but (I think) they need not be effect-ful, since they're supplied by the kernel)
Could this be a similar interface to the way Elm accepts flags on init?
Yeah so the classic argc
and argv
interface is just the way that you represent an array of strings in C. In Roc, the same information would be represented as List Str
. So I think platforms would expose command line arguments to Roc as List Str
. Once it's in Roc you'll generally want to parse the CLI arguments for your program into a nice Tag Union. That parsing library can be pure Roc code. There are similar libraries in Elm for URLs, for example.
and we do allow passing in flags in e.g. the tui
platform
at least in theory
Does tui have an example of that today or no?
mainForHost : { init : ({} -> Model) as Init, update : (Model, Str -> Model) as Update, view : (Model -> Str) as View }
mainForHost = main
so currently it takes {}
as the flags value
but that can be turned into something else
Nice!
I'm not sure if this is called tree shaking or lazy loading or what...
Is there any way for a platform to include/exclude imported libraries in/from the eventual machine code if the app does/doesn't use them? Example: A Rust-based platform imports two different stdout formatting crates, A and B. An app might do either Task.await (Stdout.lineA "hello")
or Task.await (Stdout.lineB "hello")
, and therefore the executable might only need crate A or crate B but not both. Is that a possible optimization, or does CFFI not grant us that level of compile-time insight?
(For a more realistic example, I'm thinking that a video game platform might have a lot of heavy dependencies, but an individual video game might not need everything - physics simulation, programmatic map generation, etc.)
do you mean things the host imports?
or like Roc code
maybe to put it another way: are we talking about machine code that was originally Rust code getting eliminated, or machine code that was originally Roc code getting eliminated?
Rust
My question might be: "Can we tree-shake Rust code that isn't used by Roc?"
so I believe that's possible in the general case, although it's out of scope for Roc itself
I remember reading about this
it's something like there's a way to tell compilers (and I think Rust does this by default?) to emit each function into its own separate section of the binary
The answer is definitely possible. Requires function sections and the linker doing GC
which then lets you run a tool afterwards that says "if nothing ever calls into this section, eliminate the section"
yeah exactly :point_up:
so it depends on whether the host compiler supports emitting code that way
@Brendan Hansknecht I don't think surgical linking should affect that, right?
but yeah, regardless - it'd be something you would do without getting Roc involved at all
there's just a tool you can run on the final binary that does it, assuming the binary was originally set up in a way that permits the tool do that
The surgical linker wouldn't be able to clean up unused sections. They would already be baked into the binary.
right, but it wouldn't affect those sections being there
It also would probably cost way to much performance
like you could still run the "DCE unused sections" tool separately later
and it would still work, because the surgical linker didn't (for example) combine those sections or anything
like it wouldn't touch them, so if they were set up that way by the host, they'd still be set up that way after surgical linking, and thus still eligible for DCE by a third-party tool
Yeah, the section would just sit there and never get called into
I see, and this is machine code? assembly? C?
And the app developer would run a Dead Code Elimination tool on the output of the Roc CLI's compile/build?
This will be more common if kitchen-sink-style (all-in-one) platforms are popular. Idk that they will be
This will also be more common if app developers don't want to fork their platform, for whatever reasons
I'm new to systems languages, so maybe this DCE thing is commonplace?
Motivation for this question: I saw a cool looking Bevy (Rust video game engine) plugin in my GitHub feed and thought "add it to the pile! no downside! wait..."
I wonder if you could have an empty shell host, where you'd opt into components somehow, and then codegen or let the host compile in just the chosen components.
I see, and this is machine code? assembly? C?
machine code!
And the app developer would run a Dead Code Elimination tool on the output of the Roc CLI's compile/build?
right, exactly - assuming they cared to :big_smile:
I wonder if you could have an empty shell host, where you'd opt into components somehow, and then codegen or let the host compile in just the chosen components.
that's already if you do load them on startup (or later) with dynamic linking
I hadn't thought about it, but it's conceivable we could let platforms ship with multiple binaries so they could load some of them on the fly if they wanted to
I hope platform authors wouldn't use dynamic linking that searches the system path (unless absolutely necessary, e.g. it's needed on macOS for system libraries, and in practice glibc
needs it on Linux too)
because inevitably that would mean that some people would go to run the platform and get an error on startup :confused:
I wasn't thinking dynamic linking, more I'll include these definitions in the compiled platform if you ask for them. (And if you don't, your compile goes boom before you've ever delivered an app to anyone)
ah gotcha
But now I'm thinking about configuring kernels with make menuconfig
, and I'm getting feint.
:laughing:
since the DCE thing already works without adding anything to the language, I'd say we should start with that and see if there's a compelling use case for more than that in practice!
@Andrew Thompson Am I following? :fear:
app "fear_the_night"
packages { pf: "the-changeling" }
imports []
provides [ 666 ] to pf
666 = { realPlatform: "./roc/examples/cli/platform" }
@JanCVanB sorry, I'm too new to follow. I looked at compiling roc code, but haven't made my way through the tutorials.
No worries, I'm probably misunderstanding your "empty shell host" - it conjures horror movie vibes :laughing:
There might be another way to support dead code elimination at the platform level, platform language permitting.
Por example, let's say every publicaclly exposed function in Rust gets its own feature flag (this can be done automatically). Roc knows what functions are being exposed by rust, and can therefore emit the feature flags for the exact functions it needs. Rustc would take care of DCE natively.
I believe something similar should be possible with zig, but don't know.
That definitely would work. That being said, I think generally speaking DCE doesn't matter too much, but if we want it in the future, there are many ways to add it. Personally I think feature flags are a less optimal solution. They require rebuilding the host and would have to be uniquely done for each host language.
Yes, the language by language basis would be the worst part.
You are also correct that DCE isn't super beneficial in most situations. Though I could imagine it being so for embedded systems. But as mentioned, there are alternative approaches for now.
Is the expectation that platforms would be delivered normally as binaries?
yeah the plan is that every platform ships with precompiled binaries
for each target they support (e.g. Mac, Linux, Windows, etc.)
I'm strongly opposed to the typical way that package managers handle native code, which is to say "build it on your local machine, and then some percentage of the time have installation fail with something cryptic like ld: could not resolve symbol __cxx_mumbo_jumbo in libwhatever.so
"
the design goal I have in mind is that from any clean operating system install, you can download the roc
binary and use it to start writing Roc code, including automatically downloading and installing any platforms you specify, without needing any other tools installed on the system, and with platform compilation always succeeding
ideally I'd also like the package repo to verify this - e.g. if the platform says it supports x64 Linux, then it looks for a precompiled binary and verifies that it's valid ELF, etc.
so as an application author, the only programming language you should need to know to use Roc is Roc, and the only tool you should need to program Roc applications is roc
Roc should never tell you "ok it's time to go learn about shared libraries now"
unless the particular platform you're using is for building shared libraries or something :stuck_out_tongue:
separately, I also like the idea of (but haven't thought it through completely yet) having a requirement that published platforms declare all their dynamically linked dependencies, and provide links for how to obtain them
So as a platform author the number of binaries you need to build is (number of OS's) x (number of CPUs), right? I guess that's the same as anywhere else you download pre-built binaries.
so if it's absolutely unavoidable to depend on a dynamic library (e.g. for licensing reasons), we can actually detect when it's missing right when you install the platform, and say "hey your system can't install this platform because it needs this other thing to be installed first; here are the instructions the platform author wrote for what to do if you find yourself in this situation: _______"
Jose Quesada said:
Yes, the language by language basis would be the worst part.
You are also correct that DCE isn't super beneficial in most situations. Though I could imagine it being so for embedded systems. But as mentioned, there are alternative approaches for now.
100% true. I have been messing around with embedded Roc and it currently wastes a lot memory. The current problem is that Roc compiles for speed rather than size, and it's unused functions are not garbage collected. So it is actually roc side problems, not host side.
I think I need to downgrade it's symbols to no longer be global so that they will properly get garbage collected.
its unused functions are not garbage collected
do you mean builtins?
I think unused functions written in Roc by the author of the program shouldn't get emitted in the first place :thinking:
No, I mean for example, roc__mainForHost_1_exposed_generic
when only roc__mainForHost_1_exposed
is used.
Oh, and __muloti4
which we still always emit
Hmm, actually I am wrong about the size cost here. The real problem looks to be the loss of inlining. The binary bloats by way more than the size of the roc app. So when the app was completely rust, it inlined and greatly simplified most of the code. Since part is in roc, the inline potential is much more limited.
In this case, the Roc app is 1582 bytes, but using roc leads to the binary bloating by 4120 bytes.
I like the idea of 4120 bytes being considered bloat...
I've seen codebases that generate multi-GB server applications and yet they amount to little more than request -> call 0|n other things -> response.
Noob question: would this (having the platforms already compiled) simplify cross compiling or would the same problems still exist?
If the platform is precompiled for all the architectures you want to target, roc essentially just hooks into that and is extremely portable. So cross compilation should be very easy. But yeah, depends on the platform supporting your target and llvm at least supporting the CPU architecture you're targeting.
I have an issue that perhaps someone else has run into?
I am implementing a custom platform, only modifying the roc_main
function from a hello example. Everything in the platform side seems to not have an issue, however, I seem to be getting an error that doesn't make sense to me. Provided the following code:
app "hello-rust"
packages { pf: "platform" }
imports []
provides [ main ] to pf
greeting =
hi = "Hello"
name = "World"
"\(hi), \(name)!\n"
main = greeting
I am getting the following error when running:
Finished dev [unoptimized + debuginfo] target(s) in 0.24s
Running `target/debug/roc ./examples/speak-aloud/Speak.roc DDDDD`
🔨 Rebuilding host... Done!
SIOD ERROR: unbound variable : Hello
SIOD ERROR: comma-not-inside-backquote
The Hello
it seems to be referring to is hello = "Hello"
because if I change what's inside the quotes, the error changes with it
Any guesses as to a possible cause?
I can run other examples and code without issue, so I seem to be running into something else, but no idea how to trubleshoot in this case
Your platform sees Hello
as a variable, perhaps it is expecting double quotes inside the string to interpret it as a string?
So something like:
app "hello-rust"
packages { pf: "platform" }
imports []
provides [ main ] to pf
main = "\"Hello\""
Thank you Anton. I'll check it out. It shouldn't be the case, as all I am doing is slightly modifying the original hello rust example. Though it very much could be as most of the functions aren't documented yet :sweat_smile:
I'm not following completely, for which functions would you like to see more documentation?
Feel free to push a new branch to the repo with the speak-aloud example if you'd like anyone to help make it work.
@Anton, you were 100% right. I didn't really understand what you meant. I got it to work. I was spawning a child thread, but was expecting the program to panic on invalid input, but that didn't happen. I assumed the output I was getting was actually comming from Roc, but it was the child process. Thanks again.
Happy to help :)
Why does link.rs mess environment variables? If I'm reading correctly, looks like it's clearing everything except HOME/PATH. That's a problem on Guix, it relies on environment variables to find includes and libs. Also, why isn't it using CC when building C host files?
link.rs
was cobbled together and has grown naturally. It really is not how we want to do linking in the long term. It works for most people, so it generally doesn't get messed with too much. I don't remember why, but I know that not clearing environment variables led to breakages on some machines
I know at one point there were some modifications to get it working on nixos (not sure if they got merged). Those might help with getting it working on guix.
Overall, please feel free to modify it and open pull requests. It could use some cleanup.
In the future, platforms will be downloaded precompiled and/or specify their own build commands and linking will always go through the roc linker.
LTO+musl:
jacereda@mbp ~/src/roc [env]$ size examples/hello-world/hello-world
text data bss dec hex filename
14971 568 1664 17203 4333 examples/hello-world/hello-world
Why glibc? IMHO it would be a good thing to decouple Roc from the C compiler/runtime, just use whatever CC/CFLAGS/LDFLAGS are defined... That would simplify quite a bit link.rs.
the surgical linker is already that way, it just only works on Linux right now
how do I invoke that?
pass --roc-linker
to the Roc compiler
honestly I think it might simplify conversations around link.rs
if we renamed it to temporary_pile_of_hacks.rs
so it's more self-descriptive about how we should be thinking about investing in making it more robust :laughing:
what size does that yield on your box?
and/or put a comment at the top explaining how it's going away? :thinking:
I'm on macOS at the moment, so I'm not sure offhand
Jorge Acereda said:
what size does that yield on your box?
What specifically are you looking at the size for? Also, the roc linker + some changes to how platforms specify build commands would enable the platform to choose if it wants to use musl or not.
to elaborate a bit, the surgical linker basically says "give me a working executable that attempts to dynamically link the Roc application entrypoint" (typically the main
provided by the application author) and after the Roc application gets compiled into machine code, I will incorporate it into that executable and replace that dynamic linking with static linking after the fact
Well, I do care about code bloat. I'm considering if this would be an appropriate language for embedded.
so what you start out with is an executable that has some dynamic linking for the Roc application entrypoint, and what you end up with is that same executable except it's no longer doing dynamic linking for the Roc application entrypoint; instead, the entire compiled application has been added to the binary
the Roc stdlib doesn't depend on libc at all
it just says the host needs to provide a few functions (roc_alloc
, roc_dealloc
, roc_realloc
, roc_memcpy
, and roc_panic
)
it's completely up to the host how they want to implement those, they just need to be provided
it's also completely up to the host how the executable gets compiled in the first place
so there's no need for glibc or any libc at all
at least from Roc's perspective
how does that differ from -static plus -flto or --gc-sections?
it's completely unrelated to them
we don't call ld
or anything like that
the reason we call it the surgical linker is that literally all it's doing is adding the compiled application bytes to the binary, and then rewriting some headers to make some direct calls into dynamic ones
so if you want to do another pass afterwards with a separate tool to GC sections, you totally can!
but the Roc linker is narrowly scoped to only incorporate the compiled application into the precompiled host binary executable
that's its entire job, it doesn't do anything else
which is both super fast and also gives platform authors maximum flexibility
because any linking system needs to do that at the bare minimum
do you mean patching the elf headers?
yeah exactly
and Mach-O on macOS - that one's in progress
Jorge Acereda said:
Well, I do care about code bloat. I'm considering if this would be an appropriate language for embedded.
Very cool. Great to know the goal. For the embedded case, I don't think you will touch any roc linking stuff at all. Instead, I believe that you would just have roc emit a static library. Then you would have the embedded targeting platform deal with all of the linking.
That is what I am doing here when targeting the nrf chip on the microbit.
and does that work for static and dynamic executables?
oh, also currently the surgical linker works for x64 ELF only - that's probably relevant!
Also, have you considered APE?
cosmopolitan libc
having a single binary across OSs could be really nice
never heard of APE or cosmopolitan libc
https://justine.lol/cosmopolitan/index.html
That would be tagential to Roc. That would be a platform decision.
right - from the Roc compiler's perspective, the goal is to be completely decoupled from libc
so platform authors can use whatever libc they want, or decline to use libc altogether
and everything still works!
with cosmopolitan and some hacks you can end up having a single executable with OpenGL graphics running on Windows/Linux/NetBSD/FreeBSD, I did some experiments and it's totally possible... The only showstopper so far was OpenBSD that has some security measures that would need to be circumvented, but it's also possible with some more work.
https://github.com/jacereda/cosmogfx
The main problem that I believe will limit roc for embedded is that roc and the platform are two separate chunks of code. They are not compiled together. They can't take advantage of inlining. This can add a huge number of bytes to an executable. In the roc-microbit case, using roc costs about 2500 bytes more than just using rust due to the loss of inlining.
Of course, depending on the platform boundary this could potentially be greatly reduced.
In reality, the solution is probably LTO between the platform and the Roc app, but that is hard because LTO is dependent on the specific version of LLVM. So it would be very brittle. Also more complex to setup in general.
is having a C codegen out of question?
well, another alternative could be to use -flto
that way the embedded platforms could inline just fine
C codegen technically could be done, but I don't think anyone in the project would really want to support that.
heh, I've actually thought about it for exactly that use case
it's a big can of worms though
"that use case" being compiling platforms and applications together into one executable
that can be optimized as a whole
Also, lto require llvm version matching which is doable, but brittle. If roc is using llvm version 13 and the c compiler for your device is only on gcc or is llvm 12, lto won't work.
the original idea was to do that via LLVM bytecode, but...yeah, that :point_up:
that would be a nightmare to maintain
in comparison, we could emit (for example) C99 and say "here you go, do whatever you want with it"
and that's at least a stable interface!
I guess the dependency on the LLVM version is just a matter of leaving LLVM mature a bit?
Not really. LLVM intentionally wants people not to depend on that so that they can change it if they every think of better representations or other ideas.
I see... that makes a C (or zig?) backend more interesting then.
I guess theoretically it should stabilize, but LLVM still will never make promises about the stability or that tools will work when dealing with multiple versions.
Jorge Acereda said:
I see... that makes a C (or zig?) backend more interesting then.
As weird as that sounds. 100%.
That or accepting the loss of inlining between platform and application/ architecturing platforms in ways that make it matter less.
For example, I think things are made worse by embedded rust. Embedded rust is a giant pile of "zero" cost abstractions. The issue is that they depend on inline to be abstracted away. So when we break the inlining, I think it causes the rust to bloat. I think C or rust without the abstractions could be done in a way that the inlining basically didn't cause any problems.
Oh also, we have to figure out to optimize roc for size. Currently we can only optimize for speed or not at all. Some reason size optimization isn't exposed through the libraries we use.
Also, emitting C or Zig doesn't innately solve this. If I was still trying to use embedded rust and you emit C, I still wouldn't get inlining without all of the llvm hassles.
well you could probably run a script to translate the C to Rust
But emitting C does fix it for C and C++ and Zig. Also, probably easier to get a C compiler with the right llvm version to match my rust compiler.
Brendan Hansknecht said:
Oh also, we have to figure out to optimize roc for size. Currently we can only optimize for speed or not at all. Some reason size optimization isn't exposed through the libraries we use.
I think optimizing for speed may be the same as -O2: https://llvm.org/doxygen/CodeGen_8h_source.html
Ah, that would make sense. So it looks like Os
is O2
minus some things and Oz
is Os
minus loop vectorization: https://stackoverflow.com/questions/15548023/clang-optimization-levels
So I just tested adding optimizing for size to roc. Makes quicksort 3x larger. I guess something we do in our llvm generation really depends on some of the more aggressive optimizations.
Ah, nvm. link.rs
was still compiling the platform in debug mode. That makes way more sense.
Ok, after updating link.rs
to also build for size, quicksort goes from 256K to 148K.
So definitely works
Though I guess most of that is attributed to zig having ReleaseSmall
(https://github.com/roc-lang/basic-cli/tree/main/examples)
Hi all, I am a beginner to Roc, so I might overlook something, but I can't explain the following incomplete output when running time.roc:
main =
start <- Utc.now |> Task.await
dbg start # => [time.roc 14:9] @Utc 1685614693863401198
{} <- slowTask |> Task.await # => Tried to open a file...
finish <- Utc.now |> Task.await
dbg finish # => [time.roc 18:9] @Utc 1685614693863545706
duration = Utc.deltaAsNanos start finish |> Num.toStr
dbg duration
Stdout.line "Completed in \(duration)ns"
slowTask : Task.Task {} []
slowTask =
path = Path.fromStr "not-a-file-but-try-to-read-anyway"
result <- File.readUtf8 path |> Task.attempt
when result is
_ -> Stdout.line "Tried to open a file..."
The dbg output of start and finish is shown, as well as Stdout.line from slowtask, but no output of dbg duration or no Stdout.line with durarion output. What could be a reason for this? Is this a bug? Thanks!
If you remove that |> Num.toStr
after the Utc.deltaAsNanos
would the dbg print?
I could be confusing with other topics, but I think there's some bugfixing happening around Num.toStr
and I'm just wondering if that would be the culprit as to why dbg duration
would not print :thinking:
Fábio Beirão said:
If you remove that
|> Num.toStr
after theUtc.deltaAsNanos
would the dbg print?
You're right! dbg duration then prints out! But there is no Stdout.line display of duration. The problem with Num.toStr probably still prevents that, no error is shown, just no output. Thanks anyway!
Just out of curiosity, which roc version are you using? ( roc --version
)
My fault! I was using a version of perhaps 1.5 weeks old. Before answering, I did an update to most recent source and build it, and the duration now is displayed:
Tried to open a file...
Completed in 59922ns
Thanks for being patient !
Absolutely not your fault :) I am happy I could help, with my very very limited roc knowledge. :pray:
I'm trying to write a platform in Zig and I get the following error when running roc run
thread 'main' panicked at 'There were still outstanding Arc references to module_ids', crates/compiler/load_internal/src/file.rs:1560:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
We will need to see more source to diagnose that. Probably has nothing to do with zig, but instead roc loading or specific parsing issues.
Try roc check
may reveal the real issue, if not, we probably need to see the code and mess around to figure out what is going on.
I thi k I've seen this error or similar before, something like the way Interface or Package modules imported incorrectly. It's a shame we dont have a better error here yet, but if we can get a minimal repro maybe that would help.
Though I think Agu's work implementing module params design will cleanup and fix those issues soon.
Yeah, sounds like what I remember when it was last hit.
Last updated: Jul 05 2025 at 12:14 UTC