Stream: beginners

Topic: Platform question


view this post on Zulip Martin Stewart (Oct 22 2021 at 14:51):

One thing I really like about Elm is that there's a single well written elm/http package that lots of other packages have built off of to support various rest APIs. In Roc, if I want to write a package to support a rest API, do I need to maintain multiple versions of it for each platforms own http API?

view this post on Zulip Folkert de Vries (Oct 22 2021 at 14:55):

our vision is that there is one package that exposes a Request type, and then many platforms can interpret that request description in whatever way they want

view this post on Zulip Lucas Rosa (Oct 22 2021 at 15:01):

I believe Richard called these capability modules?

view this post on Zulip Lucas Rosa (Oct 22 2021 at 15:01):

They essentially can be a base for common APIs

view this post on Zulip Martin Stewart (Oct 22 2021 at 15:08):

Cool, that's an interesting idea!

view this post on Zulip Oliver Schöning (Oct 28 2021 at 13:41):

Unsure if we should create new topics on Zulip or write in those that have a similar name to our question?

So I have been playing around with the rust examples and am also learning some more rust along the way since I am completely new to the language.
What I want to accomplish to get more familar with roc and writing platforms:

I want to write a little roc platform example that extends the cli readLine / writeLine example. But instead of shutting down after writing out your first and lastname I want it to repeat from the start and ask again! That seems like a reasonably small project.
But I don't want to just while loop / re-allocating the unsafe block in the cli lib.rs main example.
I want to learn and understand how to keep the roc program alive and let it recive / emit "events" / "callbacks".

#[no_mangle]
pub extern "C" fn rust_main() -> i32 {
    let size = unsafe { roc_main_size() } as usize;
    let layout = Layout::array::<u8>(size).unwrap();
    // I don't just want to while loop around this block and endlessly re-init the roc program...
    unsafe {
        // TODO allocate on the stack if it's under a certain size
        let buffer = std::alloc::alloc(layout);

        roc_main(buffer);

        let result = call_the_closure(buffer);

        std::alloc::dealloc(buffer, layout);

        result
    };

    // Exit code
    0
}

Could i get some help in how to accomplish that? :big_smile:
Currently I do not know how to keep a roc program alive... or how to send "messages" back and forth between roc and the platform.

view this post on Zulip Brendan Hansknecht (Oct 28 2021 at 15:29):

Yeah, platforms can be a bit complex and are still in the process of being made nicer. I think the process of making them will probably be some of the biggest Roc library and abi changes in the future. There are also still a few missing pieces that make certain platform architectures painful or at least hard to make.

Anyway, to attempt to answer your questions.
1) Your goal shouldn't need any platform changes. You should just be able to use recursion in the roc app.
2) Calling a closure is unsafe, so you would need the repeated unsafe block if you want to change the platform as you mentioned. That being said, you should be able to keep the allocation and deallocation outside of the loop and just reuse the buffer.
3) Keeping a roc program alive with some basic state and event handling done in the host is currently really annoying to maybe impossible to do as expected (should hopefully be updated soonish). Currently the sdl example is blocked on issues related to that (exposing multiple functions to host and type boxing). The easier option currently is probably to define your context in Roc and recursively use it. The false-interpreter example does everything in Roc with a context for state.

view this post on Zulip Oliver Schöning (Oct 28 2021 at 20:33):

Thank you for that detailed step by step @Brendan Hansknecht !

I added main as the last line and it did indeed work:

main : Task {} *
main =
    {} <- await (Stdout.line "What's your first name?")

    firstName <- await Stdin.line

    {} <- await (Stdout.line "What's your last name?")

    lastName <- await Stdin.line

    Stdout.line "Hi, \(firstName) \(lastName)!"

    main

Altho I feel like this is not what I had in mind exactly, it certainly is a step in the right direction :smile:
There is no really any back and forth going on between Rust and Roc here. Would not work with a webserver platform for example. (if Roc blocks the thread in an infinite recursion "loop" and never lets the platform do anything anymore).
I think part of my confusion might come from being a JS developer. The browser does not shut down after a "main" function has completed.
I am so used to event based systems. Gonna take a look at that SDL github issue. Thank you.

view this post on Zulip Brendan Hansknecht (Oct 28 2021 at 20:58):

Also, it is possible to have a webserver, it's just that currently it would have to be a webserver with only a single function exposed to the host. That function either would get called directly on each request, or it would return a list of closures to be called on specific requests. So definitely doable with current roc, but not a nice event loop structure where you expose a small handful of functions (eg: model, view, update).

view this post on Zulip Brendan Hansknecht (Oct 28 2021 at 21:00):

There is a pr with a simple webserver. It probably still works, but hasn't been touched in a while. It opts for the first option of calling the same function for every request. It may also match what you are looking for.

view this post on Zulip Brendan Hansknecht (Oct 28 2021 at 21:01):

Pr link

view this post on Zulip Matthias Beyer (Oct 30 2021 at 10:29):

I am hijacking this, hope this is okay.

So what I do not quite understand yet is how the link from my C API in the platform is made to the Roc code. Where is declared that I have to export roc__mainForHost_1_exposed_generic which then is mainForHost in the roc code? I do not see where that connection is made. And in addition: Am I allowed to export any symbol and it is automatically linked to roc? So for example if I have a fn get_four() -> i32 { 4 } and I want to make that available in roc (of course via C calling convention, not via Rust calling convention), how do I export it to roc?

view this post on Zulip Oliver Schöning (Oct 30 2021 at 10:35):

@Brendan Hansknecht but that "one function exposed to the host" is the "main" is it not?

view this post on Zulip Brendan Hansknecht (Oct 30 2021 at 16:30):

The one function exposed to the host could be any one function with any set of arguments and return value. So it could be an event handler or something similar.

view this post on Zulip Brendan Hansknecht (Oct 30 2021 at 16:35):

@Matthias Beyer we definitely need to start improving the docs around this especially for platform builders. Essentially the package-config.roc defines the c API. It specifies which functions are imported and exported from roc. Then roc, when compiling, expects those functions with some minor name changes. Exposed functions from roc are definitely the most complex because they may return closures.

view this post on Zulip Brendan Hansknecht (Oct 30 2021 at 16:38):

All functions provided to roc that are specific to the platform are in the form roc_fx_...

view this post on Zulip Brendan Hansknecht (Oct 30 2021 at 16:41):

There is also another set of functions that must always be provided that are used for memory management and panicking: roc_panic, roc_alloc roc_realloc, rpc_dealloc also 2 others that might go away in the Future: roc_memcpy and roc_memset

view this post on Zulip Brendan Hansknecht (Oct 30 2021 at 16:43):

Then, if roc is exposing only a simple function, the function is just roc__<name>_1_exposed. If it is returning a closure, another set of extra information is exposed so the host can understand the closure.

view this post on Zulip Matthias Beyer (Oct 30 2021 at 16:45):

So if I implement a function roc_fx_hereBeDragons with a C ABI, I can call hereBeDragons from roc? Is it that simple?

view this post on Zulip Brendan Hansknecht (Oct 30 2021 at 16:46):

Have to add to the Package-Config.roc as well. Like the function signiture

view this post on Zulip Brendan Hansknecht (Oct 30 2021 at 16:48):

Also, roc forces it to be an effect so calling is normally done through tasks, not directly, but it is pretty similar.

view this post on Zulip Tom Small (Oct 31 2021 at 16:45):

Can you explain a bit about how tasks and effects work? I've been trying to expose my own platform functions to my Roc code too, but haven't had much success yet.

view this post on Zulip Brendan Hansknecht (Oct 31 2021 at 21:03):

I will try, but I may not have the best answer. If I am off base, hopefully someone will jump in and correct me.

Essentially an Effect is a promise or a future. A wrapper that is capable of giving you a value. This is a type built into Roc. I think (though I could be wrong) that it is required as the return type from all platform functions. Roc deals with this translation internally, and the platform can still just return raw types.

A Task on the other hand is a common Roc module that is used to deal with the possibility of failure. A task is defined as Task ok err : Effect.Effect (Result ok err). It enables nice chain-ability when it comes to the potential of errors. For example, Task.await, takes a task, runs a transformation on result if it is not an error and then returns a task that might or might not be an error.

view this post on Zulip Tom Small (Nov 02 2021 at 03:15):

Thank you, Brendan! That's very helpful.

view this post on Zulip Jason Hobbs (Dec 01 2021 at 18:15):

Where can I find documentation for Roc Platforms? (and the Roc Platforms themselves)

view this post on Zulip Richard Feldman (Dec 01 2021 at 18:16):

it doesn't exist yet - my plan has been to write the tutorial for application authors first and then to move on to platform authors!

view this post on Zulip Jason Hobbs (Dec 01 2021 at 18:19):

Ah, okay... how would you recommend I learn a platform's API? Maybe your "Edges of Cutting-Edge Languages talk?" Just figuring out where to go from looking at the initial Hello World examples (and this seems to be the next step).

view this post on Zulip Folkert de Vries (Dec 01 2021 at 18:21):

basically yes. Most examples have platforms written in rust or zig, so pick your favorite out of the 2 and see how the platform connects to the app

view this post on Zulip Jason Hobbs (Dec 01 2021 at 18:22):

Actually, what you just said just clicked lol... I'm looking to be an Application author.

view this post on Zulip Folkert de Vries (Dec 01 2021 at 18:22):

for an app author the platform is just a bunch of roc code

view this post on Zulip Folkert de Vries (Dec 01 2021 at 18:23):

usually there is a Task module

view this post on Zulip Folkert de Vries (Dec 01 2021 at 18:23):

and then there can be some extra stuff around that

view this post on Zulip Lucas Rosa (Dec 01 2021 at 20:50):

@Jason Hobbs I also gave a talk on zig showtime with a demo on how to work with platforms

view this post on Zulip Lucas Rosa (Dec 01 2021 at 20:50):

you can skip the roc intro and go straight to the demo

view this post on Zulip Lucas Rosa (Dec 01 2021 at 20:51):

if you're curious that is, but the examples folders are also good

view this post on Zulip Jason Hobbs (Dec 01 2021 at 20:51):

So funny.. I just finished watching that like 5 minutes ago. Great talk on it... has me curious about using Node.js as a platform for kicks

view this post on Zulip Lucas Rosa (Dec 01 2021 at 20:52):

hm interesting thought, I hadn't considered that
and thanks :), it was my first public talk

view this post on Zulip Brendan Hansknecht (Dec 01 2021 at 20:53):

Probably would not be a pleasant experience. I would assume cffi in node would be annoying. Also, with rocs current compilation model and setup, I think extra work would need to be done manually or via updates to the compiler.

view this post on Zulip Lucas Rosa (Dec 01 2021 at 20:54):

maybe it could work if you use zig to import node as a c lib?

view this post on Zulip Brendan Hansknecht (Dec 01 2021 at 20:55):

Is that a thing?

view this post on Zulip Lucas Rosa (Dec 01 2021 at 20:56):

idk I'm just spit balling lol

view this post on Zulip Lucas Rosa (Dec 01 2021 at 20:56):

:rolling_on_the_floor_laughing:

view this post on Zulip Jason Hobbs (Dec 01 2021 at 21:14):

lol I guess I imagined the Platform-to-Application API as being more loosely connected. Makes sense for performance it isn't.

view this post on Zulip Brendan Hansknecht (Dec 01 2021 at 21:16):

A roc application is essentially a static library that the platform depends on.

view this post on Zulip Jason Hobbs (Dec 01 2021 at 21:22):

Is it the LLVM that makes it compatible across different languages? Or is there some other common protocol for the static libraries to work? (Might be an odd question - I don't have much experience linking libraries or using systems-level languages)

view this post on Zulip Lucas Rosa (Dec 01 2021 at 21:23):

it's the C ABI

view this post on Zulip Brendan Hansknecht (Dec 01 2021 at 21:23):

cffi is a standard protocol supported by most languages.

view this post on Zulip Jason Hobbs (Dec 01 2021 at 21:25):

oh, okay.. nice.

view this post on Zulip jan kili (Jan 06 2022 at 10:10):

If an application wants to use any pseudo-random number generation, will the application need to use a platform that exports a pRNG function?

view this post on Zulip jan kili (Jan 06 2022 at 10:11):

(or a noise function, from like a clock or thermometer)

view this post on Zulip jan kili (Jan 06 2022 at 10:19):

I'm just thinking of simple randomness ergonomics like Python's random.choice([a, b, c, ...]), but maybe randomness in FP is its own can of worms?

view this post on Zulip Folkert de Vries (Jan 06 2022 at 10:27):

in elm it's enough for the platform to provide a seed value

view this post on Zulip jan kili (Jan 06 2022 at 10:47):

(After quickly reading about RNG in Elm...) Are you referring to Elm's step function? Or would an effectful function like generate be preferable?

Perhaps RNG-supportive Roc platforms should provide a seed generator function like getSeed : {} -> U128, and then platform-agnostic randomness packages can provide ergonomic abstractions. (Ex: choice : List a, U128 -> a)

Of course, platforms could also provide their own domain-specific abstractions. (Hmm, I can't think of a good example.)

view this post on Zulip Folkert de Vries (Jan 06 2022 at 10:55):

it would have to be a {} -> Random.Random U128 function; the value returned by getSeed is different every time, so it needs to be somehow wrapped to keep the rest of the language pure

view this post on Zulip jan kili (Jan 06 2022 at 10:59):

Is that to prevent inaccurate memoization? Is that wrapper somehow recognized as opaque?

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:01):

purity (same input always gives the same output) is very core to roc, a bunch of things don't make sense without it

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:02):

for instance we can currently freely reorder definitions in a let block. That's not true without purity

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:03):

and yeah a wrapper like Seed : [ @Seed U128 ] is opaque and has no runtime cost versus just the U128

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:03):

the Random.Random (or Random.Generator) is meant to chain randomness

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:04):

it's defined as Random a : [ @Random (Seed -> { value : a, newSeed : Seed }) ]

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:04):

and on that type you can define map2 and after/andThen ect

view this post on Zulip jan kili (Jan 06 2022 at 11:09):

Gotcha! Thanks for that great explanation. Are those built-in Elm types that you expect to be emulated by many Roc platform authors?

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:15):

this is similar to http I think where there could be a generic Random package with the types and map2/andThen, and the platform provides the function to get a new seed

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:15):

actually that would be getSeed : Task Seed * in practice I think

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:17):

and of course you could also have a constantSeed : U128 -> Seed or something like that in the random package, mostly for testing

view this post on Zulip jan kili (Jan 06 2022 at 11:22):

Perfect. I wonder what percentage of platforms will support RNG! I'm used to it as a language built-in, but I suppose that some application categories need it more than others (game engines vs. text editors?). That might be reflected in different platform authors seeing it as critical vs. bloat.

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:40):

yes. Note that with the constantSeed you could use any entropy. So e.g. if your application has users, you could use the username as the initial seed (need some hashing or something to turn it into a number, but it's doable)

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:41):

so even if the platform does not provide it conveniently, if it gives you anything that is varying in a meaningful way (time of day, ...) you could use that to make an effectively random seed

view this post on Zulip Folkert de Vries (Jan 06 2022 at 11:41):

(this is all not cryptographic of course)

view this post on Zulip Richard Feldman (Jan 06 2022 at 16:23):

yeah, actually the package that eventually became elm/random was originally implemented entirely as a third-party package with no language integration or dependency on effects: https://github.com/mgold/elm-random-pcg

view this post on Zulip Richard Feldman (Jan 06 2022 at 16:23):

it was eventually moved into elm/random because it became the recommended way to do things, and at that point it also picked up the generate task

view this post on Zulip Richard Feldman (Jan 06 2022 at 16:25):

so in Roc it could just be implemented as a third-party library along the same lines as the original Elm implementation, which would naturally imply that it's platform-agnostic!

view this post on Zulip Richard Feldman (Jan 06 2022 at 16:26):

that Elm implementation is based on https://www.pcg-random.org/ (hence the name random-pcg), although it had to do some tricks to work around how browsers don't support 64-bit integers natively (or didn't at the time), such as using two 32-bit integers instead

view this post on Zulip Richard Feldman (Jan 06 2022 at 16:28):

so an ideal Roc implementation would probably have a public API with a lot of similarities to the Elm one, but an internal implementation that might look more like a Roc port of https://github.com/rust-random/rand because of the 64-bit integer support

view this post on Zulip jan kili (Jan 07 2022 at 01:29):

Well that sounds like a good ## Goals section to me!

view this post on Zulip jan kili (Jan 07 2022 at 01:29):

Let the RNG begin: https://github.com/JanCVanB/roc-random

view this post on Zulip jan kili (Jan 07 2022 at 01:30):

Contributors welcome!

view this post on Zulip Richard Feldman (Jan 07 2022 at 02:04):

yoooooo, that's awesome! lmk if you have any questions about the Elm version - Max and I used to go to San Francisco Elm meetups back in the day and talk about it when he was developing it :smiley:

view this post on Zulip jan kili (Jan 07 2022 at 02:11):

Thanks, right now I'm having fun contorting my brain to translate code with [personally-semi-advanced math concept] from [personally-unlearned programming language] to [objectively-unfinished programming language] while utilizing [newly-proposed platform-agnosticity pattern] to future-proof it for use with [nonexistent potential platforms] through [nonexistent publishing system] :sweat_smile:

view this post on Zulip Richard Feldman (Jan 07 2022 at 02:13):

I love the ambition! :smiley:

view this post on Zulip Brendan Hansknecht (Jan 07 2022 at 02:44):

The nice part is that even without a packaging system, it can probably be copied into any random platform and used.

view this post on Zulip Richard Feldman (Jan 07 2022 at 13:05):

to be fair, that's how you end up being the person who says "I wrote the random generation library everybody uses in Roc" :grinning_face_with_smiling_eyes:

view this post on Zulip Brian Carroll (Jan 07 2022 at 14:48):

@JanCVanB this is great!
Not sure if you're aware of this already but if you're trying to figure out what some unfamiliar Elm code is doing, then Debug.log is your friend. For example you could clone the repo and then start inserting _ = Debug.log "some_var" some_var into let blocks all over the place, then run the tests and see what comes out on the console.

view this post on Zulip jan kili (Jan 29 2022 at 13:57):

I (a newborn Rustling) am trying to create a Bevy https://docs.rs/bevy/latest/bevy/ "hello world" platform by adapting examples/hello-rust/platform. While I could post the wide and wild variety of errors I'm getting, what files would you expect me to need to change to get Bevy to compile (just a minimal hello popup)? I'm currently changing platform/sec/lib.rs and platform/Cargo.toml.

view this post on Zulip Folkert de Vries (Jan 29 2022 at 14:18):

that page you link has an example. In theory you could just have the platform not use any roc code at all, so changing the platform rust code to that example should do it? (swap main for rust_main)

view this post on Zulip jan kili (Jan 29 2022 at 14:29):

So no Cargo.toml changes necessary?

view this post on Zulip jan kili (Jan 29 2022 at 14:30):

Last night I tried injecting Bevy main into Rust main function and got so many angry messages haha

view this post on Zulip Folkert de Vries (Jan 29 2022 at 14:30):

oh, yes sure

view this post on Zulip Folkert de Vries (Jan 29 2022 at 14:30):

you'd need to add bevy to the dependencies section of the Cargo.toml

view this post on Zulip Folkert de Vries (Jan 29 2022 at 14:31):

typically you can find what to add on crates.io: https://crates.io/crates/bevy

view this post on Zulip Folkert de Vries (Jan 29 2022 at 14:31):

so bevy = "0.6.0" in this case

view this post on Zulip jan kili (Jan 29 2022 at 14:35):

Great - sounds like I'm on the right track :)

view this post on Zulip jan kili (Jan 29 2022 at 14:37):

Bevy recommends "dynamic" in its dependency definition, but that causes errors about bevy missing "rlib"

view this post on Zulip jan kili (Jan 29 2022 at 14:38):

Last I saw, without bevy dynamic, it was complaining about hundreds of missing files that started with underscores - can post details later

view this post on Zulip jan kili (Jan 29 2022 at 14:38):

Thanks!

view this post on Zulip Folkert de Vries (Jan 29 2022 at 14:40):

hmm you could try getting bevy to run just outside of a roc context

view this post on Zulip Folkert de Vries (Jan 29 2022 at 14:40):

it's hard to know now if the problem is with roc or with rust or bevy

view this post on Zulip jan kili (Jan 29 2022 at 15:23):

I got their hello world working normally, but when I integrated with Roc it was hard to say what failed

view this post on Zulip jan kili (Jan 30 2022 at 02:25):

Here's where I'm stuck, if anyone can help :smile: https://github.com/rtfeldman/roc/commit/88c94bf3031feb7f9bad4008cd403bd0b2f42db5
(This is not intended as a WIP contribution, it's just pushed to the repo for convenient diff-ing)

view this post on Zulip jan kili (Jan 30 2022 at 13:23):

Well, this seems solidly beyond my Rust abilities now:
ld: symbol(s) not found for architecture x86_64
error: linking with cc failed: exit status: 1
= note: clang-7: error: invalid version number in '-mmacosx-version-min=12.0'
I seem to be experiencing https://github.com/rust-lang/rust/issues/91372 via https://github.com/rust-lang/rust/issues/91781, but the fixes they mention don't seem to work for me. :shrug:

view this post on Zulip Folkert de Vries (Jan 30 2022 at 13:25):

@Richard Feldman could this be because we have some C code in the rust platforms? I always forget why that was needed, but maybe information gets lost in the process?

view this post on Zulip Richard Feldman (Jan 30 2022 at 13:36):

it's needed until we can switch over to surgical linking for everything

view this post on Zulip Richard Feldman (Jan 30 2022 at 13:37):

https://users.rust-lang.org/t/error-when-compiling-linking-with-o-files/49635/5?u=rtfeldman

view this post on Zulip Richard Feldman (Jan 30 2022 at 13:39):

@JanCVanB is there a commit somewhere I can check out so I can try it on my machine?

view this post on Zulip jan kili (Jan 30 2022 at 13:39):

Yes! https://github.com/rtfeldman/roc/commit/88c94bf3031feb7f9bad4008cd403bd0b2f42db5

view this post on Zulip jan kili (Jan 30 2022 at 13:40):

Thank you both :)

view this post on Zulip Richard Feldman (Jan 30 2022 at 15:51):

hm, I can't build bevy at all on my M1 (even outside Roc) because I'm running into https://github.com/bevyengine/bevy/issues/928#issuecomment-943882168 :sweat_smile:

view this post on Zulip Richard Feldman (Jan 30 2022 at 15:51):

@JanCVanB are you on an M1 or x64 mac?

view this post on Zulip jan kili (Jan 30 2022 at 16:14):

x64

view this post on Zulip jan kili (Jan 30 2022 at 16:14):

Their helloworld & breakout examples build for me

view this post on Zulip Ayaz Hafiz (Jan 30 2022 at 17:05):

I tried this with the latest nightly of rust, which is supposed to have the fix for #91372, and that didn't work either

view this post on Zulip Richard Feldman (Jan 30 2022 at 22:59):

so I guess a reasonable question is: how important is it that it be bevy specifically? :sweat_smile:

view this post on Zulip Richard Feldman (Jan 30 2022 at 23:00):

I have an x64 mac I can try it on tomorrow

view this post on Zulip jan kili (Jan 30 2022 at 23:08):

Haha, to me? Very. To the community? Probably not.

view this post on Zulip jan kili (Jan 30 2022 at 23:09):

I haven't tried the fix yet, but might get some coding in tonight to try it :) thanks!

view this post on Zulip Richard Feldman (Jan 30 2022 at 23:12):

shot in the dark: clang-7: error: invalid version number in '-mmacosx-version-min=12.0' - when you run clang --version what does it say?

view this post on Zulip Richard Feldman (Jan 30 2022 at 23:13):

oh also what version of macOS are you on?

view this post on Zulip jan kili (Jan 30 2022 at 23:14):

That's a good question - I'm afk right now, but that was my error when running with some recommended env var like MACOSX_DEPLOYMENT_TARGET=12.0 && cargo run examples/hello-rust/Hello.roc

view this post on Zulip jan kili (Jan 30 2022 at 23:14):

I'm on 12.0.1

view this post on Zulip jan kili (Jan 31 2022 at 04:15):

[nix-shell:~/code/clones/roc]$ clang --version
clang version 7.1.0 (tags/RELEASE_710/final)
Target: x86_64-apple-darwin
Thread model: posix
InstalledDir: /nix/store/zfh3npfhfjjgwi0dqpriklip5k15ppmj-clang-7.1.0/bin

view this post on Zulip jan kili (Jan 31 2022 at 04:39):

I'll take this debugging into https://github.com/rtfeldman/roc/issues/2432

view this post on Zulip jan kili (Jan 31 2022 at 05:09):

Unrelated to this debugging, Bevy's API looks super-resistant to first-level Roc-ification:

view this post on Zulip jan kili (Jan 31 2022 at 05:12):

However, second-level Roc-ification seems promising:

view this post on Zulip jan kili (Jan 31 2022 at 05:17):

Example for a 2D platformer (like Super Mario): The platform says that there must be a Player that jumps, but the Roc app must specify when the jump happens via keymapping, how the jump looks via animation curve, when the jump is allowed via state machine rules, and how the jump is interrupted via collision rules. :smiley:

view this post on Zulip jan kili (Feb 01 2022 at 06:11):

Can a platform requires multiple things?
I can make a one-requires platform work, even if the def isn't named main.
Now I'm trying to get this two-requires platform to work:

platform "breakout"
    requires {} { ballSpeed : Dec, paddleSpeed : Dec }
    exposes []
    packages {}
    imports []
    provides [ ballSpeedForHost, paddleSpeedForHost ]
    effects fx.Effect {}

ballSpeedForHost : Dec
ballSpeedForHost = ballSpeed

paddleSpeedForHost : Dec
paddleSpeedForHost = paddleSpeed

but I get this error:

── MISSING REQUIRES ────────────────────────────────────────────────────────────

I am partway through parsing a header, but I got stuck here:

1  platform "breakout"
2      requires {} { ballSpeed : Str, paddleSpeed : Str }
                                     ^

I am expecting the requires keyword next, like

    requires { main : Task I64 Str }

view this post on Zulip Folkert de Vries (Feb 01 2022 at 08:22):

not currently, a quick fix is to accept a record with the fields that you want

view this post on Zulip jan kili (Feb 01 2022 at 08:24):

That would be great, though since I didn't see a RocRecord in roc_std, I didn't know how to proceed with that approach.

view this post on Zulip Folkert de Vries (Feb 01 2022 at 08:25):

you can expose multiple things from the package config

view this post on Zulip Folkert de Vries (Feb 01 2022 at 08:25):

hmm wait no

view this post on Zulip Folkert de Vries (Feb 01 2022 at 08:25):

well kinda

view this post on Zulip Folkert de Vries (Feb 01 2022 at 08:27):

you can expose a record, and that turns into a rust struct in a predictable way

view this post on Zulip Folkert de Vries (Feb 01 2022 at 08:28):

on the rust side you need to make sure the struct sorts its fields first by alignment, then by name of the field, and it's annotated with #[repr(C)]

view this post on Zulip jan kili (Feb 07 2022 at 04:28):

Folkert de Vries said:

on the rust side you need to make sure the struct sorts its fields first by alignment, then by name of the field, and it's annotated with #[repr(C)]

I just tried this, and it works!

view this post on Zulip jan kili (Feb 07 2022 at 04:28):

I never expected to dive this much into platform development, but it's smooth!

view this post on Zulip jan kili (Feb 07 2022 at 06:04):

hello_world.png

view this post on Zulip jan kili (Feb 07 2022 at 06:43):

@Folkert de Vries are nested Records expose-able?

view this post on Zulip jan kili (Feb 07 2022 at 06:43):

I'm getting this: emitted runtime error function "The 8.IdentId(0) function could not be generated, likely due to a type error."
(though it may be from some unrelated mistake)

view this post on Zulip jan kili (Feb 07 2022 at 07:59):

Also, any tips on passing a record field like List F32 to Rust? Or should I just experiment?

view this post on Zulip Folkert de Vries (Feb 07 2022 at 16:24):

it should just work

view this post on Zulip Folkert de Vries (Feb 07 2022 at 16:24):

and nested records also should just work

view this post on Zulip Folkert de Vries (Feb 07 2022 at 16:25):

only tricky thing is the order of record fields on the C/Rust/... side

view this post on Zulip jan kili (Feb 07 2022 at 18:01):

How would you write/order the matching Rust struct for this Record?

a : {
    b : Str,
    c : U8,
    d : U32,
    e : List U8,
    f : List U32,
    g : {
        h : U32,
        i : I32,
        j : List U32,
        k : List I32},
    x : List { y : Str, z : Str } }

view this post on Zulip Folkert de Vries (Feb 07 2022 at 18:36):

Allright, adding an alias for clarity

T : 
    {
        h : U32,
        i : I32,
        j : List U32,
        k : List I32,
    }

U : 
    {
        b : Str,
        c : U8,
        d : U32,
        e : List U8,
        f : List U32,
        g : T,
        x : List { y : Str, z : Str } 
    }

We can start with T: the rule is sort first by alignment, then by field name.
(for this you need to know some stuff about alignment, I can elaborate if you're not sure how that works)

For this record, we're done immediately (lists store a pointer and a length (as a nat/usize) and hence have the alignment of a pointer which is equal to the alignment of a usize).

{
    h : U32,
    i : I32,
    j : List U32,
    k : List I32,
}

Compound structures have the alignment equal to the maximum of the alignment of their fields. That means we can sort the fields by alignmenthere:

{
    c : U8,
    d : U32,
    b : Str,
    e : List U8,
    f : List U32,
    g : T,
    x : List { y : Str, z : Str } 
}

because of the names you picked this is also already sorted by field name

view this post on Zulip jan kili (Feb 07 2022 at 19:34):

I see, so small-to-large alignment order goes like this?

I8 == U8
I16 == U16
I32 == U32
I64 == U64 == Str == List * == Record *
I128 == U128

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:36):

Record * is not correct there. Record fields are stored on the stack (while list elements are stored on the heap)

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:36):

that means that what types a record's fields have influence the record's alignment

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:36):

so a record { x : u8 } has an alignment of 1, but a record { x : U128 } has an alignment of 16

view this post on Zulip jan kili (Feb 07 2022 at 19:37):

Side question: Are Nats weird? Would V : { r: U32, s: U64, t: Nat } require different struct definitions for different systems?

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:38):

yes, I was thinking about that too when writing this out

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:38):

in particular this is kinda weird if you mix targets, e.g. a rust project with a wasm frontend and normal x86 backend

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:39):

then you need to order the fields depending on your target

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:39):

which you can do, but it's not awesome

view this post on Zulip jan kili (Feb 07 2022 at 19:39):

Maybe Nats shouldn't pass between platform and app

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:41):

it's inevitable

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:41):

e.g. a List * also has a different alignment

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:41):

because it contains nats and pointers

view this post on Zulip jan kili (Feb 07 2022 at 19:49):

Is this alignment-based sorting algorithm...
a) an unbending constraint?
b) a performance optimization?
c) a placeholder?
Just curious whether pure-alphabetical sorting is possible/optimal.

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:49):

it's a performance optimization

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:49):

when we don't sort by alignment, we can get gaps between fields

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:50):

e.g. { x : u32, y: i128 }

view this post on Zulip jan kili (Feb 07 2022 at 19:50):

Maybe this sorting would benefit from a compiler flag? --alphabetical-structs

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:50):

an i128 has an alignment of 16, meaning that its start address needs to be a multiple of 16

view this post on Zulip jan kili (Feb 07 2022 at 19:50):

If a platform developer wants to pass strange values or be flexible somehow

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:51):

assuming we start at 0, we put in the u32 field , taking up 4 bytes. But now we need to add 12 bytes of padding before we can put in the i128field

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:51):

the opposite order is not beneficial in this example, but in general sorting by alignment minimizes the amount of padding you need

view this post on Zulip jan kili (Feb 07 2022 at 19:51):

Ah, that makes sense

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:53):

e.g. C says: definition order is what we use, if that means we need to insert padding to get the alignment right then that is what we do

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:53):

rust says: don't worry about it, we will lay it out in an optimal way

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:53):

except if you use #[repr(C)]

view this post on Zulip jan kili (Feb 07 2022 at 19:53):

With an algorithm toggle, platform developers would be responsible for optimizing performance via field name alphabetization, which might be a nice tradeoff

view this post on Zulip jan kili (Feb 07 2022 at 19:54):

(basically to let developers use that C mode you just described)

view this post on Zulip Folkert de Vries (Feb 07 2022 at 19:54):

well the order of record fields is not well-defined

view this post on Zulip Richard Feldman (Feb 07 2022 at 20:07):

I think the best long-term solution is to have a compiler command to generate the entire definition, including ordering - but in the short term, maybe we could make a quick compiler flag that prints out the sorted versions of all type aliases in the program?

view this post on Zulip Richard Feldman (Feb 07 2022 at 20:08):

like it just takes the AST of the type alias, sorts everything, and then runs the formatter on it to get a string

view this post on Zulip Richard Feldman (Feb 07 2022 at 20:08):

and then prints the string

view this post on Zulip Richard Feldman (Feb 07 2022 at 20:08):

so at least platform authors don't have to figure all that out in their heads until we have that long-term solution

view this post on Zulip jan kili (Feb 07 2022 at 20:11):

I'd love that. To be clear, I don't have an immediate need for such complex records, but it's right around the corner (especially since only one def can be passed from app to platform right now, I think)

view this post on Zulip Richard Feldman (Feb 07 2022 at 20:20):

yeah and tag unions are even harder than records :sweat_smile:

view this post on Zulip jan kili (Feb 09 2022 at 05:51):

@Folkert de Vries I successfully implemented a Roc-to-Rust record/struct with 3 Strs and 2 U32s in the (WIP) plotting platform, but now I'm having trouble simply adding a second F32 to the game platform.
https://github.com/JanCVanB/roc-bevies/pull/3/commits/768ad5d
Do you know why only the first f32 is being heard? Depending on which field comes first in the struct, the debug print looks like
Config { ballSpeed: 400.0, paddleSpeed: 0.0 }
or
Config { paddleSpeed: 400.0, ballSpeed: 0.0 }.
:confused:

view this post on Zulip jan kili (Feb 09 2022 at 05:52):

I'm probably just not seeing a typo or something, but at this point I can't see it.

view this post on Zulip jan kili (Feb 09 2022 at 05:57):

The behavior is the same if I change them both to i16s or u16s.

view this post on Zulip jan kili (Feb 09 2022 at 06:13):

Whoa, when I add another F32 between them, it hears the 1st as the 1st and the 2nd as the 3rd!
Roc source: config = { ballSpeed: 111, manualPadding: 222, paddleSpeed: 333 }
Rust debug: Config { ballSpeed: 111.0, manualPadding: 0.0, paddleSpeed: 222.0 }
This must be related to alignment, but I don't know how. Maybe... Roc is writing the record with 8-byte alignment (because it's represented with an 8-byte pointer) but then Rust isn't recognizing the padding between the 4-byte F32s?
https://github.com/JanCVanB/roc-bevies/pull/3/commits/8eed55c

view this post on Zulip Folkert de Vries (Feb 09 2022 at 12:05):

I'm seeing no problems when using

fn roc_config() -> Config {
    extern "C" {
        #[link_name = "roc__configForHost_1_exposed_generic"]
        fn roc_config_internal(_: &mut Config);
    }

    let mut config = Config::default();
    unsafe { roc_config_internal(&mut config) };

    config
}

#[repr(C)]
#[derive(Default, Debug)]
struct Config {
    pub ballSpeed: f32,
    pub paddleSpeed: f32,
}

view this post on Zulip Folkert de Vries (Feb 09 2022 at 12:06):

I'm using the _generic version of configForHost here, which takes a pointer that it writes the value into

view this post on Zulip Folkert de Vries (Feb 09 2022 at 12:07):

that is the safer way of doing this, because simply returning values (like -> Config) seems to break in some cases across the C FFI boundary

view this post on Zulip jan kili (Feb 09 2022 at 12:26):

Well that's nice that the extern and unsafe can both be absorbed into the roc_config function - that's how I'll do every Roc call from now on.

view this post on Zulip jan kili (Feb 09 2022 at 12:27):

This Rustling never would have found that _generic solution, thank you very much

view this post on Zulip Folkert de Vries (Feb 09 2022 at 12:27):

yeah I wonder if we should actually remove the non-generic version

view this post on Zulip Folkert de Vries (Feb 09 2022 at 12:28):

since it causes a bunch of problems

view this post on Zulip Folkert de Vries (Feb 09 2022 at 12:28):

on the other hand, it's not entirely clear why it causes problems

view this post on Zulip jan kili (Feb 10 2022 at 13:52):

Okay, here's a new request... is passing a function as a record field possible?

Since only one def can be passed from an app to a platform, I see only two ways of passing both static primitive data and dynamic function logic:

  1. pass a record containing both primitive fields and function fields
  2. pass a function that either returns primitive data or performs dynamic logic, depending on the input

I've tried the following, but the Rust compiler is complaining about methods & traits, so I'm out of my Rust depth :sweat_smile:

use std::boxed::Box;
use std::ops::Fn;

#[derive(Default)]
#[repr(C)]
pub struct Config {
    pub ballSpeed: f32,
    pub paddleSpeed: f32,
    pub todo: Box<dyn Fn(i16) -> i16>,
}

view this post on Zulip Folkert de Vries (Feb 10 2022 at 13:53):

the tui example exposes a record with functions and values

view this post on Zulip Folkert de Vries (Feb 10 2022 at 13:54):

the trick here is that the field in the record only contains the captured environment of the function (usually that is a unit value, because the exposed function is top-level)

view this post on Zulip Folkert de Vries (Feb 10 2022 at 13:54):

we then expose an additional extern function to actually call the function

view this post on Zulip jan kili (Feb 10 2022 at 13:56):

tui is in Zig - does the same system apply to Rust?

view this post on Zulip Folkert de Vries (Feb 10 2022 at 13:56):

yes both languages use the C api that we expose

view this post on Zulip jan kili (Feb 10 2022 at 14:08):

Phew, complex platform APIs hurt my brain :sweat_smile:
extern fn roc__mainForHost_1_Update_caller(ConstModel, *const RocStr, [*]u8, MutModel) void;

view this post on Zulip Folkert de Vries (Feb 10 2022 at 14:11):

that's the downside. Hopefully you get that right once and can then build a nicer api on the rust/zig side

view this post on Zulip Richard Feldman (Feb 10 2022 at 14:13):

also platform author ergonomics might be most underdeveloped part of Roc tooling right now :big_smile:

view this post on Zulip Richard Feldman (Feb 10 2022 at 14:13):

we have no way to help generate these things yet

view this post on Zulip Richard Feldman (Feb 10 2022 at 14:13):

but we can in the future!

view this post on Zulip jan kili (Feb 10 2022 at 14:24):

Unrelated, I find this file hilarious:

platform "roc-plotters/bitmap-chart"
    requires {} { config : Config }
    exposes [ Config ]
    packages {}
    imports [ Config.{ Config } ]
    provides [ configForHost ]

configForHost : Config
configForHost = config

view this post on Zulip jan kili (Feb 10 2022 at 14:24):

The cherry on top is that the file name is Package-Config.roc :laughter_tears:

view this post on Zulip jan kili (Feb 10 2022 at 14:26):

config x 10 == 10x engineering

view this post on Zulip Folkert de Vries (Feb 10 2022 at 14:27):

this reminds me of the greatest talk of all time https://www.youtube.com/watch?v=yL_-1d9OSdk

view this post on Zulip Folkert de Vries (Feb 10 2022 at 14:27):

config config, config config config. Config

view this post on Zulip jan kili (Feb 10 2022 at 16:23):

Richard Feldman said:

yeah and tag unions are even harder than records :sweat_smile:

Is this a solved problem? I just ran into a use case for a tag union :stuck_out_tongue:

view this post on Zulip Folkert de Vries (Feb 10 2022 at 16:25):

what kind of tag union?

view this post on Zulip Folkert de Vries (Feb 10 2022 at 16:25):

we have support for RocResult, and an enum is actually easy

view this post on Zulip jan kili (Feb 10 2022 at 16:33):

I suppose it's an enum: format : [ Bitmap, Svg ]

view this post on Zulip jan kili (Feb 10 2022 at 16:34):

Is there an enum example in examples/?

view this post on Zulip Folkert de Vries (Feb 10 2022 at 16:48):

no but you can treat this as if you're sending a u8 over

view this post on Zulip Folkert de Vries (Feb 10 2022 at 16:49):

this might also work

#[repr(u8)]
enum {
    Bitmap = 0,
    Svg = 1,
}

view this post on Zulip Folkert de Vries (Feb 10 2022 at 16:49):

not sure if that plays nicely with extern functions

view this post on Zulip jan kili (Feb 10 2022 at 20:12):

How about lists like List I64 or List List { label: Str, values: List I64 }?

I see a List I64 passing between Zig & Roc in examples/quicksort/, but I'm hoping to pass complex lists from Roc to Rust (without knowing their length at compile-time, which requires Vecs?). Will that require any tricks that (a) aren't demonstrated in examples/quicksort/ or (b) look different in Rust than in Zig?

view this post on Zulip jan kili (Feb 10 2022 at 20:15):

Thanks for all of this help on these API patterns :) I hope to produce some useful examples of these patterns, so that future devs can simply copy/paste the goods!

view this post on Zulip jan kili (Feb 10 2022 at 20:17):

(Side note, I just fought with Rust generics for 2 hours and won!!! I feel like I just received my second Rustling badge, haha)

view this post on Zulip jan kili (Feb 10 2022 at 20:18):

(perhaps I should have read a Rust guide/tutorial/book, but instead I'm just guessing the syntax until the compiler finally gives in! :sweat_smile:)

view this post on Zulip Folkert de Vries (Feb 10 2022 at 20:35):

you can use RocList from roc_std

view this post on Zulip Folkert de Vries (Feb 10 2022 at 20:35):

although you have to carefully think about who owns the memory

view this post on Zulip jan kili (Feb 10 2022 at 23:52):

@Philippe Hosey-Vinchon Do you want to help me make some fun Roc platforms? :) I'm pretty comfortable in Roc after a couple of months with it, but Rust is brand new for me! After fighting with Rust generics today, I can see that platform-ifying existing Rust libraries can have surprise traps, some of which are best defused by a more-experience Rustacean than myself :crab:

view this post on Zulip jan kili (Feb 10 2022 at 23:59):

:loudspeaker: In fact, anyone is welcome to collaborate! Help me Roc-ify some useful+popular Rust libraries: :smiley:
Plots/charts/graphics: https://github.com/JanCVanB/roc-plotters
Games/simulations/UIs: https://github.com/JanCVanB/roc-bevies

view this post on Zulip Andrew Thompson (Feb 11 2022 at 00:05):

Hi, new here. I read through a bunch of platform posts here, but after looking at the examples, I don't think I understand platforms at all. Should I, some time in the future, decide to write a simple cli tool in Roc, would there be a standard platform for this purpose? Would it only support nix or windows at one time? I'm confused why there are rust examples and zig examples, and why anyone writing an app would care what language the platform is written in?

view this post on Zulip jan kili (Feb 11 2022 at 00:09):

Hi, Andrew! I'd recommend using examples/cli/, as it's the only platform thusfar that supports general-purpose CLI apps.

view this post on Zulip jan kili (Feb 11 2022 at 00:12):

In addition to the example CLI apps in that dir (form.roc, countdown.roc, echo.roc), I have some more (mostly-redundant) examples in this abandoned repo.

view this post on Zulip jan kili (Feb 11 2022 at 00:14):

Regarding Nix/Windows, what operating system are you on? I'm using macOS, and Nix is super convenient for hopping into a Roc-friendly environment.

view this post on Zulip Andrew Thompson (Feb 11 2022 at 00:23):

Sorry, I meant *nix, as any random linux. I run Windows, but built Roc in WSL Ubuntu last night, and will be playing with it there.
The platforms that are in the examples seem(and this is fine for this purpose) custom built out with just enough functionality to support the example they work for. This is part of my confusion. Would you expect there to be a batteries included platform that would meet most needs of a particular type of app?

view this post on Zulip jan kili (Feb 11 2022 at 00:24):

I recommend this thread: https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/A.20platform.20for.20production.20CLIs
(TL;DR: Not yet, but it's an inevitability that can be accelerated by demand & contributions)

view this post on Zulip jan kili (Feb 11 2022 at 00:33):

Regarding Rust/Zig, Roc app developers shouldn't care what language their platform is written in.

I think about Roc platforms like this:

I hope that makes platforms make more sense, but now that I've written it all out I realize this could just confuse the issue :sweat_smile:

view this post on Zulip Andrew Thompson (Feb 11 2022 at 01:03):

JanCVanB said:

These still feels like you would be in a place where a Roc app developer could need to also write their underlying platform(or copy/paste from a previous similar app and add new lib Y). Am I misunderstanding?

view this post on Zulip Richard Feldman (Feb 11 2022 at 01:06):

in the future platforms will be available in the package registry (which doesn't exist yet!) so you can just say what platform you want and the compiler download and install it behind the scenes, just like any other package

view this post on Zulip Richard Feldman (Feb 11 2022 at 01:07):

but since we don't have a package registry yet, for now it's all copying and pasting onto the local filesystem :big_smile:

view this post on Zulip jan kili (Feb 11 2022 at 01:08):

add new lib Y). Am I misunderstanding?

No, you're correct. If an app wants to do something that its platform doesn't support, then it will need to fork+modify the platform or find another platform. That's by design, so that app consumers/end-users can be absolutely certain what any app might do when they run it. For example, if an app's platform doesn't have networking or file-writing functionalities, you know it's (relatively) safe to run.

view this post on Zulip Richard Feldman (Feb 11 2022 at 01:08):

but the idea is for obtaining a platform to feel equivalent to obtaining a framework in another language: you just list it as a package dependency and it gets installed for you

view this post on Zulip Richard Feldman (Feb 11 2022 at 01:10):

of note, it's theoretically possible for a platform author to provide a C FFI (via normal function calls, just asking for the name of a dynamic library along with binary encoders and decoders for its arguments and return value) but of course any platform that does that can no longer offer any security guarantees, so it's a strictly opt-in thing :big_smile:

view this post on Zulip Richard Feldman (Feb 11 2022 at 01:11):

(that's not a special language feature or anything, it's just an observation that any platform author can in fact do that if they want to and it will work)

view this post on Zulip Andrew Thompson (Feb 11 2022 at 01:12):

I know this isn't strictly a platform question, but I think it's related to @Richard Feldman 's comment: If I wanted to use that handy dandy Random lib, would I always expect to get just the source(and drop it right in my app folder?), or is a Roc compiled binary of some kind possible/intended?

view this post on Zulip jan kili (Feb 11 2022 at 01:14):

For now, copy/paste it or use a Git submodule. In the indeterminate future, something like npm or pypi for Roc will exist.

view this post on Zulip jan kili (Feb 11 2022 at 01:16):

However, I don't believe Roc libraries will ever be pre-compiled, as compilation only happens in conjunction with a specific platform. Roc libraries must perform a delicate dance of platform-agnosticity, so that any Roc app (on any Roc platform) can use them.

view this post on Zulip Brendan Hansknecht (Feb 11 2022 at 03:44):

@JanCVanB if you are interested, I would be up for helping you with some of the platform stuff and rust type fun. I should theoretically have an ok understanding of the underlying types and rust.

view this post on Zulip Tom Dohrmann (Feb 16 2022 at 12:35):

Are there any stability guarantees regarding roc_std?

view this post on Zulip Folkert de Vries (Feb 16 2022 at 12:37):

not really, if we think a change is necessary we make it and then update all code in the compiler repo

view this post on Zulip Folkert de Vries (Feb 16 2022 at 12:41):

so far RocList is not used that much, RocStr a bit more but even there it's just more convenient to quickly turn that into a String/&str or whatever the host language wants

view this post on Zulip Richard Feldman (Feb 16 2022 at 12:51):

there will be eventually, but a major part of the reason the repo is private now is to set expectations around the fact that Roc isn't ready for production use yet, so we may make breaking changes on short notice! :big_smile:

view this post on Zulip Tom Dohrmann (Feb 16 2022 at 18:16):

When passing a List from a platform to a Roc application, does the List always have to be reference counted? compiler/str/README.md mentions that the platform can't assume anything about Lists provided by Roc application, but AFAICT doesn't say anything about Lists passed the other way around.
I'm asking because valgrind is reporting that I'm leaking memory which I tracked down to being caused by returning a unique List with a capacity to a Roc application.

view this post on Zulip Brendan Hansknecht (Feb 16 2022 at 18:19):

All list are reference counted in general. If Roc returns a list and it has only 1 reference count left (which is the normal case), it is the host's job to free the memory.

view this post on Zulip Brendan Hansknecht (Feb 16 2022 at 18:19):

If it had a higher than 1 reference count, it would be expected that the host lowers the reference count by one when dropping the list.

view this post on Zulip Folkert de Vries (Feb 16 2022 at 18:20):

that misses capacity. But the capacity as it's currently used in roc_std is an older idea that we'll likely not use any more

view this post on Zulip Folkert de Vries (Feb 16 2022 at 18:20):

currently, RC grows from isize::MIN upwards. At 0, the RC reaches "infinity" and an object with that refcount will live forever

view this post on Zulip Folkert de Vries (Feb 16 2022 at 18:21):

a positive value in that location used to indicate a capacity. But we now want to store the capacity on the stack

view this post on Zulip Folkert de Vries (Feb 16 2022 at 18:21):

so in effect that means that a positive value in the refcount position is invalid

view this post on Zulip Tom Dohrmann (Feb 16 2022 at 18:29):

Folkert de Vries said:

that misses capacity. But the capacity as it's currently used in roc_std is an older idea that we'll likely not use any more

Oh ok, good to know.

view this post on Zulip Richard Feldman (Feb 16 2022 at 19:37):

but we plan to transition that to basically exactly what Vec does in rust

view this post on Zulip Richard Feldman (Feb 16 2022 at 19:37):

with capacity stored in the struct, not behind the pointer

view this post on Zulip Jorge Acereda (Feb 18 2022 at 09:54):

Hi all, I'm trying to build Roc (update_zig_09 branch) on Guix but I'm facing unresolved LLVM symbols when linking roc_cli. LLVM_SYS_130_PREFIX is set, and llvm-config is in PATH. I have zero knowledge of Rust, any hint?

view this post on Zulip Folkert de Vries (Feb 18 2022 at 09:56):

that branch doesn't quite work for most things related to lists/strings.

view this post on Zulip Folkert de Vries (Feb 18 2022 at 09:56):

if you still want to try it: what errors are you getting exactly?

view this post on Zulip Jorge Acereda (Feb 18 2022 at 10:00):

err.txt

view this post on Zulip Jorge Acereda (Feb 18 2022 at 10:00):

Looks like every LLVM symbol is undefined.

view this post on Zulip Jorge Acereda (Feb 18 2022 at 10:01):

I went with the 09 branch because that's the zig version that happens to be packaged on Guix, but I guess I can downgrade.

view this post on Zulip Folkert de Vries (Feb 18 2022 at 10:09):

you can set the zig version with ROC_ZIG=path/to/zig

view this post on Zulip Folkert de Vries (Feb 18 2022 at 10:09):

if you don't want to touch the system version

view this post on Zulip Jorge Acereda (Feb 18 2022 at 10:14):

So, should I give up the 09 branch and try with master? Will that get rid of the llvm unresolveds?

view this post on Zulip Folkert de Vries (Feb 18 2022 at 10:17):

I think trying trunk is the best approach for now. Even if you get this branch to compile, it won't really give you much

view this post on Zulip Folkert de Vries (Feb 18 2022 at 10:17):

because most examples will just fail to compile

view this post on Zulip Folkert de Vries (Feb 18 2022 at 10:17):

note that trunk uses llvm12

view this post on Zulip Jorge Acereda (Feb 18 2022 at 10:18):

Ok, I'll try that over the weekend, thanks.

view this post on Zulip Folkert de Vries (Feb 18 2022 at 10:19):

btw it does look like llvm cannot be found by the linker. it looks like gcc is used?!

view this post on Zulip Folkert de Vries (Feb 18 2022 at 10:19):

so, using trunk may not solve that problem

view this post on Zulip Jorge Acereda (Feb 18 2022 at 10:51):

Ups, good catch. Looks like I depend on clang instead of clang-toolchain in the package definition, that might well be the problem.

view this post on Zulip Andrew Thompson (Feb 18 2022 at 16:32):

I compiled Roc in Ubuntu on Windows(WSL). Compiles seem to take a really long time. Is this expected? image.png

view this post on Zulip Folkert de Vries (Feb 18 2022 at 16:35):

WSL is very slow if the code is stored on the windows side, but compiled from the linux side

view this post on Zulip Folkert de Vries (Feb 18 2022 at 16:36):

@Brian Carroll knows more about how to fix this

view this post on Zulip Andrew Thompson (Feb 18 2022 at 16:40):

I saw something related to that, and copied all the code into the linux side, but it didn't make any appreciable difference. I'll try compiling it in native windows this weekend. (I only compiled it from linux because I thought it'd be a smoother process. Hint: it was bumpy)

view this post on Zulip Folkert de Vries (Feb 18 2022 at 16:41):

we don't currently support windows outputs, so that means that if you compile a program with the compiler, you need to then run it in WSL

view this post on Zulip Folkert de Vries (Feb 18 2022 at 16:41):

also, nobody has actually tried compiling on windows I think. I guess it should work?

view this post on Zulip Brendan Hansknecht (Feb 18 2022 at 16:58):

I remember trying and running into a lot of issues, but that was quite a while ago on a mini pc without a lot of storage. I don't remember why (maybe for llvm-as, maybe for something else), but I needed to compile llvm from source and it ate loads and loads of disk space.

view this post on Zulip Jose Quesada (Feb 18 2022 at 18:11):

I have a quick question, when specifying the path to a platform, does the path have to be absolute?
I'm asking because relative paths don't seem to work, but not sure.

app "Hello"
    packages { pf: "../bin/examples/cli/platform" }
    imports [ pf.Stdout ]
    provides [ main ] to pf

view this post on Zulip Jose Quesada (Feb 18 2022 at 18:13):

nevermind, I was being silly lol

view this post on Zulip Brian Carroll (Feb 18 2022 at 18:20):

Folkert de Vries said:

Brian Carroll knows more about how to fix this

Just to add a bit more detail, the exact issue I had was that my clone of the repo, and therefore the Cargo target folder, was on the Windows side in the NTFS filesystem. But I was running cargo test and cargo build in the Linux terminal, so Cargo was also using the ~/.cargo folder from the Linux ext4 filesystem. So it must have been talking to both filesystems, maybe copying stuff between them, probably going through some sort of translation layer...

The fix that worked for me was to move everything to the Linux filesystem. There were no other steps to it, just that.

But the commands that Andrew ran in his screenshot don't look like they would have the same issue since all the files are on the Linux side, and he's running Roc rather than Cargo.

view this post on Zulip Brian Carroll (Feb 18 2022 at 18:25):

But I'm not sure what else to suggest since the only WSL-specific issue we know about is this one

view this post on Zulip Brian Carroll (Feb 18 2022 at 18:26):

I'll try running the same command on my machine and see what happens.

view this post on Zulip Brian Carroll (Feb 18 2022 at 18:31):

brian@acer:~/code/roc$ time target/debug/roc build examples/cli/form.roc
🔨 Rebuilding host... Done!
🎉 Built examples/cli/form in 2268 ms

real    0m2.314s
user    0m1.997s
sys     0m0.380s
brian@acer:~/code/roc$ time target/debug/roc build examples/cli/form.roc
🔨 Rebuilding host... Done!
🎉 Built examples/cli/form in 317 ms

real    0m0.327s
user    0m0.400s
sys     0m0.064s
brian@acer:~/code/roc$ time target/debug/roc build --precompiled-host examples/cli/form.roc
🎉 Built examples/cli/form in 201 ms

real    0m0.210s
user    0m0.299s
sys     0m0.031s

view this post on Zulip Brian Carroll (Feb 18 2022 at 18:32):

From the Windows "System > About" menu:

Device name acer
Processor   11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz   2.80 GHz
Installed RAM   8.00 GB (7.80 GB usable)
Device ID   8E0330F2-1463-4F8C-8E66-5A35508DA65E
Product ID  00325-96756-33883-AAOEM
System type 64-bit operating system, x64-based processor
Pen and touch   Touch support with 10 touch points

view this post on Zulip Aaron (Feb 18 2022 at 19:54):

Why does hello.roc link against all these libraries on MacOS ?
AudioUnit.framework, Cocoa.framework, CoreAudio.framework, CoreVideo.framework, IOKit.framework, Metal.framework, QuartzCore.framework, Security.framework

view this post on Zulip Folkert de Vries (Feb 18 2022 at 19:55):

I think those libraries are needed for some programs, so for now they are included in all programs

view this post on Zulip Aaron (Feb 18 2022 at 19:55):

Is that something the platform does?

view this post on Zulip jan kili (Feb 18 2022 at 20:02):

Yeah, today Roc platforms can't specify which OS libraries they need, but there's a plan for that. Until then, link ALL the things!

view this post on Zulip Aaron (Feb 18 2022 at 20:03):

i haven't finished the tutorial yet but is there an argc/argv interface in the common platforms ?

view this post on Zulip Richard Feldman (Feb 18 2022 at 21:05):

all of that linking will go away when we have surgical linking working on macOS - currently that only works on Linux, but the x86 macOS implementation is partially complete

view this post on Zulip Richard Feldman (Feb 18 2022 at 21:05):

the current linking solution on trunk has a bunch of problems, but it was a lot faster to get up and running than the long-term implementation we're working towards :big_smile:

view this post on Zulip Richard Feldman (Feb 18 2022 at 21:06):

so wheneever any platform needs to link something new, we just add it in there

view this post on Zulip Richard Feldman (Feb 18 2022 at 21:06):

eventually that whole .rs file will be unnecessary and get deleted

view this post on Zulip Ayaz Hafiz (Feb 18 2022 at 21:23):

Aaron said:

i haven't finished the tutorial yet but is there an argc/argv interface in the common platforms ?

I don't think any platform in the examples currently exposes this, though it shouldn't be too hard to do. This could be a candidate for a standard platform.

(Digression: I wonder if argc/argv should be more standardized in the language, or always handled in the host, because right now if the host exposes them they must be wrapped in an Effect - but (I think) they need not be effect-ful, since they're supplied by the kernel)

view this post on Zulip Aaron (Feb 18 2022 at 21:43):

Could this be a similar interface to the way Elm accepts flags on init?

view this post on Zulip Brian Carroll (Feb 18 2022 at 22:17):

Yeah so the classic argc and argv interface is just the way that you represent an array of strings in C. In Roc, the same information would be represented as List Str. So I think platforms would expose command line arguments to Roc as List Str. Once it's in Roc you'll generally want to parse the CLI arguments for your program into a nice Tag Union. That parsing library can be pure Roc code. There are similar libraries in Elm for URLs, for example.

view this post on Zulip Folkert de Vries (Feb 18 2022 at 22:19):

and we do allow passing in flags in e.g. the tui platform

view this post on Zulip Folkert de Vries (Feb 18 2022 at 22:19):

at least in theory

view this post on Zulip jan kili (Feb 18 2022 at 22:46):

Does tui have an example of that today or no?

view this post on Zulip Folkert de Vries (Feb 18 2022 at 22:47):

mainForHost : { init : ({} -> Model) as Init, update : (Model, Str -> Model) as Update, view : (Model -> Str) as View }
mainForHost = main

view this post on Zulip Folkert de Vries (Feb 18 2022 at 22:47):

so currently it takes {} as the flags value

view this post on Zulip Folkert de Vries (Feb 18 2022 at 22:48):

but that can be turned into something else

view this post on Zulip jan kili (Feb 18 2022 at 22:48):

Nice!

view this post on Zulip jan kili (Feb 19 2022 at 01:39):

I'm not sure if this is called tree shaking or lazy loading or what...
Is there any way for a platform to include/exclude imported libraries in/from the eventual machine code if the app does/doesn't use them? Example: A Rust-based platform imports two different stdout formatting crates, A and B. An app might do either Task.await (Stdout.lineA "hello") or Task.await (Stdout.lineB "hello"), and therefore the executable might only need crate A or crate B but not both. Is that a possible optimization, or does CFFI not grant us that level of compile-time insight?

view this post on Zulip jan kili (Feb 19 2022 at 01:41):

(For a more realistic example, I'm thinking that a video game platform might have a lot of heavy dependencies, but an individual video game might not need everything - physics simulation, programmatic map generation, etc.)

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:41):

do you mean things the host imports?

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:41):

or like Roc code

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:42):

maybe to put it another way: are we talking about machine code that was originally Rust code getting eliminated, or machine code that was originally Roc code getting eliminated?

view this post on Zulip jan kili (Feb 19 2022 at 01:42):

Rust

view this post on Zulip jan kili (Feb 19 2022 at 01:42):

My question might be: "Can we tree-shake Rust code that isn't used by Roc?"

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:42):

so I believe that's possible in the general case, although it's out of scope for Roc itself

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:42):

I remember reading about this

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:43):

it's something like there's a way to tell compilers (and I think Rust does this by default?) to emit each function into its own separate section of the binary

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 01:43):

The answer is definitely possible. Requires function sections and the linker doing GC

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:43):

which then lets you run a tool afterwards that says "if nothing ever calls into this section, eliminate the section"

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:43):

yeah exactly :point_up:

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:44):

so it depends on whether the host compiler supports emitting code that way

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:44):

@Brendan Hansknecht I don't think surgical linking should affect that, right?

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:44):

but yeah, regardless - it'd be something you would do without getting Roc involved at all

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:45):

there's just a tool you can run on the final binary that does it, assuming the binary was originally set up in a way that permits the tool do that

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 01:45):

The surgical linker wouldn't be able to clean up unused sections. They would already be baked into the binary.

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:45):

right, but it wouldn't affect those sections being there

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 01:45):

It also would probably cost way to much performance

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:45):

like you could still run the "DCE unused sections" tool separately later

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:45):

and it would still work, because the surgical linker didn't (for example) combine those sections or anything

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:46):

like it wouldn't touch them, so if they were set up that way by the host, they'd still be set up that way after surgical linking, and thus still eligible for DCE by a third-party tool

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 01:47):

Yeah, the section would just sit there and never get called into

view this post on Zulip jan kili (Feb 19 2022 at 01:48):

I see, and this is machine code? assembly? C?

view this post on Zulip jan kili (Feb 19 2022 at 01:49):

And the app developer would run a Dead Code Elimination tool on the output of the Roc CLI's compile/build?

view this post on Zulip jan kili (Feb 19 2022 at 01:50):

This will be more common if kitchen-sink-style (all-in-one) platforms are popular. Idk that they will be

view this post on Zulip jan kili (Feb 19 2022 at 01:51):

This will also be more common if app developers don't want to fork their platform, for whatever reasons

view this post on Zulip jan kili (Feb 19 2022 at 01:51):

I'm new to systems languages, so maybe this DCE thing is commonplace?

view this post on Zulip jan kili (Feb 19 2022 at 01:53):

Motivation for this question: I saw a cool looking Bevy (Rust video game engine) plugin in my GitHub feed and thought "add it to the pile! no downside! wait..."

view this post on Zulip Andrew Thompson (Feb 19 2022 at 01:54):

I wonder if you could have an empty shell host, where you'd opt into components somehow, and then codegen or let the host compile in just the chosen components.

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:55):

I see, and this is machine code? assembly? C?

machine code!

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:55):

And the app developer would run a Dead Code Elimination tool on the output of the Roc CLI's compile/build?

right, exactly - assuming they cared to :big_smile:

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:56):

I wonder if you could have an empty shell host, where you'd opt into components somehow, and then codegen or let the host compile in just the chosen components.

that's already if you do load them on startup (or later) with dynamic linking

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:57):

I hadn't thought about it, but it's conceivable we could let platforms ship with multiple binaries so they could load some of them on the fly if they wanted to

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:58):

I hope platform authors wouldn't use dynamic linking that searches the system path (unless absolutely necessary, e.g. it's needed on macOS for system libraries, and in practice glibc needs it on Linux too)

view this post on Zulip Richard Feldman (Feb 19 2022 at 01:58):

because inevitably that would mean that some people would go to run the platform and get an error on startup :confused:

view this post on Zulip Andrew Thompson (Feb 19 2022 at 01:59):

I wasn't thinking dynamic linking, more I'll include these definitions in the compiled platform if you ask for them. (And if you don't, your compile goes boom before you've ever delivered an app to anyone)

view this post on Zulip Richard Feldman (Feb 19 2022 at 02:01):

ah gotcha

view this post on Zulip Andrew Thompson (Feb 19 2022 at 02:01):

But now I'm thinking about configuring kernels with make menuconfig, and I'm getting feint.

view this post on Zulip Richard Feldman (Feb 19 2022 at 02:01):

:laughing:

view this post on Zulip Richard Feldman (Feb 19 2022 at 02:02):

since the DCE thing already works without adding anything to the language, I'd say we should start with that and see if there's a compelling use case for more than that in practice!

view this post on Zulip jan kili (Feb 19 2022 at 02:03):

@Andrew Thompson Am I following? :fear:

app "fear_the_night"
    packages { pf: "the-changeling" }
    imports []
    provides [ 666 ] to pf

666 = { realPlatform: "./roc/examples/cli/platform" }

view this post on Zulip Andrew Thompson (Feb 19 2022 at 02:04):

@JanCVanB sorry, I'm too new to follow. I looked at compiling roc code, but haven't made my way through the tutorials.

view this post on Zulip jan kili (Feb 19 2022 at 02:05):

No worries, I'm probably misunderstanding your "empty shell host" - it conjures horror movie vibes :laughing:

view this post on Zulip Jose Quesada (Feb 19 2022 at 04:25):

There might be another way to support dead code elimination at the platform level, platform language permitting.

Por example, let's say every publicaclly exposed function in Rust gets its own feature flag (this can be done automatically). Roc knows what functions are being exposed by rust, and can therefore emit the feature flags for the exact functions it needs. Rustc would take care of DCE natively.

I believe something similar should be possible with zig, but don't know.

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 05:44):

That definitely would work. That being said, I think generally speaking DCE doesn't matter too much, but if we want it in the future, there are many ways to add it. Personally I think feature flags are a less optimal solution. They require rebuilding the host and would have to be uniquely done for each host language.

view this post on Zulip Jose Quesada (Feb 19 2022 at 13:09):

Yes, the language by language basis would be the worst part.
You are also correct that DCE isn't super beneficial in most situations. Though I could imagine it being so for embedded systems. But as mentioned, there are alternative approaches for now.

view this post on Zulip Andrew Thompson (Feb 19 2022 at 16:01):

Is the expectation that platforms would be delivered normally as binaries?

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:10):

yeah the plan is that every platform ships with precompiled binaries

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:10):

for each target they support (e.g. Mac, Linux, Windows, etc.)

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:12):

I'm strongly opposed to the typical way that package managers handle native code, which is to say "build it on your local machine, and then some percentage of the time have installation fail with something cryptic like ld: could not resolve symbol __cxx_mumbo_jumbo in libwhatever.so"

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:14):

the design goal I have in mind is that from any clean operating system install, you can download the roc binary and use it to start writing Roc code, including automatically downloading and installing any platforms you specify, without needing any other tools installed on the system, and with platform compilation always succeeding

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:15):

ideally I'd also like the package repo to verify this - e.g. if the platform says it supports x64 Linux, then it looks for a precompiled binary and verifies that it's valid ELF, etc.

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:16):

so as an application author, the only programming language you should need to know to use Roc is Roc, and the only tool you should need to program Roc applications is roc

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:17):

Roc should never tell you "ok it's time to go learn about shared libraries now"

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:17):

unless the particular platform you're using is for building shared libraries or something :stuck_out_tongue:

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:18):

separately, I also like the idea of (but haven't thought it through completely yet) having a requirement that published platforms declare all their dynamically linked dependencies, and provide links for how to obtain them

view this post on Zulip Brian Carroll (Feb 19 2022 at 16:19):

So as a platform author the number of binaries you need to build is (number of OS's) x (number of CPUs), right? I guess that's the same as anywhere else you download pre-built binaries.

view this post on Zulip Richard Feldman (Feb 19 2022 at 16:20):

so if it's absolutely unavoidable to depend on a dynamic library (e.g. for licensing reasons), we can actually detect when it's missing right when you install the platform, and say "hey your system can't install this platform because it needs this other thing to be installed first; here are the instructions the platform author wrote for what to do if you find yourself in this situation: _______"

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 19:24):

Jose Quesada said:

Yes, the language by language basis would be the worst part.
You are also correct that DCE isn't super beneficial in most situations. Though I could imagine it being so for embedded systems. But as mentioned, there are alternative approaches for now.

100% true. I have been messing around with embedded Roc and it currently wastes a lot memory. The current problem is that Roc compiles for speed rather than size, and it's unused functions are not garbage collected. So it is actually roc side problems, not host side.

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 19:26):

I think I need to downgrade it's symbols to no longer be global so that they will properly get garbage collected.

view this post on Zulip Richard Feldman (Feb 19 2022 at 19:36):

its unused functions are not garbage collected

do you mean builtins?

view this post on Zulip Richard Feldman (Feb 19 2022 at 19:36):

I think unused functions written in Roc by the author of the program shouldn't get emitted in the first place :thinking:

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 19:40):

No, I mean for example, roc__mainForHost_1_exposed_generic when only roc__mainForHost_1_exposed is used.

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 19:41):

Oh, and __muloti4 which we still always emit

view this post on Zulip Brendan Hansknecht (Feb 19 2022 at 20:10):

Hmm, actually I am wrong about the size cost here. The real problem looks to be the loss of inlining. The binary bloats by way more than the size of the roc app. So when the app was completely rust, it inlined and greatly simplified most of the code. Since part is in roc, the inline potential is much more limited.
In this case, the Roc app is 1582 bytes, but using roc leads to the binary bloating by 4120 bytes.

view this post on Zulip J.R. 'hiljusti' Hill (Feb 20 2022 at 08:12):

I like the idea of 4120 bytes being considered bloat...

I've seen codebases that generate multi-GB server applications and yet they amount to little more than request -> call 0|n other things -> response.

view this post on Zulip Christian Dereck (Feb 21 2022 at 18:50):

Noob question: would this (having the platforms already compiled) simplify cross compiling or would the same problems still exist?

view this post on Zulip Brendan Hansknecht (Feb 21 2022 at 19:04):

If the platform is precompiled for all the architectures you want to target, roc essentially just hooks into that and is extremely portable. So cross compilation should be very easy. But yeah, depends on the platform supporting your target and llvm at least supporting the CPU architecture you're targeting.

view this post on Zulip Jose Quesada (Feb 23 2022 at 02:31):

I have an issue that perhaps someone else has run into?

I am implementing a custom platform, only modifying the roc_main function from a hello example. Everything in the platform side seems to not have an issue, however, I seem to be getting an error that doesn't make sense to me. Provided the following code:

app "hello-rust"
    packages { pf: "platform" }
    imports []
    provides [ main ] to pf

greeting =
    hi = "Hello"
    name = "World"

    "\(hi), \(name)!\n"

main = greeting

I am getting the following error when running:

    Finished dev [unoptimized + debuginfo] target(s) in 0.24s
     Running `target/debug/roc ./examples/speak-aloud/Speak.roc DDDDD`
🔨 Rebuilding host... Done!
SIOD ERROR: unbound variable : Hello
SIOD ERROR: comma-not-inside-backquote

The Hello it seems to be referring to is hello = "Hello" because if I change what's inside the quotes, the error changes with it

view this post on Zulip Jose Quesada (Feb 23 2022 at 02:31):

Any guesses as to a possible cause?

view this post on Zulip Jose Quesada (Feb 23 2022 at 02:33):

I can run other examples and code without issue, so I seem to be running into something else, but no idea how to trubleshoot in this case

view this post on Zulip Anton (Feb 23 2022 at 08:22):

Your platform sees Hello as a variable, perhaps it is expecting double quotes inside the string to interpret it as a string?
So something like:

app "hello-rust"
    packages { pf: "platform" }
    imports []
    provides [ main ] to pf

main = "\"Hello\""

view this post on Zulip Jose Quesada (Feb 23 2022 at 14:54):

Thank you Anton. I'll check it out. It shouldn't be the case, as all I am doing is slightly modifying the original hello rust example. Though it very much could be as most of the functions aren't documented yet :sweat_smile:

view this post on Zulip Anton (Feb 23 2022 at 15:11):

I'm not following completely, for which functions would you like to see more documentation?
Feel free to push a new branch to the repo with the speak-aloud example if you'd like anyone to help make it work.

view this post on Zulip Jose Quesada (Feb 23 2022 at 16:26):

@Anton, you were 100% right. I didn't really understand what you meant. I got it to work. I was spawning a child thread, but was expecting the program to panic on invalid input, but that didn't happen. I assumed the output I was getting was actually comming from Roc, but it was the child process. Thanks again.

view this post on Zulip Anton (Feb 23 2022 at 16:53):

Happy to help :)

view this post on Zulip Jorge Acereda (Feb 23 2022 at 22:12):

Why does link.rs mess environment variables? If I'm reading correctly, looks like it's clearing everything except HOME/PATH. That's a problem on Guix, it relies on environment variables to find includes and libs. Also, why isn't it using CC when building C host files?

view this post on Zulip Brendan Hansknecht (Feb 23 2022 at 22:23):

link.rs was cobbled together and has grown naturally. It really is not how we want to do linking in the long term. It works for most people, so it generally doesn't get messed with too much. I don't remember why, but I know that not clearing environment variables led to breakages on some machines
I know at one point there were some modifications to get it working on nixos (not sure if they got merged). Those might help with getting it working on guix.
Overall, please feel free to modify it and open pull requests. It could use some cleanup.

view this post on Zulip Brendan Hansknecht (Feb 23 2022 at 22:24):

In the future, platforms will be downloaded precompiled and/or specify their own build commands and linking will always go through the roc linker.

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:11):

LTO+musl:

jacereda@mbp ~/src/roc [env]$ size examples/hello-world/hello-world text data bss dec hex filename 14971 568 1664 17203 4333 examples/hello-world/hello-world

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:13):

Why glibc? IMHO it would be a good thing to decouple Roc from the C compiler/runtime, just use whatever CC/CFLAGS/LDFLAGS are defined... That would simplify quite a bit link.rs.

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:15):

the surgical linker is already that way, it just only works on Linux right now

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:16):

how do I invoke that?

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:17):

pass --roc-linker to the Roc compiler

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:19):

honestly I think it might simplify conversations around link.rs if we renamed it to temporary_pile_of_hacks.rs so it's more self-descriptive about how we should be thinking about investing in making it more robust :laughing:

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:19):

what size does that yield on your box?

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:19):

and/or put a comment at the top explaining how it's going away? :thinking:

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:19):

I'm on macOS at the moment, so I'm not sure offhand

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:20):

Jorge Acereda said:

what size does that yield on your box?

What specifically are you looking at the size for? Also, the roc linker + some changes to how platforms specify build commands would enable the platform to choose if it wants to use musl or not.

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:22):

to elaborate a bit, the surgical linker basically says "give me a working executable that attempts to dynamically link the Roc application entrypoint" (typically the main provided by the application author) and after the Roc application gets compiled into machine code, I will incorporate it into that executable and replace that dynamic linking with static linking after the fact

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:23):

Well, I do care about code bloat. I'm considering if this would be an appropriate language for embedded.

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:24):

so what you start out with is an executable that has some dynamic linking for the Roc application entrypoint, and what you end up with is that same executable except it's no longer doing dynamic linking for the Roc application entrypoint; instead, the entire compiled application has been added to the binary

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:24):

the Roc stdlib doesn't depend on libc at all

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:24):

it just says the host needs to provide a few functions (roc_alloc, roc_dealloc, roc_realloc, roc_memcpy, and roc_panic)

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:25):

it's completely up to the host how they want to implement those, they just need to be provided

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:25):

it's also completely up to the host how the executable gets compiled in the first place

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:25):

so there's no need for glibc or any libc at all

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:25):

at least from Roc's perspective

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:26):

how does that differ from -static plus -flto or --gc-sections?

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:26):

it's completely unrelated to them

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:26):

we don't call ld or anything like that

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:27):

the reason we call it the surgical linker is that literally all it's doing is adding the compiled application bytes to the binary, and then rewriting some headers to make some direct calls into dynamic ones

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:27):

so if you want to do another pass afterwards with a separate tool to GC sections, you totally can!

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:27):

but the Roc linker is narrowly scoped to only incorporate the compiled application into the precompiled host binary executable

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:27):

that's its entire job, it doesn't do anything else

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:28):

which is both super fast and also gives platform authors maximum flexibility

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:28):

because any linking system needs to do that at the bare minimum

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:28):

do you mean patching the elf headers?

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:28):

yeah exactly

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:29):

and Mach-O on macOS - that one's in progress

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:29):

Jorge Acereda said:

Well, I do care about code bloat. I'm considering if this would be an appropriate language for embedded.

Very cool. Great to know the goal. For the embedded case, I don't think you will touch any roc linking stuff at all. Instead, I believe that you would just have roc emit a static library. Then you would have the embedded targeting platform deal with all of the linking.
That is what I am doing here when targeting the nrf chip on the microbit.

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:29):

and does that work for static and dynamic executables?

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:29):

oh, also currently the surgical linker works for x64 ELF only - that's probably relevant!

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:32):

Also, have you considered APE?

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:32):

cosmopolitan libc

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:32):

having a single binary across OSs could be really nice

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:33):

never heard of APE or cosmopolitan libc

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:33):

https://justine.lol/cosmopolitan/index.html

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:34):

That would be tagential to Roc. That would be a platform decision.

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:37):

right - from the Roc compiler's perspective, the goal is to be completely decoupled from libc

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:38):

so platform authors can use whatever libc they want, or decline to use libc altogether

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:38):

and everything still works!

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:40):

with cosmopolitan and some hacks you can end up having a single executable with OpenGL graphics running on Windows/Linux/NetBSD/FreeBSD, I did some experiments and it's totally possible... The only showstopper so far was OpenBSD that has some security measures that would need to be circumvented, but it's also possible with some more work.
https://github.com/jacereda/cosmogfx

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:41):

The main problem that I believe will limit roc for embedded is that roc and the platform are two separate chunks of code. They are not compiled together. They can't take advantage of inlining. This can add a huge number of bytes to an executable. In the roc-microbit case, using roc costs about 2500 bytes more than just using rust due to the loss of inlining.

Of course, depending on the platform boundary this could potentially be greatly reduced.
In reality, the solution is probably LTO between the platform and the Roc app, but that is hard because LTO is dependent on the specific version of LLVM. So it would be very brittle. Also more complex to setup in general.

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:43):

is having a C codegen out of question?

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:43):

well, another alternative could be to use -flto

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:44):

that way the embedded platforms could inline just fine

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:44):

C codegen technically could be done, but I don't think anyone in the project would really want to support that.

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:44):

heh, I've actually thought about it for exactly that use case

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:44):

it's a big can of worms though

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:45):

"that use case" being compiling platforms and applications together into one executable

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:45):

that can be optimized as a whole

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:45):

Also, lto require llvm version matching which is doable, but brittle. If roc is using llvm version 13 and the c compiler for your device is only on gcc or is llvm 12, lto won't work.

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:45):

the original idea was to do that via LLVM bytecode, but...yeah, that :point_up:

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:45):

that would be a nightmare to maintain

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:46):

in comparison, we could emit (for example) C99 and say "here you go, do whatever you want with it"

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:46):

and that's at least a stable interface!

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:48):

I guess the dependency on the LLVM version is just a matter of leaving LLVM mature a bit?

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:49):

Not really. LLVM intentionally wants people not to depend on that so that they can change it if they every think of better representations or other ideas.

view this post on Zulip Jorge Acereda (Feb 24 2022 at 22:50):

I see... that makes a C (or zig?) backend more interesting then.

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:50):

I guess theoretically it should stabilize, but LLVM still will never make promises about the stability or that tools will work when dealing with multiple versions.

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:51):

Jorge Acereda said:

I see... that makes a C (or zig?) backend more interesting then.

As weird as that sounds. 100%.

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:52):

That or accepting the loss of inlining between platform and application/ architecturing platforms in ways that make it matter less.

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:53):

For example, I think things are made worse by embedded rust. Embedded rust is a giant pile of "zero" cost abstractions. The issue is that they depend on inline to be abstracted away. So when we break the inlining, I think it causes the rust to bloat. I think C or rust without the abstractions could be done in a way that the inlining basically didn't cause any problems.

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:54):

Oh also, we have to figure out to optimize roc for size. Currently we can only optimize for speed or not at all. Some reason size optimization isn't exposed through the libraries we use.

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:58):

Also, emitting C or Zig doesn't innately solve this. If I was still trying to use embedded rust and you emit C, I still wouldn't get inlining without all of the llvm hassles.

view this post on Zulip Richard Feldman (Feb 24 2022 at 22:59):

well you could probably run a script to translate the C to Rust

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 22:59):

But emitting C does fix it for C and C++ and Zig. Also, probably easier to get a C compiler with the right llvm version to match my rust compiler.

view this post on Zulip Ayaz Hafiz (Feb 24 2022 at 23:01):

Brendan Hansknecht said:

Oh also, we have to figure out to optimize roc for size. Currently we can only optimize for speed or not at all. Some reason size optimization isn't exposed through the libraries we use.

I think optimizing for speed may be the same as -O2: https://llvm.org/doxygen/CodeGen_8h_source.html

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 23:08):

Ah, that would make sense. So it looks like Os is O2 minus some things and Oz is Os minus loop vectorization: https://stackoverflow.com/questions/15548023/clang-optimization-levels

view this post on Zulip Brendan Hansknecht (Feb 24 2022 at 23:51):

So I just tested adding optimizing for size to roc. Makes quicksort 3x larger. I guess something we do in our llvm generation really depends on some of the more aggressive optimizations.

view this post on Zulip Brendan Hansknecht (Feb 25 2022 at 00:00):

Ah, nvm. link.rs was still compiling the platform in debug mode. That makes way more sense.

view this post on Zulip Brendan Hansknecht (Feb 25 2022 at 00:16):

Ok, after updating link.rs to also build for size, quicksort goes from 256K to 148K.

view this post on Zulip Brendan Hansknecht (Feb 25 2022 at 00:16):

So definitely works

view this post on Zulip Brendan Hansknecht (Feb 25 2022 at 00:18):

Though I guess most of that is attributed to zig having ReleaseSmall

view this post on Zulip Ivo Balbaert (Jun 01 2023 at 10:25):

(https://github.com/roc-lang/basic-cli/tree/main/examples)
Hi all, I am a beginner to Roc, so I might overlook something, but I can't explain the following incomplete output when running time.roc:

main =
    start <- Utc.now |> Task.await
    dbg start       # => [time.roc 14:9] @Utc 1685614693863401198
    {} <- slowTask |> Task.await    # => Tried to open a file...
    finish <- Utc.now |> Task.await
    dbg finish      # => [time.roc 18:9] @Utc 1685614693863545706
    duration = Utc.deltaAsNanos start finish |> Num.toStr
    dbg duration
    Stdout.line "Completed in \(duration)ns"

slowTask : Task.Task {} []
slowTask =
    path = Path.fromStr "not-a-file-but-try-to-read-anyway"
    result <- File.readUtf8 path |> Task.attempt
    when result is
        _ -> Stdout.line "Tried to open a file..."

The dbg output of start and finish is shown, as well as Stdout.line from slowtask, but no output of dbg duration or no Stdout.line with durarion output. What could be a reason for this? Is this a bug? Thanks!

view this post on Zulip Fábio Beirão (Jun 01 2023 at 11:34):

If you remove that |> Num.toStr after the Utc.deltaAsNanos would the dbg print?

view this post on Zulip Fábio Beirão (Jun 01 2023 at 11:35):

I could be confusing with other topics, but I think there's some bugfixing happening around Num.toStr and I'm just wondering if that would be the culprit as to why dbg duration would not print :thinking:

view this post on Zulip Ivo Balbaert (Jun 01 2023 at 12:12):

Fábio Beirão said:

If you remove that |> Num.toStr after the Utc.deltaAsNanos would the dbg print?

You're right! dbg duration then prints out! But there is no Stdout.line display of duration. The problem with Num.toStr probably still prevents that, no error is shown, just no output. Thanks anyway!

view this post on Zulip Fábio Beirão (Jun 01 2023 at 12:14):

Just out of curiosity, which roc version are you using? ( roc --version )

view this post on Zulip Ivo Balbaert (Jun 01 2023 at 12:34):

My fault! I was using a version of perhaps 1.5 weeks old. Before answering, I did an update to most recent source and build it, and the duration now is displayed:
Tried to open a file...
Completed in 59922ns
Thanks for being patient !

view this post on Zulip Fábio Beirão (Jun 01 2023 at 12:37):

Absolutely not your fault :) I am happy I could help, with my very very limited roc knowledge. :pray:

view this post on Zulip Slazaa (Dec 25 2023 at 18:48):

I'm trying to write a platform in Zig and I get the following error when running roc run

thread 'main' panicked at 'There were still outstanding Arc references to module_ids', crates/compiler/load_internal/src/file.rs:1560:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

view this post on Zulip Brendan Hansknecht (Dec 25 2023 at 20:52):

We will need to see more source to diagnose that. Probably has nothing to do with zig, but instead roc loading or specific parsing issues.

view this post on Zulip Brendan Hansknecht (Dec 25 2023 at 20:53):

Try roc check may reveal the real issue, if not, we probably need to see the code and mess around to figure out what is going on.

view this post on Zulip Luke Boswell (Dec 25 2023 at 20:56):

I thi k I've seen this error or similar before, something like the way Interface or Package modules imported incorrectly. It's a shame we dont have a better error here yet, but if we can get a minimal repro maybe that would help.

view this post on Zulip Luke Boswell (Dec 25 2023 at 20:57):

Though I think Agu's work implementing module params design will cleanup and fix those issues soon.

view this post on Zulip Brendan Hansknecht (Dec 25 2023 at 20:59):

Yeah, sounds like what I remember when it was last hit.


Last updated: Jul 05 2025 at 12:14 UTC