Thinking out loud · platform development

I’m going to do some thinking out loud here as I start to get my bearings by reading code and documentation. Feel free to chime in if you’d like, but I don’t have any expectation of community responsiveness.

Luis Gutierrez (Oct 05 2021 at 21:05):

So far I’ve looked at the Hello World examples in C and zig. It looks like the host files implement some common methods (memory management… anything else?). I guess each host must implement some minimal interface to be a valid platform. I wonder if that interface is documented anywhere. (Hmm, also what’s with the “platform” and “host” names? Are they synonyms, or are they different somehow?) If not, I wonder if it makes sense to start this documentation. I’m also curious what this interface depends on— might give some insight on thinking about the stability of this interface.

Thomas (Oct 11 2022 at 11:41):

What is the "glue"?
What is needed to start a platform from scratch using a C compiler and the roc binary?
Are there any assumptions or dependency on Rust tooling in order to set up a platform from scratch?

Anton (Oct 11 2022 at 11:53):

Our glue generator is made for rust platforms. If you want to work with a c platform no Rust tooling is necessary.

Thomas (Oct 11 2022 at 11:56):

I have looked at c-platform / zig-platform and friends here, but I'm mostly confused about how to go from pf.roc + main.roc into something runnable I think...

Does Roc make assumptions about the name or location about my host.c file? (or host.o or similar?)

What exactly do I need to provide to a Roc platform from the native side assuming I start out with a host.c file exactly like the above link

Thomas (Oct 11 2022 at 12:00):

platform "basic"
    requires {} { main : Str }
    exposes []
    packages {}
    imports []
    provides [mainForHost]

mainForHost = main

🔨 Rebuilding platform...
An internal compiler expectation was broken.
This is definitely a compiler bug.
Please file an issue here: https://github.com/roc-lang/roc/issues/new/choose
thread '<unnamed>' panicked at 'No such file or directory (os error 2)', crates/linker/src/lib.rs:357:29
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', crates/cli/src/build.rs:307:46

What is the "internal compiler expectation", and how to I fulfill it rather than breaking it?

Thomas (Oct 11 2022 at 12:01):

Interestingly, if I make an empty host.c file in the same directory, I get the following:

🔨 Rebuilding platform...
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', crates/compiler/build/src/link.rs:456:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', crates/cli/src/build.rs:307:46

Anton (Oct 11 2022 at 12:02):

Thomas (Oct 11 2022 at 12:02):

Anton (Oct 11 2022 at 12:04):

Thomas (Oct 11 2022 at 12:05):

I'll have to clone the repo, give me a second
I just have the nightly release atm

Thomas (Oct 11 2022 at 12:06):

~/Projects/github.com/roc-lang/roc/crates/cli_testing_examples/platform-switching$ roc run rocLovesC.roc

# (Padding added for separation)

🔨 Rebuilding platform...
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', crates/compiler/build/src/link.rs:456:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', crates/cli/src/build.rs:307:46

Thomas (Oct 11 2022 at 12:07):

Thomas (Oct 11 2022 at 12:08):

I do have a boken rustc in my PATH, could that be messing with Roc's assumptions? I'll try removing it

Anton (Oct 11 2022 at 12:09):

Not sure what OS you are on but I think you need to do your equivalent of sudo apt install build-essential clang

Thomas (Oct 11 2022 at 12:10):

I am on Debian 11, but is there a way for me to do my host building myself? Or will Roc always rely on system-installed build tools no matter what I do?

Anton (Oct 11 2022 at 12:12):

We have a flag you can pass to the roc binary something like --prebuilt-platform that would allow you to build the host yourself I think.

Thomas (Oct 11 2022 at 12:12):

Anton (Oct 11 2022 at 12:12):

Thomas (Oct 11 2022 at 12:16):

Roc now spits out libapp.so apparently! (but still errors with the above error) I'll mess around a bit with this file

Thomas (Oct 11 2022 at 12:17):

Anton (Oct 11 2022 at 12:26):

Thomas (Oct 11 2022 at 12:32):

I didn't really build anything yet, I don't have clang installed globally, but I would use zig cc, so that returns clang version 15.0.0 (git@github.com:ziglang/zig-bootstrap.git 9be8396b715b10f64d8a94b2d0d9acb77126d8ca)

Thomas (Oct 11 2022 at 12:33):

Roc spits out libapp.so without any global clang installation when I just do roc build now, and it seems to have roc__mainForHost symbols in it, although Roc still crashes with that Rust error above

Thomas (Oct 11 2022 at 12:33):

Thomas (Oct 11 2022 at 12:36):

platform "basic"
    requires {} { main : Str }
    exposes []
    packages {}
    imports []
    provides [mainForHost]

mainForHost = main

app "basic"
    packages { pf: "pf.roc" }
    imports []
    provides [main] to pf

main = "Hello"

Running roc build will produce libapp.so in current folder and then crash with the following message (with RUST_BACKTRACE=1)

🔨 Rebuilding platform...
An internal compiler expectation was broken.
This is definitely a compiler bug.
Please file an issue here: https://github.com/roc-lang/roc/issues/new/choose
thread '<unnamed>' panicked at 'No such file or directory (os error 2)', crates/linker/src/lib.rs:357:29
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: roc_linker::open_mmap
   3: roc_linker::elf::preprocess_elf
   4: roc_linker::build_and_preprocess_host
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', crates/cli/src/build.rs:307:46
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1785:5
   3: roc_cli::build::build_file
   4: roc_cli::build
   5: roc::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Thomas (Oct 11 2022 at 12:36):

Thomas (Oct 11 2022 at 12:40):

If I pass --prebuilt-platform true it will NOT produce a libapp.so file, and will crash with a similar, but slightly different error:

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', crates/cli/src/build.rs:451:74
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1785:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', crates/cli/src/build.rs:307:46
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1785:5
   3: roc_cli::build::build_file
   4: roc_cli::build
   5: roc::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Anton (Oct 11 2022 at 13:19):

The "No such file or directory" is because roc is trying to call clang, see this line.

Thomas (Oct 11 2022 at 13:54):

Only the code path where I don't pass --prebuilt-platform true actually compiles the Roc code though, I'm not sure how to tell it to not compile anything except the Roc parts

Travis (Oct 11 2022 at 14:12):

i've seen the same error due to missing clang install. i wonder how difficult it would be to improve that from a panic to error message stating something like "'clang' couldn't be found"? and the same for when rust/zig arent found. i've been confused by similar error messages due to not having a rust compiler.

Travis (Oct 11 2022 at 14:15):

i'm on debian too. at the time i had clang-14 installed. my solution was to just ln -s /usr/bin/clang-14 ~/.local/bin/clang. this assumes ~/.local/bin is in your path.

Travis (Oct 11 2022 at 14:18):

but then i was told by richard that roc expects clang-13 and that while 14 might be an ok alternative its not recommended

Brendan Hansknecht (Oct 11 2022 at 14:20):

Platforms and building them currently have a lot of sharp edges. We definitely need to plan out and improve their development. Currently, the compiler just implicitly knows a number of commands for building various types of platforms, but it is definitely not scalable. It also does not give good error messages. Long term there are many larger plans around platforms (and the ecosystem), but currently the knowledge is pretty sparse.

Brendan Hansknecht (Oct 11 2022 at 14:20):

Shouldn't actually matter. Richard was trying to match llvm versions, but we consume the platform as an object file which is fully compiled to assembly, so no llvm matching needs to be done.

Brendan Hansknecht (Oct 11 2022 at 14:21):

Extra not on glue, the long term plan is for it to generate a mix of languages using plugins that anyone can write. Currently rust is just the only supported language.

Travis (Oct 11 2022 at 14:22):

hmm in that case if Thomas already has zig installed maybe just symlink or alias clang to 'zig cc'?

Brendan Hansknecht (Oct 11 2022 at 14:22):

Brendan Hansknecht (Oct 11 2022 at 14:24):

Yeah, that should theoretically work. Though have to make sure the roc program actually reads the alias. Some shells don't seem to pass aliases to run applications (not sure exactly how that works).

Brendan Hansknecht (Oct 11 2022 at 14:24):

Definitely should work to put a clang shell script in your path that just called zig cc with all of the args.

Brendan Hansknecht (Oct 11 2022 at 14:26):

As an extra note, platform pains are extra apparent because we don't have any form of package ecosystem yet. Long term, end users won't need to compile platforms (at least generally speaking). The compiler will download a prebuilt platform and link to that. As such, you would see these errors unless creating a new platform

Brendan Hansknecht (Oct 11 2022 at 14:28):

Also, roc has an option to not link and just dump an object file that can be consumed by anything. That is how I got roc running on a microcontroller, for example. Using that feature means you could use another languages build tooling if wanted.

Brendan Hansknecht (Oct 11 2022 at 14:30):

Also, I think this points more towards the need for a platform dev beginners sessions (along with a write up probably). It used to be that the api change a lot, so we really didn't want to try and document it. Now it has been much more stable, so some good docs and better error messages would probably be worth investing in.

jan kili (Oct 11 2022 at 14:52):

Any appetite for Roc Meetup #2? I was thinking maybe mid/late November, to give time for both Roctoberfest and AoC23

Brendan Hansknecht (Oct 11 2022 at 14:52):

Thomas (Oct 13 2022 at 15:53):

Thomas (Oct 20 2022 at 07:53):

Seems like I got quite a bit further this time around, I made a host.zig file and populated it with stubs, now I'm running into this (roc build && ./basic):

🔨 Rebuilding platform...
0 errors and 0 warnings found in 767 ms while successfully building:

    basic
./basic: symbol lookup error: ./basic: undefined symbol: roc__mainForHost_1_exposed_generic

app "basic"
    packages { pf: "pf.roc" }
    imports []
    provides [main] to pf

main = "Hello"

platform "basic"
    requires {} { main : Str }
    exposes []
    packages {}
    imports []
    provides [mainForHost]

mainForHost = main

Thomas (Oct 20 2022 at 07:54):

const std = @import("std");

extern fn roc__mainForHost_1_exposed_generic([*]const u8) void;

pub fn main() !void {
    var buf: [1024]u8 = undefined;
    roc__mainForHost_1_exposed_generic(&buf);
    std.io.getStdOut().writer().print("{any}\n", .{buf}) catch {};
}

export fn roc_alloc() void {
    std.io.getStdOut().writeAll("roc_alloc\n") catch {};
}
export fn roc_realloc() void {
    std.io.getStdOut().writeAll("roc_realloc\n") catch {};
}
export fn roc_dealloc() void {
    std.io.getStdOut().writeAll("roc_dealloc\n") catch {};
}
export fn roc_panic() void {
    std.io.getStdOut().writeAll("roc_panic\n") catch {};
}
export fn roc_memcpy() void {
    std.io.getStdOut().writeAll("roc_memcpy\n") catch {};
}
export fn roc_memset() void {
    std.io.getStdOut().writeAll("roc_memset\n") catch {};
}