Stream: contributing

Topic: playground/repl with LSP support


view this post on Zulip Bram (Jul 24 2024 at 16:44):

Hey, I think nothing is preventing us from compiling the language server to WASM and run it in the browser. Would there be interest for this?

If so, I would like to make a small PoC that hooks up a CodeMirror editor with some basic LSP functionality (code completions, hover documentation & diagnostics). If this is a success, we could create a neat playground. An interactive tutorial such as https://go.dev/play/ would especially be cool.

view this post on Zulip Anton (Jul 24 2024 at 16:57):

Excellent idea @Bram! That could work, I suspect @Eli Dowling may know best

view this post on Zulip Richard Feldman (Jul 24 2024 at 17:28):

that sounds awesome! I'd love to have a play.roc-lang.org

view this post on Zulip Richard Feldman (Jul 24 2024 at 17:32):

I have some thoughts on both the software and infrastructure architecture for the production version of something like that, so @Bram maybe we can chat about that once you're at the proof-of-concept stage! :smiley:

view this post on Zulip Eli Dowling (Jul 24 2024 at 21:59):

@Bram has suggested exactly my inclination.
Use Monaco for the editor and connect it to the wasm compiled roc language server.
(Let me know if you have any major issues trying to run it in a browser, I suspect there may be some challenges)
The obvious question is: What platform should we use?
If there isn't an existing wasm platform we'd want to make one that just exposes browser stuff: console log, http request etc

view this post on Zulip Eli Dowling (Jul 24 2024 at 22:21):

Well, after soem research, running a language server in the browser via wasm is not "the done thing"

I did find a little prior art:

https://www.hiro.so/blog/write-clarity-smart-contracts-with-zero-installations-how-we-built-an-in-browser-language-server-using-wasm

But it looks pretty complex to me.

view this post on Zulip Eli Dowling (Jul 24 2024 at 22:23):

This Is pretty promising:
https://github.com/qualified/lsps

view this post on Zulip Richard Feldman (Jul 24 2024 at 22:23):

Eli Dowling said:

Well, after soem research, running a language server in the browser via wasm is not "the done thing"

doing the thing that's not "the done thing" is kind of our specialty around here :big_smile:

view this post on Zulip Richard Feldman (Jul 24 2024 at 22:26):

since we have our own ls code base written in Rust, I'd have to imagine we could compile that code to wasm and change it to expose its operations as ordinary js function calls (via wasm-bindgen) and then the code editor could call them directly, yeah?

view this post on Zulip Richard Feldman (Jul 24 2024 at 22:27):

or I guess maybe it would need to run in a web worker so it didn't block the main thread

view this post on Zulip Eli Dowling (Jul 24 2024 at 22:28):

Definitely needs to be in a we worker.
This looks promising too:
https://github.com/FurqanSoftware/codemirror-languageserver

view this post on Zulip Richard Feldman (Jul 24 2024 at 22:28):

so not quite direct function calls, but rather sending json to/from a web worker running the compiled wasm

view this post on Zulip Eli Dowling (Jul 24 2024 at 22:29):

Everything I'm seeing points towards codemirror as being a more popular option for the editor.
I think that's because Monaco may not actually provide the completion ui stuff. I think that might be something vscode adds on-top

view this post on Zulip Eli Dowling (Jul 24 2024 at 22:34):

I think the main issue I've seen, is filesystem related things which some langue severs must use.
It seems some solutions involve simulating a filesystem In the browser.
I'd have to check if that's an issue we'd face

view this post on Zulip Richard Feldman (Jul 24 2024 at 23:08):

since we're in charge of both the compiler and the language server, there are ways we can get around that :big_smile:

view this post on Zulip Eli Dowling (Jul 25 2024 at 00:02):

Yeah, from memory I don't think we actually make any filesystem calls because we invoke the compile with the source directly.

view this post on Zulip Bram (Jul 25 2024 at 07:29):

Cool, I will try to hack something together this weekend. It will indeed run in a web worker. Regarding the editor, there is a popular language server extension for Monaco, but CodeMirror is a bit more lightweight, which might be beneficial if we want to reuse the editor for the repl on the landing page for example.

That said, _a lot_ of people use vscode, so that would feel more familiar. We can discuss the choice of editor and the overall architecture after I got something working ;)

view this post on Zulip Bram (Jul 25 2024 at 11:56):

After browsing the source code of the LSP a bit more, we use Url::to_file_path and Url::from_file_path in analysis.rs which are not supported in wasm32-unknown-unknown

view this post on Zulip Richard Feldman (Jul 25 2024 at 12:20):

we could vendor those

view this post on Zulip Richard Feldman (Jul 25 2024 at 12:21):

or swap to this implementation which is no_std

view this post on Zulip Bram (Jul 25 2024 at 13:04):

I managed to build a wasm version of the LSP : D
I want to split up the language_server crate in three:

view this post on Zulip Bram (Jul 25 2024 at 13:05):

any feedback on the naming of the crates, or sounds good?

view this post on Zulip Bram (Jul 25 2024 at 13:10):

I think the only thing I need in language_server_wasm is this lib.rs:

use tower_lsp::{LspService, Server};
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub async fn serve(input: web_sys::ReadableStream, output: web_sys::WritableStream) {
    console_error_panic_hook::set_once();
    web_sys::console::log_1(&"::serve".into());

    let input = wasm_streams::ReadableStream::from_raw(input).into_async_read();
    let output = wasm_streams::WritableStream::from_raw(output).into_async_write();

    let (service, messages) = LspService::new(|client| roc_language_server_core::RocServer::new(client));
    Server::new(input, output, messages).serve(service).await;
}

view this post on Zulip Bram (Jul 25 2024 at 21:43):

and we got diagnostics in the browser:
image.png

view this post on Zulip Bram (Jul 25 2024 at 21:51):

Which also brings up this question:
image.png

view this post on Zulip Luke Boswell (Jul 25 2024 at 22:40):

Some thought. We have compiled the LS and linked it up so that it works in the browser.

So, when the LS analyses an app module, it is going to try and then compile the platform roc files that comes along with it. But we haven't yet provided these.

One option is that we might be able to have a "hostless" platform just for working in the browser. I've been calling this a "plugin" use-case. We have an example in roc glue, where the glue platform is all roc files and there is no host. Perhaps we could even have some effects that are supported.

Another option, is that we enable the use of any platform ... using a URL. But we would need some machinery in the browser to go and fetch the platform package and then make it available to the LS somehow. We probably can't provide any effects in this case... but at the risk of going down another tangent, wouldn't it be cool if the platform authors also bundled a WASM build of their host which includes effect implementations tailored just for the web editor.

view this post on Zulip Richard Feldman (Jul 25 2024 at 23:10):

yeah let's do that!

view this post on Zulip Richard Feldman (Jul 25 2024 at 23:10):

I think that's exactly what we should do

view this post on Zulip Richard Feldman (Jul 25 2024 at 23:11):

in fact I think bundles are already set up for that, right?

view this post on Zulip Luke Boswell (Jul 25 2024 at 23:34):

@Richard Feldman -- I mentioned two options.

  1. make a host-less platform just for editor
  2. fetch platform URLs (and maybe suupport an editor specific WASM module)

view this post on Zulip Luke Boswell (Jul 25 2024 at 23:41):

Thinking about this some, I think the obvious answer is option 2.

We definitely should be able to download any platform for the purpose of a code editor.

view this post on Zulip Luke Boswell (Jul 25 2024 at 23:42):

The added platform authors including a WASM bundle for editor effect simulation thing... is probably more of a cool future idea, than something we should attempt right now.

view this post on Zulip Luke Boswell (Jul 25 2024 at 23:45):

So I guess the LS for the editor would get the URL from the app module header, then pass that back to JS which goes and downloads the package and then provides the platform roc files so the LS can continue. I'm not sure how all this would get plumbed together to work in a browser.

view this post on Zulip Luke Boswell (Jul 25 2024 at 23:47):

Any host binaries in the platform package, like macos-aarch64.a or linux-x64.rh can just be ignored -- or even removed and not retained in the browser cache.

I assume browsers are more than happy downloading larger files, in the order of 10's of MB, and caching this in local storage or somewhere.

view this post on Zulip Luke Boswell (Jul 25 2024 at 23:48):

Also we would need to untar the .tar.br or .tar.gz in the browser

view this post on Zulip Luke Boswell (Jul 25 2024 at 23:48):

This is a super cool project!!

view this post on Zulip Richard Feldman (Jul 26 2024 at 00:00):

Luke Boswell said:

Any host binaries in the platform package, like macos-aarch64.a or linux-x64.rh can just be ignored -- or even removed and not retained in the browser cache.

I assume browsers are more than happy downloading larger files, in the order of 10's of MB, and caching this in local storage or somewhere.

yep! It’ll have to be indexeddb for binary data, but totally doable!

view this post on Zulip Richard Feldman (Jul 26 2024 at 00:01):

although night be able to just get away with using the browser’s http cache

view this post on Zulip Richard Feldman (Jul 26 2024 at 00:03):

also I think depending on the platform it might not even be that hard to offer a wasm version

view this post on Zulip Richard Feldman (Jul 26 2024 at 00:03):

for example, the playground could offer a fake wasi interface maybe (somebody has surely made a wasi implementation that runs in the browser to that extent possible)

view this post on Zulip Richard Feldman (Jul 26 2024 at 00:04):

and then basic-cli could ship a wasm binary that expects to be connected to a wasi implementation somehow

view this post on Zulip Richard Feldman (Jul 26 2024 at 00:04):

so you could do the normal basic-cli tutorial in the playground! :smiley:

view this post on Zulip Luke Boswell (Jul 26 2024 at 00:08):

One thought about the platform package... if we do the work to unzip it in the browser, and then strip out the binary files, we are just left with the .roc files. So my thoughts we would only keep these in the "cache", and the next time the LS recompiles we wont need to unzip the file again.

view this post on Zulip Bram (Jul 26 2024 at 09:45):

So I guess the LS for the editor would get the URL from the app module header, then pass that back to JS which goes and downloads the package and then provides the platform roc files so the LS can continue. I'm not sure how all this would get plumbed together to work in a browser.

I think we can just make the roc_packaging crate wasm compatible, right?
reqwest already supports wasm (will use the fetch API under the hood), the only thing is that because of that, it only supports the async client and not the blocking one that is used here: https://github.com/roc-lang/roc/blob/d5db3137a3d8da46f92c31b6bf088bc495f759c2/crates/packaging/src/https.rs#L254

view this post on Zulip Bram (Jul 26 2024 at 11:03):

The cache implementation of roc_packaging should probably be made custom for the browser though (storing it in indexed db, for example).

Another option would be to use the new file system API: https://developer.mozilla.org/en-US/docs/Web/API/File_System_API

There is also: https://github.com/WebAssembly/wasi-filesystem
And experimental shim for the browser: https://github.com/bytecodealliance/jco/tree/main/packages/preview2-shim

view this post on Zulip Bram (Jul 26 2024 at 11:19):

so you could do the normal basic-cli tutorial in the playground!

Yep, and that means not just LSP support, but compiling & running complete roc apps, all from within the browser, right?

view this post on Zulip Luke Boswell (Jul 31 2024 at 10:45):

@Bram @Eli Dowling I've been thinking about this work, and am excited about the possibilities.

One idea I've had is that this could be an opportunity to upgrade our website examples site, source fiels at roc-lang/examples to be more interactive.

I know it's early -- we're still working on how to get the editor compiling, and everything linked up.

But I wonder, imagining we had a playground/repl, could we/should we use this for the roc website examples? are there things that we could do to prepare for this that might also be beneficial in the short term?

For example, would it be useful if our examples were in another format? or you could download an index of the roc source files?

view this post on Zulip Sam Mohr (Jul 31 2024 at 10:46):

The gleam tour seems like good reference material here: https://tour.gleam.run/

view this post on Zulip Sam Mohr (Jul 31 2024 at 10:47):

It has examples of the main language features that are loaded into the playground already correct and runnable, and the user can mess with it to change things and see what happens

view this post on Zulip Sam Mohr (Jul 31 2024 at 10:48):

Having code that's already ready to go should make it more useful for users since they don't have to go find non-trivial code to mess with

view this post on Zulip Luke Boswell (Jul 31 2024 at 10:51):

It's also really nicely styled, and reads like a book.

view this post on Zulip Sam Mohr (Jul 31 2024 at 10:51):

Yes, Gleam is nothing if not polished

view this post on Zulip Eli Dowling (Jul 31 2024 at 10:53):

Oh hey, I have another very specific suggestion for this!

You can use a templating system.
I recently had to write some documentation and blog posts that included code snippets and wanted the code to be runnable and always work and be up to date with any changes.

I made a simple templating system that lets you define indexed code blocks using comments and then insert those into points in a document.

#$$S1
a= myfunc
#$$E1
As you can see we define a function called a:
$$1

Then you remove all those line comments, do the templating and you can have documentation as well as the full working source.
That way the documentation part stays in sync with the full code.

view this post on Zulip Eli Dowling (Jul 31 2024 at 10:53):

My system is in Ocaml but I could translate it to roc.

view this post on Zulip Sam Mohr (Jul 31 2024 at 10:55):

So yeah, I like the idea of setting up the examples repo to somehow be automatically loaded as a set of toys for a REPL playground

view this post on Zulip Sam Mohr (Jul 31 2024 at 10:58):

@Eli Dowling sounds cool! If you know of a way to get a system like the one you've got running to work with a fast dev loop, as an LSP would provide, that'd be super cool! That tends to be the thing missing in generated code contexts

view this post on Zulip Luke Boswell (Jul 31 2024 at 11:00):

The examples site (along with the roc website) are currently generated from markdown files using https://github.com/lukewilliamboswell/basic-ssg

view this post on Zulip Luke Boswell (Jul 31 2024 at 11:01):

Could we use something like this?

view this post on Zulip Luke Boswell (Jul 31 2024 at 11:01):

Can we do it all with static files and just a content server?

view this post on Zulip Eli Dowling (Jul 31 2024 at 11:02):

Sam Mohr said:

Eli Dowling sounds cool! If you know of a way to get a system like the one you've got running to work with a fast dev loop, as an LSP would provide, that'd be super cool! That tends to be the thing missing in generated code contexts

It's almost instant, so I run it with a file watcher.

view this post on Zulip Eli Dowling (Jul 31 2024 at 11:03):

Yeah, you just have a set of input files and then an output folder that contains the generated markdown and code files with the special templating comments removed

view this post on Zulip Eli Dowling (Jul 31 2024 at 11:03):

so you can just check everything into source control

view this post on Zulip Eli Dowling (Jul 31 2024 at 11:05):

you could run the template tool as an extension to the ssg generator. As pass that runs before that.

view this post on Zulip Luke Boswell (Jul 31 2024 at 11:06):

We could add effects, like a Task that could be run by the roc app before generating the html files.

view this post on Zulip Eli Dowling (Jul 31 2024 at 11:06):

Yeah that would be cool, having a little pre-build step we can add in there

view this post on Zulip Anton (Jul 31 2024 at 13:14):

I've thought about our current system before and I'd really like to include the Roc code directly into the .md file.
So currently in our md files we do:

# Code
'''roc
file:main.roc
'''

And I'd love this to be:

# Code
'''roc
app [main] { pf: platform "..." }

import pf.Stdout
import pf.Task

main =
    Stdout.line! "Hello, World!"
'''

I tried to set up our tutorial to work with file:main.roc but it's just way too many files and needless indirection.

The templating references ($$1) would indeed be great to reference small parts of the code.

view this post on Zulip Anton (Jul 31 2024 at 13:14):

I've been wondering if we should make the compiler able to handle markdown files, so it can check or execute or test all the roc code blocks. Doc comments are basically already markdown documents and we definitely want to be able to check any roc code blocks in those doc comments.

view this post on Zulip Bram (Jul 31 2024 at 14:41):

Luke Boswell said:

Bram Eli Dowling I've been thinking about this work, and am excited about the possibilities.

One idea I've had is that this could be an opportunity to upgrade our website examples site, source fiels at roc-lang/examples to be more interactive.

I know it's early -- we're still working on how to get the editor compiling, and everything linked up.

But I wonder, imagining we had a playground/repl, could we/should we use this for the roc website examples? are there things that we could do to prepare for this that might also be beneficial in the short term?

For example, would it be useful if our examples were in another format? or you could download an index of the roc source files?

So the LSP in the browser is already functional. I hooked up diagnostics, docs on hover and auto completions into codemirror (if we decide to use this editor is to be decided, but works well so far). The main blocker now is getting platform support for wasm-unknown-unknown and/or wasi (there must be a wasi browser runtime out there).

Regarding the examples, I think we can just do SSG and embed the example source code into the HTML. This is also what gleam's playground does: https://github.com/gleam-lang/language-tour/blob/main/src/tour.gleam#L695 and https://github.com/gleam-lang/language-tour/blob/main/static/index.js#L7

Then all there is left to do is host it on a CDN

view this post on Zulip Anton (Jul 31 2024 at 14:50):

Super cool @Bram!

view this post on Zulip Luke Boswell (Jul 31 2024 at 22:43):

@Ryan Barth did some exploration into platformless roc, which may be relevant. Also he and I have been working on the build-host PR which refactors the build pipeline a bit and I think will make it easier if we need to change anything.

view this post on Zulip Luke Boswell (Jul 31 2024 at 22:44):

I've got builtins and and things happy with zig 13 on a branch -- one of the main motivations for that was that I think it unlocks linking with .wasm hosts.

view this post on Zulip Luke Boswell (Jul 31 2024 at 22:47):

Just thinking about what we might be able to do to experiment with these ideas and test them out.

I imagine we need a platform with a release, that has a binary for a wasm host in it.

Then when the roc compiler (that is compiled to wasm and running in the browser) loads an app that uses a wasm supported platform, we fetch the platform files which include the prebuilt wasm host. Compile the app to wasm and then link the app and wasm together so we can run the combined thing in the browser.

view this post on Zulip Luke Boswell (Jul 31 2024 at 22:48):

We could possibly upgrade @Oskar Hahn's https://github.com/ostcar/roc-wasi-platform to zig-13 and I think that will give us a nice WASI host.

view this post on Zulip Luke Boswell (Jul 31 2024 at 22:49):

Or maybe now we have a working llvm 18/zig 13 compiler we could continue with https://github.com/lukewilliamboswell/roc-platform-template-wasi and see if we can get something minimal working there.

view this post on Zulip Luke Boswell (Jul 31 2024 at 22:54):

For reference the zig builtins source on this branch have been upgraded to zig 13.

view this post on Zulip Luke Boswell (Jul 31 2024 at 23:09):

So I guess what I'm thinking is we can isolate the different platform parts we need. Build something minimal and test/validate it works outside the browser.

view this post on Zulip Luke Boswell (Jul 31 2024 at 23:18):

If we build an end-2-end MVP now while those PR's are still WIP, we can probably have this working when the build-host and upgrade-llvm-zig PR's land. But it's also a bit more involved working on branches and digging into/understanding the build/linking pipeline.

So it would also be reasonable to wait for those changes before adding in more scope. Though it could take us a month or more before we see these changes in main with a latest release.

view this post on Zulip Bram (Jul 31 2024 at 23:22):

Sounds like a good plan @Luke Boswell , I am not sure how much I can be of help in the short term (upcoming weeks) as I am a bit busy, don’t know Zig yet, and have to read a bit into how platforms are implemented, but if anyone wants to do an experiment to make a wasm compatible host, I would gladly try to hook it up to my experiment and see if we can get a complete web based Roc editor going

view this post on Zulip Bram (Jul 31 2024 at 23:28):

One, maybe unrelated, thought I had about platforms, what if you want to interop with other languages? Let’s say some JavaScript library if you build a frontend app, in that case you would lose the security/ sandboxing like features that platforms offer? But not being able to interface with external code would also be quite limiting…

view this post on Zulip Luke Boswell (Jul 31 2024 at 23:33):

This is a slight tangent, I can't move the messages.

But in general the platform controls that interface. So the platform would be able to expose some kind of API to interop (or choose not to).

view this post on Zulip Luke Boswell (Jul 31 2024 at 23:34):

If you are building to WASM and then using javascript to load that into the browser, there is an additional interface there between the platform host and the javascript/wasm runtime which would need to be considered.

view this post on Zulip Luke Boswell (Aug 01 2024 at 00:57):

Ok, so I think I've got something you can work with. After writing all of ^^^ I couldn't help myself.

Just pushed an update for https://github.com/lukewilliamboswell/roc-platform-template-wasi

You can grab the compiled wasm binary here https://github.com/lukewilliamboswell/roc-platform-template-wasi/tree/main/zig-out/bin

view this post on Zulip Luke Boswell (Aug 01 2024 at 01:03):

I think the next step is linking. I can think of two options that might work;

  1. Use the roc --dev backend and Additive linker somehow -- we probably need to use the rebuild-platform branch and make some changes in crates/compiler/build/src/link.rs so it doesn't expect a hozt.zig and try to rebuild the host.
  2. Compile the roc app using e.g. $ roc build --target=wasm32 --no-link app.roc which gives us a app.wasm and then linking that with our host.wasm using some other tool. Maybe this can even be done in the browser?

Hopefully everything combines and we get a nice WASM binary that meets the requirements for WASI and we can run it using a runtime like wasmer, or load into the browser with a shim.

view this post on Zulip Bram (Aug 01 2024 at 11:23):

Luke Boswell said:

I think the next step is linking. I can think of two options that might work;

  1. Use the roc --dev backend and Additive linker somehow -- we probably need to use the rebuild-platform branch and make some changes in crates/compiler/build/src/link.rs so it doesn't expect a hozt.zig and try to rebuild the host.
  2. Compile the roc app using e.g. $ roc build --target=wasm32 --no-link app.roc which gives us a app.wasm and then linking that with our host.wasm using some other tool. Maybe this can even be done in the browser?

Hopefully everything combines and we get a nice WASM binary that meets the requirements for WASI and we can run it using a runtime like wasmer, or load into the browser with a shim.

Maybe a stupid question, but shouldn't the compiler compile the Zig-based host with the wasm target?

view this post on Zulip Luke Boswell (Aug 01 2024 at 23:29):

I'm definitely skipping over a lot of detail here, so I apologise for that. I've been working to try and understand WASM and platform things for a while and it's still not completely clear in my head.

I'm thinking about how it should work in future, with a couple of WIP PR's that should land soon and change the way platform hosts are built. Specifically, the platforms will be responsible for building their own binaries, which frees roc from that responsibility and leaves roc to just produce the "roc" binary and link it with the host.

In our case, this means that for any platform to support the playground it would need to provide a binary that can be run in the browser. WASM is the obvious choice here as the other formats (macos-aarch64, linux-x64 etc) wouldn't be very useful.

It makes sense for the WASM host to be targeting WASI as that means it would be runnable using any of the various runtimes (after linking with the app). We could build the app and run it using wasmer or wasmtime etc, or just upload the app.wasm file to any of the edge compute platforms like cloudfare or fastly. We can probably assume that there will be a WASI shim or runtime we can piggy-back off to do this in the browser too for our playground.

The other alternative, is that we have some kind of playground specific API that platform authors can choose to support. The platform author builds the host into freestanding WASM with this API, so it can be loaded into the playground environment. This may be the easier (or only feasible) way to do things.

My wasi platform template is an attempt at doing the first. It currently provides a host.wasm binary that targets WASI. Thought I'm not totally sure if we can even do things this way. I had a look yesterday at linking with the app manually and got an error about the wasm file no being relocatable. At the time I needed to get back to work, so I paused my exploration.

I think the way WASM is currently done in roc is that there must be a host.zig file, and roc will "rebuild the platform". The host is compiled to llvm bitcode I think, and then linked with the app. So it's not really a standalone host, and doesn't work with a URL platform that needs to provide a pre-built host.

view this post on Zulip Luke Boswell (Aug 03 2024 at 00:34):

I spent a lot of time yesterday looking at WASM hosts and linking. I don't think I made any progress unfortunately. I'll come back to this topic again at some point, but just wanted to let you know I'm stuck and will probably move onto to other priorities for now.

view this post on Zulip Richard Feldman (Aug 03 2024 at 00:41):

thanks for giving it a shot!

view this post on Zulip Luke Boswell (Aug 03 2024 at 08:52):

Great news... I managed to get it working!!! :tada:

https://github.com/lukewilliamboswell/roc-platform-template-wasi

I'm super stoked about this. I've included a build.sh script which steps through things manually, but from here I think there is a clear path to supporting WASI prebuilt-hosts.

view this post on Zulip Luke Boswell (Aug 03 2024 at 08:52):

$ wasmtime out.wasm
Roc loves WASI
Runtime: 0.223ms

view this post on Zulip Luke Boswell (Aug 03 2024 at 09:02):

And I think it consistency takes <100ms to build and link the app (with the prebuilt host). I'm having trouble measuring it tbh.

view this post on Zulip Bram (Sep 02 2024 at 16:06):

Hey @Luke Boswell great to see it working. I have been busy lately with work, and more importantly, my vacation :sunglasses:

I was wondering if there is any low-hanging fruit that I could help with?
What was the conclusion from the exploration of a platformless roc? Should we wait on build pipeline changes that potentially emerge from that exploration, or go ahead and try to hook wasm platform support into what we currently have now?

view this post on Zulip Luke Boswell (Sep 03 2024 at 01:07):

Gday Bram, good to see you back post-vacation.

It's difficult to talk about this topic because there are so many different parts involved, and I'm also learning as I explore things, so please ask for clarification if anything here isn't clear.

The answer depends on what we would like to achieve I think.

The longer term goal of a generic playground in the browser is achievable, but there is some blockers that need to be investigated. The first issue I can see is that we want the compiler to run inside the browser, and in that context we can't use LLVM or Zig (so the legacy linker and llvm backend aren't available). We should be able use the gen_wasm backend (and the wasm additive linker) like the wasm repl that is used on the website but this hasn't been done before, and so I think there is a bit of research and development here to do.

I spent some time exploring in this direction and was able to get a WASI version of the compiler that compiles a roc app using gen_wasm -- it's actually pretty simple to get working**. I can share it with you but I haven't made it a public repository because it's pretty hacky/proof of concept stage.

** by working, I mean it compiles and produces something... I don't know if it's actually doing things correctly as I haven't got a fully working example yet.

I ran into some complications around generating a WASI app and also potentially trying to compile a roc app that uses a platform and/or with multiple files -- I think I was trying to do too much at once. This is unlike the current wasm repl, which loads a single module and links with a simple host implementation (which we know works well).

I'm not sure what my issue was; it could be a bug in gen_wasm, or perhaps the way I was setting up the prebuilt WASM host (for additive linking), or perhaps in the way I was using the compiler to set things up.

Basically, I got as far as a WASI compiler that compiles a roc app (that was using a platform with a Task based API main: Task {} I32) to WASI, but the output binary wasn't valid. Using wam binary tools, wasm-validate app.wasm would complain about a leb128 encoding error. Brian helped me look into this, and his recommendation was to simplify things because there are currently too many variables that could be giving us grief. At this point, I've had to put this on the backburner to focus on other things -- but the next step here would be to pare things back and use a simpler platform, simpler host, and maybe generate a wasm32-unknown-unknown application.

If we want to build a simple playground that is is limited to a single module and hardcoded interface e.g. it only exposes a couple of functions to solve an Advent of Code puzzle or similar. This would be almost identical to the wasm repl setup we have now, and I think it would be much easier to achieve. I'm not sure howthis interacts with the LSP though.

If we are wanting to use the roc cli to compile WASM/WASI apps, and are not talking about a playground in the browser, then it's also much easier to do. The build-host PR should get us pretty close. We can use LLVM and Zig to link with a prebuilt host in the platform. The key thing I'm hoping to gain over the status quo is that the platform can be prebuilt and roc is just linking things together.

If you want a nice WASI platform, and are happy to let zig (or the host toolchain) drive the linking, it's quite easy to do today. Just build the app first roc build --target wasm32 --emit-llvm-ir and then use that to build the host.


Last updated: Jul 06 2025 at 12:14 UTC