Stream: ideas

Topic: Bundle the language server in the CLI

Isaac Van Doren (Jun 09 2025 at 02:52):

What if we included the language server directly in the compiler binary under the roc lsp command? That way there's only one binary to install, you automatically have the language server ready to go, and the language server version always matches that of the cli. This seems like a really nice user experience to me

Isaac Van Doren (Jun 09 2025 at 02:53):

After searching, Deno already does this

Hannes (Jun 09 2025 at 03:26):

I know Richard has said that he wanted to avoid doing this because he didn't want to block LSP improvements behind a full compiler release.

I think that's a good idea when releases are slow, for example the Gleam compiler is getting new LSP features in every release, but I don't get them until my package manager updates Gleam and I can be bothered to upgrade each project to use the new Gleam version. Roc isn't at that stable stage yet, so it might not be a bad idea to bundle them for now. :+1:

Richard Feldman (Jun 09 2025 at 17:09):

yeah I want to do this like roc glue

Richard Feldman (Jun 09 2025 at 17:10):

where instead of roc lsp it's like roc tooling https://example.com/roc-lsp/abcdfg123a.tar.gz

Richard Feldman (Jun 09 2025 at 17:11):

so you provide a roc script that can be lsp or whatever else - we just expose to it (much like roc glue) with all the inputs it needs to do its job

Richard Feldman (Jun 09 2025 at 17:12):

so that way whoever is implementing the language server extension for a given editor can use whatever lsp.roc (or whatever) along with the user's local roc binary, and those can be independent

Richard Feldman (Jun 09 2025 at 17:14):

also, cards on the table, some of my design goals for the roc binary include:

• it is never coupled to JSON (e.g. no roc.json file ever), or for that matter other configuration file formats like TOML
• it is never coupled to LSP

Isaac Van Doren (Jun 09 2025 at 18:55):

Hey that’s a cool idea!

Isaac Van Doren (Jun 09 2025 at 18:57):

It seems like the interface that would need to be exposed by the compiler to support building a rich LSP could be large. Having to deal with maintaining that interface and not breaking it all the time sounds like it could be difficult.

Isaac Van Doren (Jun 09 2025 at 18:58):

That’s one advantage of embedding the language server, then you don’t have to expose and maintain any kind of public interface like that

Isaac Van Doren (Jun 09 2025 at 18:58):

But I totally see the appeal of keeping json and lsp out of the compiler

Isaac Van Doren (Jun 09 2025 at 18:59):

And supporting other interesting tooling to be created

Richard Feldman (Jun 09 2025 at 19:07):

I don't think it's actually much more difficult honestly

Richard Feldman (Jun 09 2025 at 19:08):

for example, I think it's fine if we do something like exposing essentially all the functions the language server wants access to, and then the .roc file's job is basically just to run a little server and call those functions and send the results as JSON across the server

Isaac Van Doren (Jun 09 2025 at 19:10):

I'm more thinking of difficulty in terms of dealing with backwards compatibility and breaking changes. I.e. if something changes about how the compiler works internally that impacts the interface and now we have to worry about either breaking all of the tooling built on those APIs or somehow maintaining the original interface

Isaac Van Doren (Jun 09 2025 at 19:11):

Whereas if we only exposed an LSP, it would be the only client that we'd need to update and their wouldn't be an exposed API

Isaac Van Doren (Jun 09 2025 at 19:11):

It does sound worth it to have the ability for non-lsp tooling to be built though so I'm not saying this is a reason to not go with that approach

Richard Feldman (Jun 09 2025 at 19:14):

ah I see - I think the same consideration is true of glue though

Isaac Van Doren (Jun 09 2025 at 19:19):

Yes for sure

Brendan Hansknecht (Jun 09 2025 at 22:10):

I honestly think the right way to do this is to expose some sort of sexpr like API. Just give the roc script the raw data in a flexible form.

Brendan Hansknecht (Jun 09 2025 at 22:10):

This way the API is simple but exposes a lot

Brendan Hansknecht (Jun 09 2025 at 22:11):

Than leave the roc script to do whatever it wants

Richard Feldman (Jun 10 2025 at 00:26):

but it also breaks every time we change our internal representation :sweat_smile:

Luke Boswell (Jun 10 2025 at 00:39):

Could we provide a roc library that wraps that sexpr API and keep it in sync with our internal representation?

It would be useful for glue and other code generation things, LSP etc.

The alternative is that the platform abstraction we provide for tooling scripts would be need to be more detailed. Like instead of just exposing nodes of a few types, we would need to expose a representation for expressions, statements, types etc.

Luke Boswell (Jun 10 2025 at 00:49):

Also, we're pretty much there already with the sexpr thing, since we are using that for our snapshot tests.

The Parser SExpr's feel pretty mature and I could imagine using something like that to code-gen roc source. Here's the current snapshot for hello-world.

(file (1:1-5:42)
    (app (1:1-1:57)
        (provides (1:6-1:12) (exposed_item (lower_ident "main!")))
        (record_field (1:15-1:57)
            "pf"
            (string (1:28-1:55) (string_part (1:29-1:54) "../basic-cli/platform.roc")))
        (packages (1:13-1:57)
            (record_field (1:15-1:57)
                "pf"
                (string (1:28-1:55) (string_part (1:29-1:54) "../basic-cli/platform.roc")))))
    (statements
        (import (3:1-3:17) ".Stdout" (qualifier "pf"))
        (decl (5:1-5:42)
            (ident (5:1-5:6) "main!")
            (lambda (5:9-5:42)
                (args (underscore))
                (apply (5:13-5:42)
                    (ident (5:13-5:25) "Stdout" ".line!")
                    (string (5:26-5:41) (string_part (5:27-5:40) "Hello, world!")))))))

So I imagine a roc library that generates the Parser sexpr nodes and provides some type safety etc, so your not writing raw strings for "file", "statements", "apply" etc.

I think once we have the basics of an interpreter for running simple roc apps it will be easy to try out the simple sexpr API for roc tooling scripts.

Brendan Hansknecht (Jun 10 2025 at 01:34):

Yeah, I want to essentially expose what Luke shared there

Brendan Hansknecht (Jun 10 2025 at 01:34):

And yes, if roc updates it make break the lsp, but that is correct behaviour. If the lsp cares about specific mode types and roc changes nodes, that means that the language itself is actually changed....so the lsp needs to update.

Brendan Hansknecht (Jun 10 2025 at 01:35):

In practice, I expect things to be mostly stable.

Brendan Hansknecht (Jun 10 2025 at 01:36):

If the roc compiler understanding of roc the language changes, surely the lsp understanding of roc the language must change too, right?

Kiryl Dziamura (Jul 14 2025 at 08:57):

1. sexprs don't have newlines and comments, right? How would it work for codegen then? I assumed tooling provides complete information about the file.
2. sexpr means parsing them on the client side. So it's a separate parser for it?
3. roc would provide roc types for such scripts? Would tooling be a special case of a platform? pf: platform tooling?
4. do you think roc experimental-tooling would be a good name for now?

Anton (Jul 14 2025 at 12:17):

1. codegen does not require comments or newlines.

Anton (Jul 14 2025 at 12:18):

4. Sounds good, while we're figuring things out

Anton (Jul 14 2025 at 12:23):

I assumed tooling provides complete information about the file.

We can provide the information as people request it (on zulip), I don't think we need to provide everything.

Last updated: Jun 16 2026 at 16:19 UTC