Eli Dowling (Feb 03 2025 at 17:51): Eli Dowling (Feb 03 2025 at 17:51):I've had some more time to work on the tree-sitter grammar, but I have a few open questions.
Here is my current todo list btw:
If some of this is still largely in flux, I can finish testing what I've done and get a release out, that at least gets it up to speed with all the current changes.
Thhoughts and suggestions welcome.
Anton (Feb 03 2025 at 18:23):I would not worry about features like SD and for loop that are not currently in Roc, with the big rewrite these will take a while
Eli Dowling (Feb 03 2025 at 18:24):Sure, but while I'm making changes it'd be nice to get ahead on anything that's pretty locked in :)
Anton (Feb 03 2025 at 18:26):For the list destructuring changes: https://github.com/roc-lang/roc/issues/7091
Anton (Feb 03 2025 at 18:32):How sure are we about:
I think these all could change somewhat after getting serious usage.
Eli Dowling (Feb 03 2025 at 18:35):Sounds good, I might rough in at least the for loop, and leave the rest for now. They won't be a particularly big job to do later anyway.
Sam Mohr (Feb 03 2025 at 18:54):Here's the doc for for and while, I'd be surprised if they change once we implement them: https://docs.google.com/document/d/1Ly5Cp_Z7dY8KLQkkDYZlGCldxQj4jLzZ0vIeB-F8lJI/edit?tab=t.0
Sam Mohr (Feb 03 2025 at 18:55):Abilities are very likely to be removed IMO, if you care about removing code that won't be used anymore
Eli Dowling (Feb 03 2025 at 18:56):Yup, but I can't drop them just yet. Don't want to break everyone. They can exist happily, until we totally deprecate them.
Sam Mohr (Feb 03 2025 at 18:56):Custom types will look (almost?) exactly like opaque aliases at the type def site
Sam Mohr (Feb 03 2025 at 18:57):Their use in other types will also be the same
Sam Mohr (Feb 03 2025 at 18:57):In expressions, that's still in flux
Sam Mohr (Feb 03 2025 at 18:57):The new autoderive syntax is also in flux
Sam Mohr (Feb 03 2025 at 18:58):What we've talked about a whole lot and would be unlikely to change IMO is the foo.(local_func)(args) and foo.(only_takes_foo_func) "method" syntax
Sam Mohr (Feb 03 2025 at 18:59):Lists and records now have spreads
Eli Dowling (Feb 03 2025 at 18:59):Oh?
Sam Mohr (Feb 03 2025 at 19:00):Which also applies to record types and types that are written as lists, AKA tag unions
Sam Mohr (Feb 03 2025 at 19:00):Instead of using { foo : Str }ext, we're doing { foo : Str, ..ext } (and { foo : Str, .. } for open records)
Sam Mohr (Feb 03 2025 at 19:02):Let me write a few things down
Sam Mohr (Feb 03 2025 at 19:04):Types:
{ list of "name: type", optionally followed by ".." (which is optionally followed by another type for the extension) }List Str, still the exact same[ list of "Name payload-type-1 payload-type-2", optionally followed by ".." (which is optionally followed by another type for the extension) ] Sam Mohr (Feb 03 2025 at 19:07):Patterns:
{ x, y: 3 } is a closed record{ x, y, .. } is an open record{ x, y, ..{ other, other2 } } is a capture of the rest of the fields in the record besides x and y[a, b, c] or [a, b, .., d] or [a, b, c, ..d] (not .. as d) Sam Mohr (Feb 03 2025 at 19:08):Expressions:
{ (field | (field: <expr>) | ..<expr>)* }[ (<expr> | ..<expr of type list>)* ] Sam Mohr (Feb 03 2025 at 19:09):Optional types and values are also being removed
Sam Mohr (Feb 03 2025 at 19:09):As well as module params
Eli Dowling (Feb 07 2025 at 09:54):@Sam Mohr I don't understand this at all:
{ x, y, ..{ other, other2 } }
why is there a nested record here?
Eli Dowling (Feb 07 2025 at 09:54):This makes sense to me: { x, y, .. other }
Sam Mohr (Feb 07 2025 at 09:55):I guess we could limit the spread sub pattern to the second one, which I'd agree would be much more sane
Eli Dowling (Feb 07 2025 at 09:55):Would this:{ x, y, ..{ other, other2 } } not be the same as { x, y, other, other2, ..}?
Sam Mohr (Feb 07 2025 at 09:55):It's just that we have been employing a philosophy of not disallowing specific combinations of syntax
Eli Dowling (Feb 07 2025 at 09:56):sure, I'm just looking to make sure I understand what's going on, that's all
Sam Mohr (Feb 07 2025 at 09:56):You're right that it's a worse version of the ident based spread
Eli Dowling (Feb 07 2025 at 10:00):How soon do you foresee .. and ..rest coming to the compiler? just checking to see if I should remove the as syntax or not
Eli Dowling (Feb 07 2025 at 10:02):or is that likely to be post compiler- rewrite?
Eli Dowling (Feb 07 2025 at 10:04):if so, I'll do a release with what I've currently done, covering all the changes that are actually in the compiler. Then I'll keep adding the proposed stuff too
Sam Mohr (Feb 07 2025 at 10:32):That's post compiler rewrite, there's less than a 1% chance that we make any big changes to the Rust-based Roc compiler at this point
Eli Dowling (Feb 07 2025 at 11:13):Okay, sounds good. I'll do two releases then.
Brendan Hansknecht (Feb 07 2025 at 16:42):Sam Mohr said:
That's post compiler rewrite, there's less than a 1% chance that we make any big changes to the Rust-based Roc compiler at this point
:thinking: I guess it depends on what you call big. I think we are definitely folding in a number of things, but many are not syntax
Sam Mohr (Feb 07 2025 at 17:25):Yes, I should be more clear: we are rewriting the compiler, and trying to start with a minimal version of the compiler to prove that everything works. After that, we aren't trying to implement to parity with the Rust-based compiler, but rather only implement syntax for the new planned behavior
Sam Mohr (Feb 07 2025 at 17:25):So when I said "post-rewrite", I was thinking "post-get it working", which was confusing
Eli Dowling (Feb 07 2025 at 18:33):I got some more time to work on changes and I have now implemented all the syntax @Sam Mohr mentioned. I've also removed some features being deprecated like space based function calling.... A sad loss for sure.
Eli Dowling (Feb 07 2025 at 18:34):Basically now it's just building out the corpus to find edge cases
jan kili (Feb 07 2025 at 19:33):Eli Dowling said:
I got some more time to work on changes and I have now implemented all the syntax Sam Mohr mentioned. I've also removed some features being deprecated like space based function calling.... A sad loss for sure.
@Luke Boswell @Sam Mohr Is this the first step in Roc's ecosystem to un-support WSA?
jan kili (Feb 07 2025 at 19:34):I remember y'all mentioning a long deprecation for it.
Sam Mohr (Feb 07 2025 at 19:35):I'd say so, yes. Technically, it's basically a parsing local change, so it wouldn't be a giant change to add/remove it, but we seem to be trying to only implement the features we'll see in Roc 0.1.0
Sam Mohr (Feb 07 2025 at 19:35):However, you're right that we said we'd support it and are now moving away from that promise without discussion
Sam Mohr (Feb 07 2025 at 19:36):I think this is a gap between what we think the future is for Roc that we are trying to plaster over with politeness, unsuccessfully
jan kili (Feb 07 2025 at 19:37):Thanks for that perspective! I'll start a separate topic for it.
Sam Mohr (Feb 07 2025 at 19:37):That's good
Eli Dowling (Feb 07 2025 at 19:45):Just to be clear @JanCVanB:
The upcoming release making the grammar up to date with the existing rust compiler will not remove WSA.
I expect though, that the compiler rewrite will only contain one syntax, having both would seem highly unusual to me...
Sam Mohr (Feb 07 2025 at 19:46):Yes, I think it's extremely unlikely that we'd have two function call syntaxes, and PNC seems to work with static dispatch in a way that WSA doesn't
Sam Mohr (Feb 07 2025 at 19:47):We're trying to discuss what to do in the #contributing > 🚨 Whitespace application and whitespace - breaking change! 🚨 thread
jan kili (Feb 07 2025 at 20:04):(moved to #ideas > reflecting on static dispatch)
jan kili (Feb 18 2025 at 16:35):Thanks for this tree sitter, @Eli Dowling! I'm switching from VSCode to Helix, and with this Roc support it feels like it could be an ideal daily driver for me. I'm working on a coloring theme as my entrypoint into customizing/contributing to editor support tools.
I think I see some gaps in our grammar coverage, but I might just be using it wrong. Does it look accurate to you that these gray portions are untokenized/unscoped?
using tree-sitter-roc on master:
Screenshot_20250218_084547.png
using tree-sitter-roc on new-syntax-old-compiler: (seems unchanged)
Screenshot_20250218_091553.png
jan kili (Feb 18 2025 at 17:01):According to Helix docs, my question might be about our coverage of only the highlights query, but idk how interconnected that query is with the rest of the tree sitter.
jan kili (Feb 18 2025 at 17:14):After reading queries/highlights.scm, it does seem mostly powered by the syntax tree. It's super readable, and now I want to learn Scheme syntax to understand it fully. While I expect I could debug the grey gaps (above) in time, I'm still curious to hear your assessment of them.
Eli Dowling (Feb 18 2025 at 20:33):Hey @JanCVanB my holiday is over this week. So I expect to get the updated grammar out in the next week which will fix all the issues with the existing syntax.
I know there is heaps of errors right now, sorry about that :sweat_smile:.
Eli Dowling (Feb 18 2025 at 20:34):I have been dabbling a little bit with the upcoming helix plugin system (which uses a scheme dialect called steel as it's language)
Eli Dowling (Feb 18 2025 at 20:35):Sadly steel has very limited documentation and the plugin system has been pretty slow to develop
Eli Dowling (Feb 18 2025 at 20:37):It's a real shame because I don't use helix so much these days. In my day job, not using AI tools is just so much slower it's not worth it.
I've been looking to try out zed though, as my main editor is currently cursor.
For my personal stuff though, it's actually hard and so AI is basically useless :joy:
jan kili (Feb 18 2025 at 20:45):Woohoo! No rush.
jan kili (Feb 18 2025 at 20:47):The only token distinction I'm unsure whether you plan to support is when Nevermind, I see you intend to support that distinction.: is used in types vs in records - I'd like to color them differently.
jan kili (Feb 18 2025 at 20:48):Is the gray text above the result of syntax changes or of a small corpus? I'm happy to provide corpus/test cases.
Richard Feldman (Feb 18 2025 at 22:07):@Eli Dowling welcome back! FYI we're moving to a context-free grammar (not kidding!) that doesn't do significant whitespace
Richard Feldman (Feb 19 2025 at 00:46):see #ideas > braces syntax and #ideas > Using parens for types
Joshua Warner (Feb 19 2025 at 00:50):Technically... it does do significant _whitespace_, but not significant indentation nor significant newlines
Anthony Bullard (Feb 19 2025 at 14:57):Indentation significance is the only thing in the grammar that was forcing Roc's grammar to not be context-free
Eli Dowling (Feb 21 2025 at 09:35):@Richard Feldman Thanks :)
I've been following that, with admittedly quite mixed feelings.
As someone who uses a variety of programming languages regularly (php,js,rust,ocaml) and has tried/used many more. I'm pretty strongly in favour of whitespace sensitivity and WSA.
(technically ocaml isn't WS sensitive, but I preferred F# which is almost the same but with whitespace sensitivity)
Eli Dowling (Feb 21 2025 at 09:40):I saw a lot of discussion about things that make code easier to write:
. autocomplete,
pasting,
static dispatch aiding function discovery.
I love all that too, but I actually spend the majority of my time, reading and making small edits/refactors to code.
I think PNC and braces mostly add noise and annoyance in those cases.
Still, if roc can deliver in other ways, I'll get over it :sweat_smile:
jan kili (Mar 09 2025 at 22:00):Eli Dowling said:
Hey folks, I know it's been a while, but I finally got around to finishing the update to the TS syntax for roc.
Should support all syntax in the existing rust compiler. (except WSA)
:tada: Thank you so much for updating it!
It's not working for me in Helix, so where would you like to discuss debugging/feedback - here or a new thread or DMs or GitHub?
Eli Dowling (Mar 09 2025 at 22:11):Here is fine, may as well keep it visible for others!
Send me a copy of your helix log after opening a roc file.
Eli Dowling (Mar 09 2025 at 22:12):Also @JanCVanB if you can explain what's not working a little more that'd be great. Is it just totally no syntax highlighting?
Eli Dowling (Mar 09 2025 at 22:12):Also, what was your procedure for updating?
Sam Mohr (Mar 09 2025 at 22:34):@JanCVanB did you copy over the query files?
jan kili (Mar 09 2025 at 22:41):Eli Dowling said:
Also, what was your procedure for updating?
~/.config/helix/languages.tomlhx --grammar fetch and build~/.config/helix/runtime/queries/roc to ~/.config/helix/runtime/queries/roc-februaryrepo/queries to ~/.config/helix/runtime/queries/rocWith any Helix theme, all text in .roc files is the ui.text color.
Eli Dowling (Mar 09 2025 at 22:43):That all sounds reasonable to me, and your log output?
jan kili (Mar 09 2025 at 22:47):lol fixed!
~/_code/roc-lang/basic-cli/examples$ ls ~/.config/helix/runtime/queries/roc
highlights.scm indents.scm injections.scm locals.scm queries tags.scm textobjects.sc
Nested dirs. I must have messed up step 5.
Eli Dowling (Mar 09 2025 at 22:50):All, good. let me know if you notice any edge cases.
I didn't know exactly what projects had what level of syntax newness so the corpus is pretty small.
jan kili (Mar 09 2025 at 23:05):Here are some of the basic-cli examples with the theme:
inherits = "dracula"
"ui.text" = "grey"
Screenshot_20250309_170300.png
jan kili (Mar 09 2025 at 23:06):I'll resume my work on a set of themes that utilize all the highlights you've exposed!
jan kili (Mar 09 2025 at 23:08):I wonder why args in the top-left isn't highlighted but first & last in the bottom-right are.
Eli Dowling (Mar 09 2025 at 23:09):some stuff may not have more specific highlighting queries. Though I'm surprised the modules don't have a query, they certainly used to
Eli Dowling (Mar 09 2025 at 23:10):I'll check if I have the same issue with operators too
Eli Dowling (Mar 09 2025 at 23:11):You can use the tree sitter commands in helix to check the parse tree to show what is what
jan kili (Mar 09 2025 at 23:15):Whoa cool, and such variety here!
Screenshot_20250309_171437.png
Eli Dowling (Mar 09 2025 at 23:18):Weeel that explains things :sweat_smile:
Looks like I need to add variable assignment after question mark postfix to the test corpus
Eli Dowling (Mar 09 2025 at 23:18):Job for tomorrow
jan kili (Mar 10 2025 at 16:19)::smiley_cat: The bottom two changed with your updates:
Screenshot_20250310_101744.png
Looking good now?
Eli Dowling (Mar 10 2025 at 16:26):all looks good to me :)
jan kili (Mar 10 2025 at 16:29):https://media2.giphy.com/media/ZcGStMD6r07de/giphy.gif?cid=c623cb355kuj6iwzsxzd2t4sl2c2bwz14a1eu20qcg2c8e7g&ep=v1_gifs_search&rid=giphy.gif&ct=g
It's coloring time.
Eli Dowling (Mar 10 2025 at 16:31):hahah, I'm curious to see what you come up with.
I personally changed my theme to have a colour for modules.
Oh and if we wanted to we could probably improve the scopes to make exports have the right colour. eg functions have correct function colouring.
Eli Dowling (Mar 10 2025 at 16:31):sorry locals.scm not scopes.scm
jan kili (Mar 10 2025 at 17:25):My ongoing feedback seems sufficiently unrelated to this topic's original discussion and audience, so I'll start a new topic. This one seems resolvable!
Last updated: Jul 06 2025 at 12:14 UTC