Stream: ideas

Topic: LLM assistance reviewing old GH Issues

Luke Boswell (Jan 17 2026 at 01:14):

nandi said:

I was tempted to make a suggestion to close our over 1000 open issues and then have a channel/topic where people discuss the ones they want to reopen

I had a crazy idea the other day I haven't got around to trailing yet. But I thought about giving some of the (open source) LLM's an instruction to go through each GH Issue one by one and do some exploration against the new compiler and write up a status of the GH issue in one sentence, categorize, and then format into a summary table etc.

I thought it might be useful to identify clearly outdated issues that are no longer relevant -- and save time by being able to go straight to these issues and close them. This kind of work I have found the LLM's to be generally very helpful, you can ask it to make a small reproduction or test and if it can reproduce the bug or feature.

The biggest risk here is false positives/negatives -- but I think that is acceptable because we aren't asking the agents to auto close issues or anything, just summarise the status.

Luke Boswell (Jan 17 2026 at 01:20):

I also suspect this idea is kind of doomed with two completely different and subtle syntaxes between the compilers. So it's hard for an LLM to distinguish between n issue that was intended for the old Rust compiler or one for the new Zig one. Maybe we could triage based on date or other heuristics.

As I mentioned it's a half baked idea that needs work or exploration before it could work I think.

nandi (Jan 17 2026 at 01:25):

Luke Boswell said:

I also suspect this idea is kind of doomed with two completely different and subtle syntaxes between the compilers.

I think we might be able to create a skill with examples from each compiler and see how well it does telling them apart

nandi (Jan 17 2026 at 01:27):

Luke Boswell said:

The biggest risk here is false positives/negatives -- but I think that is acceptable because we aren't asking the agents to auto close issues or anything, just summarise the status.

The downside to that is we still have to manually close hundred of tickets potentially

Luke Boswell (Jan 17 2026 at 01:28):

Yeah I kind of put this idea into the bucket of "would be cool to explore someday" -- but I'm currently focussing my energy on roc-ray and fixing priority roc bugs in the lead up to our online meetup. I'd like to have some cool things to show everyone :smiley:

nandi (Jan 17 2026 at 01:29):

Luke Boswell said:

I thought it might be useful to identify clearly outdated issues that are no longer relevant

Having a markdown file breaking down issues by category/relevancy could be useful indeed

Luke Boswell (Jan 17 2026 at 01:29):

The downside to that is we still have to manually close hundred of tickets potentially

Nah, that's easy I think. I don't mind doing that.

Not that long ago there was someone who was looking at the issues, like 30 a day or something and just dropped a comment on each. It was super helpful and we closed out hundreds of issues in like no time.

Richard Feldman (Jan 17 2026 at 01:53):

I do think we can safely close all the issues prior to the first commit of the new compiler

Sky Rose (Jan 19 2026 at 22:07):

Hey I'm the one who reviewed all the issues last time. That thread is here: #contributing > Cleaning up GitHub Issues @ 💬 I think I averaged about a minute per issue, with high variance. There were a lot of issues that I closed in 10 seconds and a few that I put actual investigation time into. I closed about 200 issues. I bet you could close more than that this time.

For bugs (400 issues), most old bugs probably aren't relevant anymore, but most have reproductions that can be tested, and some are probably still relevant (e.g. https://github.com/roc-lang/roc/issues/1670 ) and there were some that I thought were obviously outdated but had to be reopened: e.g. (https://github.com/roc-lang/roc/issues/5909 ). For everything else, there's a big mix of ideas, some of which could still be relevant (e.g. https://github.com/roc-lang/roc/issues/6076 ) and some that are definitely not (everything about abilities).

I do recommend having a human review them. Deciding which issues to leave open is a way to decide what's important and worth doing in the future. That's a management decisions for humans, not an execution tasks for robots. Even though it's boring.

I don't think having an LLM summarize the issues would help much. Most issues are short, the hard part is loading all the context into your brain to make a decision on what to do. Reading 1 summary instead of 2 comments won't make it faster and will just obscure the facts. The best place for LLMs in this would be in translating old bug reproductions into modern Roc.

If I were doing this again (I'm not volunteering, I don't think I have enough context on recent developments to make any decisions), here's how I'd do it:

• First, search for issues about old features you know aren't relevant anymore (e.g. abilities), and for each one, skim the issue to verify it's about the obsolete topic, and close it. This is probably an easy win for a hundred issues you won't have to look at later.
• Then go through the bugs (400 old issues). This is the part that's most likely to be outdated and also the part I looked at the least a year ago:
  - If it's complex or intermittent, close it, it's probably deeply tied to the old compiler and not worth the time to investigate.
  - If there's an easy reproduction, try it, if it still exists, comment that it does (and a modern Roc version of the reproduction) and leave it open. If it doesn't reproduce, comment and close.
  - Otherwise, use your judgement on whether it's likely to still be around or worth the time to investigate if we do a big bug push later. Lean towards closing them.
• Then do enhancements (263 old issues). There are two kinds of enhancement issues:
  - "I have a big new idea". These will never be outdated really, but can still be pruned if they don't align with the current vision for Roc, or if you don't think that the idea is worth spending time thinking about anymore.
  - "This feature should be a little better". These can be outdated if the feature is no longer relevant
• Then everything else (~300 issues).

Anton (Jan 20 2026 at 09:04):

Thanks for the write-up @Sky Rose :)

Rick Hull (Jan 20 2026 at 21:12):

did we use gh for this? this tool has been on my radar for a while, but I've only started using it in earnest in the last week. LLMs tend to understand its capabilities natively

Luke Boswell (Jan 21 2026 at 01:33):

No I just manually went through them all. Only took a few hours to close ~ 900

Last updated: Jun 16 2026 at 16:19 UTC