notramo (Sep 29 2026 at 22:06): notramo (Sep 29 2026 at 22:06):I experimented with LLM-based “fuzzing”. What to do with the insane amount of bugs found? Open them one by one on GitHub, or dump them into a tar archive and upload it as attachment in one single ticket?
The bugs are validated, but the explanations may be low quality. Or just publish the proven snippets without the explanations?
Luke Boswell (Sep 29 2026 at 23:22):I'm curious ... how did you do it?
Luke Boswell (Sep 29 2026 at 23:23):Are they all the same bug repeated, or really different bugs?
notramo (Sep 29 2026 at 23:25):I'm curious ... how did you do it?
Nothing special, just trying different prompts and analysing the tool traces to see how the agents approach the problems, and how could be better guided towards exploring obscure corner cases.
Luke Boswell (Sep 29 2026 at 23:25):The way I have been doing things is one at a time, having an Agent "find the root cause" using a worktree of Roc so I can be sure it's a higher quality bug report. Would you have capacity to work through them like that one at a time?
I guess one at a time would be more helpful
notramo (Sep 29 2026 at 23:27):Are they all the same bug repeated, or really different bugs?
Yes, different bugs. I manually validated some of the examples.
Richard Feldman (Sep 29 2026 at 23:28):I appreciate lots of bug reports as long as they're real! :smile:
notramo (Sep 29 2026 at 23:28):The way I have been doing things is one at a time, having an Agent "find the root cause" using a worktree of Roc so I can be sure it's a higher quality bug report.
Unfortunately, I'm not familiar with the compiler source, so I can't check whether the agent actually found the root cause.
notramo (Sep 29 2026 at 23:28):I appreciate lots of bug reports as long as they're real! :smile:
Should I open them one by one? Or put in one ticket?
Richard Feldman (Sep 29 2026 at 23:28):github issues are fine, and honestly no need to find a root cause - steps to reproduce in a minimal example of the most helpful
Richard Feldman (Sep 29 2026 at 23:29):up to you as to how to group them!
notramo (Sep 29 2026 at 23:32):I'll open a bunch of them manually. I'll post the reproducer snippet and the outputs. If I should change something in the format or the method, let me know.
notramo (Sep 30 2026 at 01:40):I opened a few tickets. If these are useful, I'll open more.
Luke Boswell (Sep 30 2026 at 01:46):They look good :thumbs_up:
Luke Boswell (Sep 30 2026 at 03:24):One thought @notramo is using a ReleaseSafe or Debug build instead of a nightly... then the reports wont be segfault but you can likely provide a more targeted failure. It's not necessary or anything, but seeing that detail helps to categorize the bug.
Luke Boswell (Sep 30 2026 at 03:25):Knowing the nightly segfaults is helpful though too
Luke Boswell (Sep 30 2026 at 03:27):Also if you mention the source of the bug, like "This was found using AI assisted fuzzing" or something then it helps to understand if this is blocking someone's current work which would attract a more urgent priority. I'm sure all the bugs will get fixed quickly, but as we find more it may get to a point we just have to pick the most urgent and priority ones off first
notramo (Sep 30 2026 at 20:22):One thought notramo is using a ReleaseSafe or Debug build instead of a nightly... then the reports wont be segfault but you can likely provide a more targeted failure. It's not necessary or anything, but seeing that detail helps to categorize the bug.
Unfortunately, my machine don't have enough resources to compile a ReleaseSafe or Debug build. But I think it's not that necessary, because copy-pasting the snippet is ~3 seconds, and maintainers can check it with their own debug builds.
notramo (Sep 30 2026 at 20:24):Also if you mention the source of the bug, like "This was found using AI assisted fuzzing" or something then it helps to understand if this is blocking someone's current work which would attract a more urgent priority.
I'll add it from now on.
notramo (Oct 01 2026 at 00:11):Also if you mention the source of the bug, like "This was found using AI assisted fuzzing" or something then it helps to understand if this is blocking someone's current work which would attract a more urgent priority.
Is this really needed for every issue? I think it's better to write "This is blocking me." than "This is not blocking me."
notramo (Oct 01 2026 at 00:12):First, I often find bugs manually while coding, but can easily work around and leave a TODO to remove the workaround hack when fixed upstream. These would not get the LLM label, but are not blocking me.
notramo (Oct 01 2026 at 00:14):Second, I find many high-quality reports using LLMs, but also low quality corner cases too. Some of the high quality ones are causing silent logic errors or data loss, that are not obvious, and may be affecting somebody in production, despite not noticing it.
notramo (Oct 01 2026 at 00:15):These get the LLM label despite being important.
Luke Boswell (Oct 01 2026 at 00:16):I dont see the LLM thing as a negative
notramo (Oct 01 2026 at 00:16):Is it enough to write this at the very low priority, corner cases like "parser crashes with this weird syntax that nobody would write during normal coding", or also apply to the very high quality ones?
notramo (Oct 01 2026 at 00:16):I dont see the LLM thing as a negative
Me neither. It's just noise, because don't necessarily reflects the quality or urgency of the issue.
Luke Boswell (Oct 01 2026 at 00:18):It's just helpful I think to know how the bug was discovered. It doesnt diminish the bug at all.
If someone is working on a project or using Roc in production and there is a regression or something its helpful to know too.
It's just a LLM or fuzzer style bug typically is less likely to be something someone has run into and is currently blocking them.
Luke Boswell (Oct 01 2026 at 00:19):Its also no necessary to include that or anything. You just asked for ideas that might help
notramo (Oct 01 2026 at 00:20):I try to guide the LLMs towards realistic bugs that can appear during actual programming scenarios. It turns out they are quite good at finding realistic bugs when it's explicitly stated as a goal.
Richard Feldman (Oct 01 2026 at 01:40):yeah "this is blocking me" is the useful signal :smile:
Richard Feldman (Oct 01 2026 at 01:40):for prioritization
notramo (Oct 02 2026 at 14:57):Should I report when roc check hangs too? E.g. some infinite recursion. Should the comptime evaluation have a timeout (fuel limit)? Or is this expected?
Richard Feldman (Oct 02 2026 at 15:16):short answer is that we don't need a report for it
Steve Howell (Oct 02 2026 at 22:45):@notramo See https://github.com/roc-lang/roc/issues/11334 for a little more context. (That's an issue I filed for essentially the same thing you're saying, unless I misunderstand you.)
notramo (Oct 02 2026 at 22:47):I thought there was some kind of limit which failed to trigger, but if it's the intended behavior, it's okay for me.
notramo (Oct 02 2026 at 22:57):What about hangs where it's not caused by comptime eval? For example, the type checker is stuck in a loop?
notramo (Oct 02 2026 at 22:59):@Richard Feldman, should I report those, or not?
Richard Feldman (Oct 02 2026 at 23:16):yeah! That should never happen
Zak Kohler (Oct 04 2026 at 18:42):I know zig has an official stance, closely following what sqlite did.
Steve Howell (Oct 04 2026 at 21:33):Zak Kohler said:
- Then find all bugs that appear from 100% ranch coverage tests [...]
Does ranch coverage find missing fields?
Zak Kohler (Oct 05 2026 at 15:55):Steve Howell said:
Does ranch coverage find missing fields?
Depends on the cow semantics being followed.
notramo (Oct 13 2026 at 22:09):@Richard Feldman, I noticed that the LLM agents often roll their own fuzzers when I instruct them to cause crashes.
notramo (Oct 13 2026 at 22:10):And they are quite successful with these custom fuzzers.
notramo (Oct 13 2026 at 22:11):This, got me curios about whether the current fuzzing architecture could be improved in even more places, not just https://github.com/roc-lang/roc/issues/11918
notramo (Oct 13 2026 at 22:12):I ran some prompts to investigate the fuzzing architecture, similar to this:
notramo (Oct 13 2026 at 22:12):There have been many compiler crashes reported and fixed since September 30. Why did the fuzzer miss them? What are the areas where the fuzzer could be improved? Investigate based on these reports, the fix commits, and the current fuzzing architecture.
notramo (Oct 13 2026 at 22:14):The LLMs produced some interesting suggestions, that seemed like good observations, however, since I'm not familiar with the compiler, I could not assess the quality of the reports.
notramo (Oct 13 2026 at 22:15):Could somebody from the core team run this prompt, and see whether it produces good suggestions?
notramo (Oct 13 2026 at 22:20):If the suggestions are useful, should I bother opening the crash reports as I did for the past few weeks, or most of them would be catched by the fuzzer if the fuzzing setup is improved?
Anton (Oct 14 2026 at 15:05):Can you make an issue for this @notramo? That way we can get to it when we have the time.
notramo (Oct 14 2026 at 18:31):https://github.com/roc-lang/roc/issues/12289
Last updated: Oct 15 2026 at 17:56 UTC