Steve Howell (Oct 26 2026 at 12:02): Steve Howell (Oct 26 2026 at 12:02):notramo said:
So with everything that I build with zig, I'm kinda on my own.
Is it about questions regarding Zig or OS dev? Or design advice? Or project contributions?
I filed an issue with a clear reproducer for a clear problem, and I clearly disclosed it was done by an LLM, and they auto-closed the issue with no gratitude. I understand it's their policy and they can do whatever they want, but I think it's a bad policy.
notramo (Oct 26 2026 at 12:05):It's not that bad, because LLMs tend to create low-quality issue reports. I appreciate the no-LLM policy of the Zig project.
notramo (Oct 26 2026 at 12:06):If you have some actual bugs, you can always file it manually. It takes at most 5 minutes and the result is a much higher quality report.
notramo (Oct 26 2026 at 12:07):I think it does not matter that an LLM found it originally, if you understand the reproducer, and you test it manually.
Steve Howell (Oct 26 2026 at 12:10):notramo said:
If you have some actual bugs, you can always file it manually. It takes at most 5 minutes and the result is a much higher quality report.
I don't know you improve it manually, though. The LLM got the problem down to a 15-line reproducer even though the original bug was for a much longer file. The bug was that their stack traces got the wrong line numbers, which is kinda amateur. I would show you the issue to show you how clear it was, but they nuked it.
notramo (Oct 26 2026 at 12:12):I noticed that LLMs tend to stop at a ~20 line reproducer, but then I instruct to shrink it even further, and they often get it down to ~5 lines. At least in the case of Roc snippets.
notramo (Oct 26 2026 at 12:13):The bug was that their stack traces got the wrong line numbers, which is kinda amateur.
You could file this with a human-written description. It takes about ~2 minutes, and the'll accept it.
Steve Howell (Oct 26 2026 at 12:15):I still have the draft, actually: http://143.244.172.148:9100/notes/zig-issue-generic-debug-lines.md
I was wrong about ~20 lines. It's a two-line function with a two-line test.
notramo (Oct 26 2026 at 12:31):What about this cleaned-up version?
Title: Self-hosted backend stores incorrect line numbers in debug info.
When building with self-hosted backend, a generic function's prologue/epilogue rows contain line numbers past the end of its file.
It causes bugs in coverage tools.
System info:
- 0.17.0
- x86_64 Linux
- self-hosted x86_64 backend
- `-fllvm`
`r2.zig` (11 lines)
```zig
const std = @import("std");
pub fn pick(r: std.Random, comptime weights: anytype) std.meta.FieldEnum(@TypeOf(weights)) {
_ = r;
return @enumFromInt(0);
}
test "pick" {
var prng: std.Random.DefaultPrng = .init(1);
_ = pick(prng.random(), .{ .a = 3, .b = 1 });
}
```
```console
$ zig test r2.zig --test-no-exec -femit-bin=bin
readelf --debug-dump=decodedline bin | awk '$1=="r2.zig" && $2+0>11'
```
**Expected:**
- prologue at line 3 (the `fn` line)
- epilogue at line 6 (the closing brace)
- all within the file (11 lines)
**Actual:**
- rows that name `r2.zig`:
- self-hosted: lines **24 and 25**
- `-fllvm`: line **24**
- the raw line program for `pick` instance:
- sets the file to entry 0 (`r2.zig`)
- sets the column to 24
- advances the line to 24 for the prologue
- has the body at line 5 and line 25 for the epilogue
**Notes:**
- It looks like the source location of the generic instance's entry and exit, not a backend's. Both backends agree on line 24.
- Also happens with a **plain function** with a **comptime parameter** but a simple return type (rows at 15, 16 and 36 in a 12-line file).
- In a larger file, a generic pick at line 227 of a 1,221-line file gets lines 1291, 1292, 1541 and 1542, column 1081.
Don't file this as-is, unless you fully understand it.
notramo (Oct 26 2026 at 12:32):I'm not familiar with debug info and coverage, and I don't understand this issue.
notramo (Oct 26 2026 at 12:34):The last two points in the Notes section are basically useless without the snippets. If the file is too big, add it as an attachment.
notramo (Oct 26 2026 at 12:35):LLMs don't understand how valuable a piece of info is. Often pack the report with useless facts and leave out important details.
Steve Howell (Oct 26 2026 at 12:45):I'm just having the LLM fix it now. There's a simple one-line fix (modulo some diff noise), but the real problem is slightly more structural. The compiler has the right line numbers in the parser phase but doesn't pass those to the next phases.
notramo (Oct 26 2026 at 12:48):How do you know it fixed correctly?
Notification Bot (Oct 26 2026 at 12:49):15 messages were moved here from #show and tell > roc on a unikernel by Steve Howell.
Steve Howell (Oct 26 2026 at 12:50):notramo said:
How do you know it fixed correctly?
I have a reproducer, so I run it. I also have Claude write tests that are red before the fix and turn green. This is child's play for modern agents.
notramo (Oct 26 2026 at 12:58):An issue I have with LLM-written tests, is that they are passing, and I think everything is great, but then I look at the tests, and I find that they specify a different behavior that I wanted.
notramo (Oct 26 2026 at 12:58):How do you solve this?
Steve Howell (Oct 26 2026 at 13:00):You can read the test that the LLM writes. :)
Steve Howell (Oct 26 2026 at 13:04):Here's another bug in 0.17: http://143.244.172.148:9100/notes/zig-issue-arena-overshoot.md
Claude wrote a perfectly good test without any prompting:
pub fn main() !void {
var buf: [1 << 20]u8 = undefined;
var fba: std.heap.FixedBufferAllocator = .init(&buf);
var logging: Logging = .{ .child = fba.allocator() };
var arena: std.heap.ArenaAllocator = .init(logging.allocator());
defer arena.deinit();
const a = arena.allocator();
std.debug.print("alloc 100:\n", .{});
_ = try a.alloc(u8, 100);
std.debug.print("alloc 100000 (does not fit the first node):\n", .{});
_ = try a.alloc(u8, 100_000);
std.debug.print("arena holds {d} bytes for 100,100 requested\n", .{arena.queryCapacity()});
}
What if you file this with a short human-written description?
Richard Feldman (Oct 26 2026 at 13:08):I really strongly recommend following the rules of the repo
Richard Feldman (Oct 26 2026 at 13:08):they aggressively ban for rules violations
Richard Feldman (Oct 26 2026 at 13:10):also if you have a repro that's tiny enough, and you understand it, it's probably not going to take very long to handwrite it :smile:
Steve Howell (Oct 26 2026 at 13:12):The only way to follow the rules of the repo is to not submit the bugs. So they are gonna stay broken upstream.
If folks are concerned about arena allocations allocating twice the amount of memory that you intend, I recommend this patch to your fork of zig:
http://143.244.172.148:9100/notes/zig-issue-arena-overshoot.md
If folks want the line numbers in their test coverage reports to actually match the lines in the source files, I recommend this one:
http://143.244.172.148:9100/notes/zig-issue-generic-debug-lines.md
There's a much better patch coming that's actually a net reduction in code but a more complicated diff.
Richard Feldman (Oct 26 2026 at 13:12):e.g. it did not take me long to handwite https://codeberg.org/ziglang/zig/issues/37341 or https://codeberg.org/ziglang/zig/issues/30572
Richard Feldman (Oct 26 2026 at 13:16):the rules don't say anything about how you discovered the bugs, just about how you create reports/patches/comments/etc
notramo (Oct 26 2026 at 13:18):the rules don't say anything about how you discovered the bugs, just about how you create reports/patches/comments/etc
It used to be. But I now noticed that they recently changed the rules:
No LLMs for finding bugs.
notramo (Oct 26 2026 at 13:18):Basically if you find an RCE with a 9.0 score using an LLM, it is going to stay unpatched, just because an LLM found it.
notramo (Oct 26 2026 at 13:20):Instead of patching it, they expect downstream projects to ship with vulnerabilities in their code, and simply ask all bad actors to obey their nice code-of-conduct.
notramo (Oct 26 2026 at 13:21):It's the new security paradigm: Security-by-asking-attackers-to-be-nice.
notramo (Oct 26 2026 at 13:21):This is simply bad.
Richard Feldman (Oct 26 2026 at 13:22):oh TIL
notramo (Oct 26 2026 at 13:22):I'm fully supporting their stance of not letting any LLM written code into the project, because LLMs are currently a very weak tool to write code, but LLMs are a very strong tool to find bugs, and neglecting it will cause harm to the Zig project.
Steve Howell (Oct 26 2026 at 13:23):Their policies are pretty clearly anti-LLM. All I am saying here is that their policy is gonna have consequences. I don't think you're insinuating that I'm a bad actor necessarily, but just to be clear, I'm not a bad actor. I used a very powerful tool to find two bugs in the compiler, neither of which is particularly subtle. I wanted to send the patch upstream, which already required real work on my part.
Steve Howell (Oct 26 2026 at 13:23):@notramo you are already using some LLM patches that were submitted by me to Roc.
notramo (Oct 26 2026 at 13:24):I don't think you're insinuating that I'm a bad actor necessarily, but just to be clear, I'm not a bad actor.
Obviously I'm not saying that. (I edited my previous message to make it less ambigous.)
notramo (Oct 26 2026 at 13:30):notramo you are already using some LLM patches that were submitted by me to Roc.
And the LLM usage in Roc is clearly visible by the amount of crashes and miscompilations.
Steve Howell (Oct 26 2026 at 13:30):I do think zig is an amazing project, so I appreciate all the hard work that went into it. That's why I've done a bunch of non-trivial projects with it, including a unikernel. I just hate this particular policy. I've been on the other side, to be clear. Mostly with human slop, not AI slop.
notramo (Oct 26 2026 at 13:34):The Roc project is even more resource limited than Zig, so I guess it's useful to be able to quickly experiment with ideas even if a feature is buggy and unreliable, because if that feature turns out to be a bad idea and need to be removed or changed, the team wasn't wasting time on implementing a useless thing correctly.
Steve Howell (Oct 26 2026 at 13:35):notramo said:
And the LLM usage in Roc is clearly visible by the amount of crashes and miscompilations.
I wish Roc were a bit more robust, but I attribute that to other reasons:
notramo (Oct 26 2026 at 13:36):humans are involved
Do you think if it were entirely LLM-generated, it would be better?
Steve Howell (Oct 26 2026 at 13:36):ah, and to notramo's point, Roc is a bold experiment.
Steve Howell (Oct 26 2026 at 13:40):notramo said:
humans are involved
Do you think if it were entirely LLM-generated, it would be better?
Yes, but that's not what I am claiming Roc should do.
I write all my projects with LLMs now, and I think they are decent quality. I do nontrivial things like unikernels, custom programming languages, and chat servers with Claude. Claude's actually really good at compilers; I spend much more time babysitting Claude for things like silly video games.
Richard Feldman (Oct 26 2026 at 13:41):the new version of the compiler is significantly less buggy and prone to miscompilations than the original Rust version that had no LLM involvement
notramo (Oct 26 2026 at 13:43):I may be simply because of using a different language.
notramo (Oct 26 2026 at 13:43):Or maybe insufficient testing.
notramo (Oct 26 2026 at 13:44):https://github.com/roc-lang/roc/issues/12289
Steve Howell (Oct 26 2026 at 13:44):Richard Feldman said:
the new version of the compiler is significantly less buggy and prone to miscompilations than the original Rust version that had no LLM involvement
Agreed. I should point out that the fixes that I made to Roc were not for core things like getting line numbers correct in traces or allocating the right amount of memory. They were way more obscure, and I found them by fairly deliberately asking my LLM to hunt for bugs. (For context, I was using Roc as an oracle for another compiler, so I was mostly hunting bugs in the other compiler, but that revealed Roc bugs as well.)
notramo (Oct 26 2026 at 13:45):If the fuzzer improvements in #12289 would be implemented, it would uncover hundreds of bugs. Maybe even thousands.
Steve Howell (Oct 26 2026 at 13:46):This is the proper fix for line numbers: http://143.244.172.148:9100/notes/zig-issue-generic-debug-lines.md
It's a classic compiler bug that could have been avoided by making sure you capture the data at the right time during compilation and don't throw it away.
Zak Kohler (Oct 26 2026 at 13:47):I've been coding for 20+, and fabel/opus5.5 has finally hit the point where code quality is at least as good as mine, and, if you take time to ask it to teach, you can really learn a lot.
notramo (Oct 26 2026 at 13:47):if you take time to ask it to teach, you can really learn a lot.
Yes, I use it for learning all the time.
Richard Feldman (Oct 26 2026 at 13:48):@notramo happy to accept a PR for fuzzing improvements! I didn't see anything concrete in that issue but maybe I'm misunderstanding :smile:
notramo (Oct 26 2026 at 13:49):There's a prompt in the issue. Since I'm not familiar with the compiler, I don't know how much actual value do the report have. But it takes ~5 minutes for a core team member to review what the LLMs output when given this prompt. I don't want to submit vibe-coded slop, because I value the time of the maintainers.
Zak Kohler (Oct 26 2026 at 13:49):notramo said:
If the fuzzer improvements in #12289 would be implemented, it would uncover hundreds of bugs. Maybe even thousands.
I believe fixing organically found blockers _before_ the fuzzathon might be prudent. Not only for time constraints, but also architecture. Fuzzing could find endless bugs in corner that don't even exist in a year from now because of larger scale changes. So you don't want to harden that too early.
notramo (Oct 26 2026 at 13:52):LLMs are great at applying patterns in their training data. So I would definitely not write an e-commerce site or a static site by hand.
They are also great at glueing components together.
And are excellent at pattern matching, e.g. security audits and fuzzing.
But they are very weak at designing new patterns and paradigms.
notramo (Oct 26 2026 at 13:53):I believe fixing organically found blockers _before_ the fuzzathon might be prudent. Not only for time constraints, but also architecture. Fuzzing could find endless bugs in corner that don't even exist in a year from now because of larger scale changes. So you don't want to harden that too early.
Isn't the policy of the Roc project is to fuzz early?
Steve Howell (Oct 26 2026 at 13:55):notramo said:
But they are very weak at designing new patterns and paradigms.
That's what the human's for. But I actually don't agree with your premise. Claude knows a lot of patterns and paradigms that are very useful for software engineering. I have Claude actively participate in the design. I would say that Claude does about 40% of the design work and 100% of the coding in my projects. The former number keeps growing.
notramo (Oct 26 2026 at 13:57):I'm not saying that LLMs are useless. I use a wide range of LLMs from different labs, when architecting code. But I do it in a chat interface, not an agent interface, because agents just write slop.
Zak Kohler (Oct 26 2026 at 13:57):On another note, I hit a potential zig bug this morning. Would anyone more familar with zig be willing to sanity check it before I bother them with potential slop?
Steve Howell (Oct 26 2026 at 13:58):notramo said:
If you have some actual bugs, you can always file it manually. It takes at most 5 minutes and the result is a much higher quality report.
You can test that claim out by submitting my bug reports if you want. :)
I'm being snarky, of course, but it's more than five minutes.
Steve Howell (Oct 26 2026 at 13:59):Zak Kohler said:
On another note, I hit a potential zig bug this morning. Would anyone more familar with zig be willing to sanity check it before I bother them with potential slop?
Sure, I can look.
notramo (Oct 26 2026 at 14:00):I'm not measuring LLM usefulness in "how often it can slop together something that somewhat do what I want", but "how often it produces an architecture idea that I accept".
Fable 5.1 is sometimes surprising me with clean and good ideas, but it could not come up with these autonomously, in an agent harness, but only in a chat interface with lot of human feedback.
Richard Feldman (Oct 26 2026 at 14:01):I think that's how pretty much everyone involved with Roc's development uses them :smile:
Richard Feldman (Oct 26 2026 at 14:02):very hands-on
notramo (Oct 26 2026 at 14:02):Steve Howell said:
You can test that claim out by submitting my bug reports if you want. :)
I can't, because unfortunately I'm not familiar with debug info and coverage, and I don't want to submit anything that I don't understand. Maybe sometimes when I get to learn these areas enough that I can validate the claims of the report.
Steve Howell (Oct 26 2026 at 14:02):Yeah, nobody here is vibe coding. I didn't say "Hey Claude, write me a unikernel while I go to the gym."
Steve Howell (Oct 26 2026 at 14:05):notramo said:
[...] I can't, because unfortunately I'm not familiar with debug info and coverage.
Surely you understand how line numbers work. There's some depth to how test coverage works under the hood, but it's literally just keeping track of which lines get executed during tests.
notramo (Oct 26 2026 at 14:09):Surely you understand how line numbers work.
I don't think this is sufficient to validate the report.
https://en.wikipedia.org/wiki/Curse_of_knowledge
Steve Howell (Oct 26 2026 at 14:15):Have you actually read the report? http://143.244.172.148:9100/notes/zig-issue-generic-debug-lines.md
Tested with a Debug build of the patched compiler (no LLVM): the repro gives 3, 5, 6, the anytype one 1, 2, 3, and a function moved down or given a longer body moves its rows with it; zig test test/behavior.zig -fno-llvm -fno-lld: 2194 passed, 27 skipped, 0 failed, the same result test by test as unpatched master.
I will admit that Claude uses funny grammar here.
notramo (Oct 26 2026 at 14:17):Yes, I read it. And I don't understand fully, because I never used tools like this, I'm not familiar with the ELF file format, not familiar with debug info, and only know a few RISC-V assembly instructions. I am mainly a web developer.
notramo (Oct 26 2026 at 14:27):Steve Howell said:
I will admit that Claude uses funny grammar here.
I recommend using other models too, from other labs. Many of them can phrase things better than Claude. I usually try multiple, and select, because the best LLM varies by task.
Steve Howell (Oct 26 2026 at 14:28):I put my patches on GH for now:
https://github.com/showell/patched-zig-compiler/commits/main/
I don't know if either bug affects the Roc compiler, but I suspect the arena allocator does. I'm having Claude get a Roc-specific reproducer.
Zak Kohler (Oct 26 2026 at 14:32):Steve Howell said:
Sure, I can look.
https://gist.github.com/y2kbugger/7143243463a973e490b11b1bac72d48a
To me this seems solid. How much would you include on zig bug tracker to make it fit their policy?
Steve Howell (Oct 26 2026 at 14:37):I would have Claude look at Dir.createDirPath. I'm guessing it's a simple fix.
notramo (Oct 26 2026 at 14:38):Title should be more like:
Dir.createDirPathstuck in infinite loop when given./inside nonexistent CWD (CPU spins at 100%)
The never returns is a typical LLM phrasing and says nothing about the reason.
Steve Howell (Oct 26 2026 at 14:38):If you don't have the zig compiler source handy, I can look.
Zak Kohler (Oct 26 2026 at 14:50):I definitely don't want to propose a fix. I'm already doubting we are allow to submit, even if completely:
No LLMs for brainstorming and then sharing the results of that brainstorming, even if you create the prose. If you use a chatbot to give you advice on a comment on the issue tracker, that comment is unwelcome.
No LLMs for finding bugs.
It's like "no fruit of a tainted brain"
anyway,
and the gist is updated, it proposes a three line fix, and even identifies that zig has already patched the same flavor of bug for dangling symlinks.
Steve Howell (Oct 26 2026 at 14:50):see #bugs > zig arena bug affects Roc for the Roc impact of the zig arena bug
Steve Howell (Oct 26 2026 at 14:58):What's the three-line fix? I'll add the patch to my repo.
(never mind, I had Claude just debug it on my end)
Steve Howell (Oct 26 2026 at 15:20):https://github.com/showell/patched-zig-compiler/commit/384ad687a8e43e69ac5d32038fa8451fdb4e7cb3 is the fix for the infinite loops in making directories. I've probably poisoned this fix for everyone haha.
Zak Kohler (Oct 26 2026 at 15:20):Steve Howell said:
What's the three-line fix? I'll add the patch to my repo.(never mind, I had Claude just debug it on my end)
Steve Howell (Oct 26 2026 at 15:24):As the commit indicates, my Claude ended up extracting a fixed version of the directory walk that all three call sites use now.
removewingman (Oct 26 2026 at 16:28):imo, looks good. I could reproduce with zig master on Alpine Linux x86_64. Maybe also add your OS and architecture.
I am not sure on how to understand the docs tho and what you are trying to achieve: https://ziglang.org/documentation/master/std/#std.Io.Dir.createDirPath
maybe a solution: https://ziglang.org/documentation/master/std/#std.Io.Dir.cwd should probably return an error that it has also the option that it does not exist
Zig compiler is awesome, but sadly their std lib lacks functionality.
Steve Howell (Oct 26 2026 at 16:57):removewingman said:
maybe a solution: https://ziglang.org/documentation/master/std/#std.Io.Dir.cwd should probably return an error that it has also the option that it does not exist
I'm having a hard time parsing your sentence.
notramo (Oct 26 2026 at 17:04):Dir.cwd() should return an error if the CWD directory does not exist on the filesystem.
notramo (Oct 26 2026 at 17:04):But I don't think the OS exposes this info.
notramo (Oct 26 2026 at 17:05):Maybe it's only an inode unlink, but the dir itself is kept until every process releases it?
removewingman (Oct 26 2026 at 18:03):Was just a thought, idk.
Zig implementation uses this: Special value used to indicate openat should use the current working directory https://ziglang.org/documentation/master/std/#std.os.linux.AT.FDCWD
I am guessing the path resolution fails.
Rust has https://doc.rust-lang.org/std/env/fn.current_dir.html and uses https://www.man7.org/linux/man-pages/man3/getcwd.3.html (posix standard) and it can have errors.
Steve Howell (Oct 26 2026 at 18:38):The Roc compiler has an alignment bug: https://github.com/roc-lang/roc/pull/12641
Anton (Oct 26 2026 at 18:42):Thanks for the fix @Steve Howell :)
I can look at it once it has passed CI.
Steve Howell (Oct 26 2026 at 21:57):@Anton It failed a tidy test, but it's fixed now. It's still working through CI and will hopefully pass this time.
Karakatiza (Oct 27 2026 at 08:49):@notramo the Claude Code harness is no worse than a chat for designing a solution or architecture. In my experience and other testimonials the "design step" is very important. You can just talk in the chat or ask the agent to write a design document and then you go over it yourself or even ask the agent to go over it, loosely pointing at the choices or behavior you don't like; in both cases Claude produces good results after multiple "passes over the design". You can't rely on it one-shotting a design, rather converge on it
Steve Howell (Oct 27 2026 at 10:01):I totally agree with Karakatiza about multiple "passes over the design". One thing that I've found super helpful is that Claude can use tools like graphviz to help you visualize the designs in your systems.
Here's an example:
Steve Howell (Oct 27 2026 at 16:52):Steve Howell said:
The Roc compiler has an alignment bug: https://github.com/roc-lang/roc/pull/12641
This is passing now and is just blocked by review. Anton please let me know if you need anything more on this. I assume it's just waiting on your time now--yesterday it was blocked on CI, but I think I fixed that.
Steve Howell (Oct 27 2026 at 18:34):Anton Thanks for merging! I should have another alignment PR up by tomorrow. I had Claude do a systematic hunt for bugs, and it found a couple more.
Loris Cro (Oct 28 2026 at 11:39):Zak Kohler said:
I definitely don't want to propose a fix. I'm already doubting we are allow to submit, even if completely:
No LLMs for brainstorming and then sharing the results of that brainstorming, even if you create the prose. If you use a chatbot to give you advice on a comment on the issue tracker, that comment is unwelcome.
No LLMs for finding bugs.It's like "no fruit of a tainted brain"
Hi from ZSF!
Just wanted to clarify our perspective a bit and explain how to open an issue to report a bug in a way that will be well-received.
What we care about the most when looking at issues is to make sure that spending time to solve one unblocks Zig users who are trying to use Zig to make software. Zig is still work in progress and lots of things are known to be incomplete or are likely to be rewritten as new features are added or existing ones are polished.
A bug report that affects nobody takes away energy from the main development work and is even likely to be regressed if the relative subsystem will change significantly enough in the future.
For us, a bug report is worth acting on only if it unblocks Zig users and, even then, we're not able to serve all users so some prioritization has to be done (ie some bug reports won't be acted on before we end up overhauling the subsystem, see the new Io interface for example which came in and invalidated many old I/O related issues, while also creating new ones).
If you ever wondered why would we ever not want LLMs to analyze the Zig codebase and report all bugs, that's why. At some point we'll reach a greater level of maturity and we will become increasingly more interested in all bug reports, but right now not only they're not interesting, they would also drown in noise issues that we do want to act on.
So, Roc development makes use of LLMs, and if an LLM tells you about a bug in Zig, then
you can't act on that (as the contribution rules state), how do you unblock yourself then?
It's essentially two steps:
For both steps we ask that you do all the work yourself and don't involve LLMs. This is done to lower the chances of the bug report not being good (i.e. not pointing at a real bug or not being a blocker for you), and to give you the amount of first-hand knowledge necessary to interact on the issue tracker with Zig core contributors, in case that there are follow up questions.
There are also other reasons behind our request to follow this procedure, one for example is that, even if you have complete certainty that your LLM generated report were to always be good, without any form of human authoring, it would still be possible for one project to overwhelm us with by-construction legitimate issues, but we don't want that either.
Additionally, there are some situations where the concept of 'blocker' becomes fuzzy: if you're running a bunch of experiments in parallel, encountering a bug in that setting counts as a blocker? To some degree yes, to some degree no.
These are uncharted waters in software collaboration so I don't know for sure what the best approach would be, but given the set of constraints, my thinking is that it would make sense for the Roc project to have a soft fork of Zig that contains all the trivial fixes that you can have your LLM solve quickly, and reach for the Zig issue tracker for more serious issues, where slowing down and double checking everything is worth your time (e.g. a Zig miscompilation).
You need to be careful to not go too far with your fork (in order to be able to pull easily from upstream), but with some discipline you should be able to get the best of both worlds. Related to this topic, Martin Wickham recently gave a talk at SYCL Vancouver titled Maintaining a Long Term Fork, where he talks about how his previous company used to maintain a long term soft fork of Unreal Engine.
Anyway this is just an idea, hopefully I've made our perspective clear and from there I'm also happy to discuss altrenative approaches if any come to mind. Also at that point we probably should split off a dedicated thread maybe.
Steve Howell (Oct 28 2026 at 11:46):Loris Cro said:
For both steps we ask that you do all the work yourself and don't involve LLMs. This is done to lower the chances of the bug report not being good (i.e. not pointing at a real bug or not being a blocker for you), and to give you the amount of first-hand knowledge necessary to interact on the issue tracker with Zig core contributors, in case that there are follow up questions.
I use LLMs a tool to get work done. I do a much better job with LLMs than without, so I find the whole premise kind of absurd that throwing away the tool "lowers the chances of the bug report not being good." I've been coding since the mid-80s, so I feel empowered to make that judgment.
I do appreciate the long, thoughtful response (and I know it wasn't directly addressed to me, but I started the topic.)
Steve Howell (Oct 28 2026 at 11:54):I am just having Claude fix the issues going forward. I know how to maintain a fork, so that's what I am doing until the policy changes: https://github.com/showell/patched-zig-compiler/commits/main/
I fixed the bug that @Zak Kohler reported on my branch despite it not affecting me yet, because I don't want to stumble over it myself today.
The other bugs directly impact me, because I'm actually using zig to build stuff!
@Loris Cro I encourage you to look at these patches:
I think the best approach for the zig project to improve its bandwidth under the reality that some bug reports/fixes are coming from LLMs (with human supervision, of course!) is to use LLMs on your side to triage bugs. This works on other projects.
Steve Howell (Oct 28 2026 at 12:05):On my branch, the commit messages don't describe how we encountered the bugs, but I can provide the context:
Zak Kohler (Oct 28 2026 at 12:59):Loris Cro said:
If you ever wondered why would we ever not want LLMs to analyze the Zig codebase and report all bugs, that's why.
I've actually referenced and supported this stance in the past and I think it makes a lot of sense.
Zak Kohler said:
I know zig has an official stance, closely following what sqlite did.
- First find all bugs that are blocking users
- Then find all bugs that appear from 100% branch coverage tests (they have a technical term where they analyze machine code branches I forget what it is)
- Traditional fuzzing
- Adversarial LLM "fuzzing"/cornercase search
Zak Kohler said:
notramo said:
If the fuzzer improvements in #12289 would be implemented, it would uncover hundreds of bugs. Maybe even thousands.
I believe fixing organically found blockers _before_ the fuzzathon might be prudent. Not only for time constraints, but also architecture. Fuzzing could find endless bugs in corner that doesn't even exist in a year from now because of larger scale changes. So you don't want to harden that too early.
I believe your two step process is sufficient to allow someone to submit an LLM discovered blocker in good faith. Thanks for the detailed reponse Loris.
Steve Howell (Oct 28 2026 at 13:05):@Zak Kohler Are you gonna submit the create-directory bug upstream? If so, I would encourage you to point out that in addition to the infinite loop problem, the human-written code was repeated in 3 different places.
Of course, it's already fixed on https://github.com/showell/patched-zig-compiler/commit/384ad687a8e43e69ac5d32038fa8451fdb4e7cb3
notramo (Oct 28 2026 at 13:13):@Loris Cro, I highly recommend adding this explanation here:
https://ziglang.org/code-of-conduct/#strict-no-llm-no-ai-policy
notramo (Oct 28 2026 at 13:14):It would help many people to understand the reasons behind the no-LLM policy.
Steve Howell (Oct 28 2026 at 13:16):Loris Cro said:
These are uncharted waters in software collaboration so I don't know for sure what the best approach would be, but given the set of constraints, my thinking is that it would make sense for the Roc project to have a soft fork of Zig that contains all the trivial fixes that you can have your LLM solve quickly, and reach for the Zig issue tracker for more serious issues, where slowing down and double checking everything is worth your time (e.g. a Zig miscompilation).
Just to be clear, none of the three fixes directly affect Roc. Most of my Roc projects have a little zig under the hood, so that's why my zig venting came up in a Roc context.
notramo (Oct 28 2026 at 13:18):No LLMs for finding bugs.
This section could be extended with a simple explanation:
No LLMs for finding bugs. The Zig codebase is often drastically changed. We don't want to spend time fixing bugs in code that we are going to rewrite anyways. If that bug affects real users, it'll come up anyways, even without LLM fuzzing. If it doesn't, there's no need to find it.
Richard Feldman (Oct 28 2026 at 13:27):thanks Loris, and hi! :smiley:
Zak Kohler (Oct 28 2026 at 15:58):Steve Howell said:
I've been coding since the mid-80s, so I feel empowered to make that judgment.
Same, but since mid-90s. But skeptics need not take our word for it: Linus was the final canary; as ruthless defender of quality and self-professed man of practicality (results>theories, edison>tesla, etc), the LLM usage at kernel.org is all the proof anyone should need.
Steve Howell (Oct 28 2026 at 16:11):Haha I think the Linus argument is complicated. He's a bit polarizing.
Dan G Knutson (Oct 28 2026 at 17:46):Whether you or I agree with Zig's policy is 100% orthogonal to whether it's good behavior to break the rules of a community when you're trying to join or contribute.
Zak Kohler (Oct 28 2026 at 17:54):Dan G Knutson said:
Whether you or I agree with Zig's policy is 100% orthogonal to whether it's good behavior to break the rules of a community when you're trying to join or contribute.
Zak Kohler said:
I definitely don't want to propose a fix. I'm already doubting we are allow to submit, even if completely rewritten by hand
hence my hesistation earlier, before Loris kindly clarified
Steve Howell (Oct 28 2026 at 18:00):Dan G Knutson said:
Whether you or I agree with Zig's policy is 100% orthogonal to whether it's good behavior to break the rules of a community when you're trying to join or contribute.
Was this addressed to me?
Dan G Knutson (Oct 28 2026 at 18:02):You, Zak, and notramo upthread, yeah
notramo (Oct 28 2026 at 21:42):Linus was the final canary; as ruthless defender of quality
Not so much. Linus very strongly defends his own opinions. Whether those opinions are always about quality is another question.
His opinion is often about quality, but there are many instances where quality is not the most important.
notramo (Oct 28 2026 at 21:42):https://en.wikipedia.org/wiki/Criticism_of_Linux#Kernel_code_quality
notramo (Oct 28 2026 at 21:42):The Rust introduction into the kernel is famously bad. It's arguably worse than if the kernel were pure C (and way worse than if it were pure Rust).
The two languages use very different data structures and organization practices.
notramo (Oct 28 2026 at 21:43):Mix of Rust and C is likened to "cancer" by Cristoph Hellwig:
https://lore.kernel.org/rust-for-linux/20250128092334.GA28548@lst.de/
Hellwig also mentioned in a different thread that the kernel Rust code is not idiomatic Rust, for example, a linked list data structure that is used by the kernel is difficult in Rust.
There are more messages, a search-enabled LLM can summarize it better.
Prompt:
Find the LKML conversation aobut the mix of Rust and C being likened to "cancer".
notramo (Oct 28 2026 at 21:43):Drew DeVault blog post:
Does Rust belong in the Linux kernel?
notramo (Oct 28 2026 at 21:43):Loris Cro blog post:
RAII and the Rust/Linux Drama
notramo (Oct 28 2026 at 21:52):Redox OS has a no-LLM policy:
https://gitlab.redox-os.org/redox-os/redox/-/blob/master/CONTRIBUTING.md#ai-policy
notramo (Oct 28 2026 at 21:54):I'm not saying that LLM usage is a bad thing. But I understand that some projects might want to ban it, and often agree with it.
Richard Feldman (Oct 29 2026 at 03:55):I think it's really important for each project to find a dynamic that works for the people in that project - what works for Roc may not work well for Zig, and vice versa
Richard Feldman (Oct 29 2026 at 03:57):I think it's easy to imagine a world where Zig adopted LLMs and it really did not go well for them, and as someone who has benefited from how they've done things so far, I'm in no hurry to see them gamble with their successful formula :smile:
Last updated: Nov 04 2026 at 18:54 UTC