High Priority Issues (Jul 08 2026 at 01:04): High Priority Issues (Jul 08 2026 at 01:04):Nightly Gate failed on main @ 8f7bb2d9f8716044ba23601f50bb5485dd10f063 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/28900122768
Luke Boswell (Jul 08 2026 at 01:14):It works! :grinning_face_with_smiling_eyes:
Richard Feldman (Jul 08 2026 at 02:29):I'm looking into fixes
Luke Boswell (Jul 08 2026 at 02:31):Let me know if you need Valgrind or Windows assistance. My machines are backed up with other things, but I can prioritize this if we need.
Richard Feldman (Jul 08 2026 at 02:53):I have them in progress, thanks though!
High Priority Issues (Jul 08 2026 at 07:33):Nightly Gate failed on main @ b3a8c0d869f17661c5ac870c976758cfc8f03d87 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/28916021273
Anton (Jul 08 2026 at 10:43):Earlier I also started investigating the windows bug, do you already have a PR up @Richard Feldman?
Anton (Jul 08 2026 at 11:03):I assume no one is on the ubuntu-24.04-arm failure yet, so I'll take that one.
Richard Feldman (Jul 08 2026 at 11:31):yeah I had GPT-5.5 running on the Windows one overnight - PR: https://github.com/roc-lang/roc/pull/10037
Richard Feldman (Jul 08 2026 at 11:33):I'm looking into a better long-term fix but that will at least unblock nightlies, which I'd rather so soon bc I have a bunch of docs website updates that are blocked on a new nightly landing :smile:
Anton (Jul 08 2026 at 15:53):Richard Feldman said:
I'm looking into a better long-term fix but that will at least unblock nightlies, which I'd rather so soon bc I have a bunch of docs website updates that are blocked on a new nightly landing :smile:
The nightlies repo is not yet using the new nightly branch, so a new nightly was released 4 hours ago.
High Priority Issues (Jul 09 2026 at 06:36):Nightly Gate failed on main @ 5c318204a3ffc5c3e897d7e1080b052b2b3673f3 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/28993640719
High Priority Issues (Jul 10 2026 at 06:42):Nightly Gate failed on main @ 108e57d54fe20d61c743609212a47a83f452fbd0 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29068732521
High Priority Issues (Jul 11 2026 at 07:22):Nightly Gate failed on main @ 252a59ae74d430c2650f38007cf4e74dcfc86656 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29138509723
Richard Feldman (Jul 11 2026 at 19:54):PR to fix :point_up:
Richard Feldman (Jul 11 2026 at 19:55):it would be cool if we made this notify the channel again when nightly gate passes after having failed, so that we can know just by reading this channel if the previously-mentioned failure has been addressed :smile:
High Priority Issues (Jul 12 2026 at 08:14):Nightly Gate failed on main @ 6535dc1bddac4e3c261c546f3129115c18c73ddb — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29179066775
Anton (Jul 12 2026 at 10:53):I just fixed some nightly test failures in https://github.com/roc-lang/roc/pull/10066 but not everything
High Priority Issues (Jul 14 2026 at 04:41):Nightly Gate failed on main @ 012eb3d50f3cd0673a653e1b9bc4f653dbee1eb2 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29304004961
High Priority Issues (Jul 15 2026 at 04:11):Nightly Gate failed on main @ 7ae89a19d20cdf8eea7378d0eef2b8cf92783aa0 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29387049474
Anton (Jul 15 2026 at 12:46):I will look at the echo.wasm integration test failure
High Priority Issues (Jul 16 2026 at 07:27):Nightly Gate failed on main @ bed8e44eef0ceb0444d2fffff088d60adb64b1f1 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29469562160
High Priority Issues (Jul 17 2026 at 06:25):Nightly Gate failed on main @ bed8e44eef0ceb0444d2fffff088d60adb64b1f1 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29552984190
High Priority Issues (Jul 18 2026 at 06:55):Nightly Gate failed on main @ 7cec8f714169a8a1012186508ab4194e5f685ba5 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29629106607
Richard Feldman (Jul 18 2026 at 20:39):https://github.com/roc-lang/roc/pull/10212 should now have fixed :point_up:
High Priority Issues (Jul 19 2026 at 06:14):Nightly Gate failed on main @ af2262192914eb19823f8790be9935b014ded0fa — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29672573994
Anton (Jul 19 2026 at 12:00):I will look at the failure of roc test gives tag parsers direct payload boundaries
Anton (Jul 19 2026 at 16:06):Looks like we have several failures on main minici, I will check if there is already a PR for any of those issues and will fix the rest of them.
Richard Feldman (Jul 19 2026 at 16:08):one of my PRs has some fixes
Anton (Jul 19 2026 at 20:48): Richard Feldman (Jul 19 2026 at 22:42):@Anton there were still some CI failures on that PR so I'll make a PR into your PR :smile:
Richard Feldman (Jul 20 2026 at 03:36):https://github.com/roc-lang/roc/pull/10260 fixed everything except for windows timeouts
Luke Boswell (Jul 20 2026 at 03:56):I'll investigate Windows timeouts and see if I can find anything
High Priority Issues (Jul 20 2026 at 06:30):Nightly Gate failed on main @ 8eaa9abdea7486f4b3737bcefae64d8731462c96 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29716131919
Anton (Jul 20 2026 at 12:01):Luke Boswell said:
I'll investigate Windows timeouts and see if I can find anything
It feels like minici has increased significantly in runtime recently (on macOS too)
Luke Boswell (Jul 20 2026 at 12:03):See https://github.com/roc-lang/roc/pull/10266
Luke Boswell (Jul 20 2026 at 12:03):I havent had a chance to read through that in great detail, Ive been out playing sport this evening.
Luke Boswell (Jul 20 2026 at 12:04):But the analysis looked like it was on the right track when I was following that
Luke Boswell (Jul 20 2026 at 12:15):Im heading off now for the evening... but this analysis suggests there is something really broken with macos we need to find https://github.com/roc-lang/roc/pull/10266#issuecomment-5021739529
Luke Boswell (Jul 20 2026 at 12:16):I can pick this back up again in like 12hrs
Luke Boswell (Jul 20 2026 at 12:19):If you want to continue investigating in the meantime feel free to make commits to that PR etc, I dont have any WIP or anything.
Anton (Jul 20 2026 at 14:03):I am looking into fixing these:
Four separate failures in this test run:
ARC borrow certifier panic (http_header_decoder_platform_test) — compiler panics with a "did not match payload" error while certifying proc 5, during LIR lowering of
http-headers/app.roc.ARC borrow certifier panic (json_decoder_platform_test, camel) — same class of certifier panic (mismatched switch/payload condition), this time in proc 16 while compiling
json-decoder/camel_app.roc.Missing method
parser_for(json_decoder_platform_test) — type-checking error:Json.parse(json)is called on a record type that has noparser_formethod defined, injson-decoder/app.roc:21.Docs snapshot mismatch (snapshot_tool_test) — one of 9 snapshot tests fails because generated docs output uses abbreviated names (
mod,type_mod) instead of the expected full names (module,type_module) indocs_package_hides_private_modules.md.
I will check if there any PRs up that already fix any of these.
Richard Feldman (Jul 20 2026 at 15:16):the abbreviated names are intentional
Richard Feldman (Jul 20 2026 at 15:17):I got sick of agents thinking that module is a keyword and writing .roc files that use it, and I had a hunch that this was happening because of existing uses of module in the code base (since the same model would often in the same exact file use correct modern syntax in every other way except the module keyword)
Richard Feldman (Jul 20 2026 at 15:18):so I made a lint rule banning the word module in all snapshot tests, and minimized where it appears in the code base to just specifically the compiler warning that tells you that keyword is deprecated
Richard Feldman (Jul 20 2026 at 15:18):and the ban means our s-expressions need to say mod instead of module :smile:
Anton (Jul 20 2026 at 15:52):Richard Feldman said:
the abbreviated names are intentional
Interestingly Fable figured that out by itself :)
Anton (Jul 20 2026 at 17:11): Will Hawkins (Jul 20 2026 at 21:30):Anton said:
Luke Boswell said:
I'll investigate Windows timeouts and see if I can find anything
It feels like minici has increased significantly in runtime recently (on macOS too)
The CI failures on Windows, then, were/are about timeouts? (Sorry for not being able to track the conversation clearly!)
Luke Boswell (Jul 20 2026 at 21:45):The way the build graph was set up on Windows, each test run was calling all the prior runs also instead of running independently, so it stacked up exponentially with the earlier test runs being repeated many times.
Will Hawkins (Jul 20 2026 at 21:46):Luke Boswell said:
The way the build graph was set up on Windows, each test run was calling all the prior runs also instead of running independently, so it stacked up exponentially with the earlier test runs being repeated many times.
Thanks for clarifying! Really helps!
Luke Boswell (Jul 20 2026 at 21:46):So on Windows, asking for one suite ran every suite registered before it — and MiniCI invokes each granular step as its own
zig buildprocess, replaying the chain prefix once per affected job.
Luke Boswell (Jul 20 2026 at 21:47):For whatever reason when Claude drafts PR descriptions they read like a novel with all kinds of overly verbose language
Luke Boswell (Jul 20 2026 at 21:49):Anton said:
I will wait for this to merge, then rebase the Windows fixes onto that
Will Hawkins (Jul 20 2026 at 22:46):It seems like
if (self.serialize_runs) {
if (self.last_run) |last_run| {
run_step.dependOn(last_run);
}
self.last_run = run_step;
}
is the real culprit?
As a small example of the bizarro that seems to be happening as a result of that snippet, consider these two consecutive calls to addRun for tests_summary (references to code from before the PR):
run_fx_platform_test.step.dependOn(build_roc_step);
tests_summary.addRun(&run_fx_platform_test.step);
the next place where a step is added to the tests_summary is
run_http_header_decoder_platform_test_for_summary.step.dependOn(build_roc_step);
tests_summary.addRun(&run_http_header_decoder_platform_test_for_summary.step);
So, in just this very particular instance, because addRun would add a dependency between run_http_header_decoder_platform_test_for_summary.step and run_fx_platform_test.step (because run_fx_platform_test.step is last_step at the time of the invocation of addRun),
run_http_header_decoder_platform_test_for_summary.step depends on run_fx_platform_test.step andrun_http_header_decoder_platform_test_for_summary.step depends on build_roc_step.step andrun_fx_platform_test.step depends on build_roc_step.step.And so on with whatever build_roc_step.step depends on and whatever that depends on and ...
I would be really interested to know whether moving
if (builtin.os.tag == .windows) {
// Zig 0.16's Windows test runner IPC can time out while many Roc test
// binaries are starting at once. Keep the same tests, but start them
// in a deterministic order.
tests_summary.setRunSerialization();
}
after building the dependency tree would have solved the problem.
Of course, the solution being pursued now is much more comprehensive -- which is awesome! Things like this always pique my curiosity.
Luke Boswell (Jul 21 2026 at 00:05):Oh the other major issue was that our .zig-cache was over the setup-zig action limit, so it deleted the cache after that step and then subsequent build and run steps were all cold and had to rebuild the caches
Will Hawkins (Jul 21 2026 at 00:25):Luke Boswell said:
Oh the other major issue was that our
.zig-cachewas over the setup-zig action limit, so it deleted the cache after that step and then subsequent build and run steps were all cold and had to rebuild the caches
Of course! I just thought it was an interesting experiment to reason about the root cause?
Luke Boswell (Jul 21 2026 at 00:39):@Richard Feldman -- I'm porting the lambda mono differential on Windows to re-use the proper parallel test harness, and I'm also investigating the macos specific issue. Both of these I'll stack onto this https://github.com/roc-lang/roc/pull/10266 PR I think.
Luke Boswell (Jul 21 2026 at 00:57):I think I found the macos issue -- stack capture/tracing from the debug allocator. Wiring the debug_gpa_traces build flag through looks like it eliminates the issue. Just digging a little deeper to see if there are other places where this needs to be applied.
Luke Boswell (Jul 21 2026 at 00:57):Also the Windows parallel runner gives us a 12x speedup for that (432s -> 32s) which is nice -- and also removes a lot of duplicate logic that is around.
Richard Feldman (Jul 21 2026 at 01:00):ha, yeah I just found the same thing :laughing:
Richard Feldman (Jul 21 2026 at 01:00):among some other speedups
Richard Feldman (Jul 21 2026 at 01:00):I have another draft pr in the works
Luke Boswell (Jul 21 2026 at 01:01):what's the best way to coordinate on this then? I might pause on the macos allocator work
Richard Feldman (Jul 21 2026 at 01:01):I think after I land this next batch we'll be ok (famous last words)
Richard Feldman (Jul 21 2026 at 01:01):I'll merge yours in too
Luke Boswell (Jul 21 2026 at 01:02):I haven't pushed the macos stack capture changes they are just local
Richard Feldman (Jul 21 2026 at 01:02):yeah I already have those on a branch
Richard Feldman (Jul 21 2026 at 01:02):I have that covered
Richard Feldman (Jul 21 2026 at 01:02):I'm also gonna try to add a lint to prevent that from happening again
Luke Boswell (Jul 21 2026 at 01:03):Ok, just pushed to https://github.com/roc-lang/roc/pull/10266 so you can merge that now if you like
Richard Feldman (Jul 21 2026 at 01:04):cool thanks!
High Priority Issues (Jul 21 2026 at 04:41):Nightly Gate failed on main @ afef9119194708c1bacebcef063e6bc39fc4a72f — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29799533315
Luke Boswell (Jul 21 2026 at 04:42):Tracking... https://github.com/roc-lang/roc/pull/10278 should be good to land after this CI run, then I'll kick of a fresh nightly
Will Hawkins (Jul 21 2026 at 08:34):Luke Boswell said:
Tracking... https://github.com/roc-lang/roc/pull/10278 should be good to land after this CI run, then I'll kick of a fresh nightly
Nice work!
Luke Boswell (Jul 21 2026 at 09:46):It didnt quite get there. Still one issue to track down
High Priority Issues (Jul 23 2026 at 08:33):Nightly Gate failed on main @ 76755ed696abb8d358d59ff7524e6ffbb2e5e6e8 — the nightly branch was not advanced: https://github.com/roc-lang/roc/actions/runs/29977909807
Last updated: Jul 23 2026 at 13:15 UTC