Stream: compiler development

Topic: broken tests on main


view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 04:29):

I noticed that my PR #5955 was hitting failures that seem totally unrelated. Did a bisect and it looks like the failures are from an older commit/PR. My guess is that the failures may depend on the specific CI machine and that is why they aren't being hit all the time (or maybe related to caching). We have 4 failing cli_run tests:

Anyway, I did a bisect and the failures are from #5557 specifically this commit: https://github.com/roc-lang/roc/commit/ac13ce57c22e618782437b4cbc75012a19b96d4d

The failures only happen on the linux x86_64. Any ideas?

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 04:35):

Really weird to me is that the final commit in #5557 passed our ci: https://github.com/roc-lang/roc/pull/5557/commits/7ab96e664a8bfd33201e5d5905d94d5bb9255667 is failing locally for me on these tests.

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 04:36):

I stand corrected, failure or not depends on caching...need to do a more thorough bisect.

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 04:57):

Ok. update. When running git clean -fdx examples/ before testing, I actually can't reproduce this locally at all. I wonder if we need to run git clean -fdx examples/ on the ci machine to get the tests working again.

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 05:25):

Ok. Adding the cleaning to the ci job looks to have fixed the issue! waiting for all of the builders to finish

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 06:50):

@Anton, Does this look fine to you as a general change for ci? https://github.com/roc-lang/roc/pull/5955/commits/2af9abdcc55335561f5b2a2893165ea7fbf8f7c2

view this post on Zulip Anton (Nov 07 2023 at 10:08):

Hmm, the github checkout action already does git clean -ffdx
Screenshot_20231107_110726.png

view this post on Zulip Anton (Nov 07 2023 at 10:17):

So I don't think that addition is what fixed the issue.

view this post on Zulip Anton (Nov 07 2023 at 10:30):

Looking at all the runs of that failing workflow Richard's PR also hit that issue twice and then magically didn't anymore, all before you ran with git clean -fdx on CI.

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 15:06):

Do these all run on the same physical machine?

view this post on Zulip Anton (Nov 07 2023 at 15:26):

Yes, indeed

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 15:29):

:thinking: so that fixed locally for me, but remotely something else fixed it? I wonder what. Maybe this is a concurrency related flake? Any other ideas?

view this post on Zulip Anton (Nov 07 2023 at 15:49):

Richard forced through some PRs without all the required checks having finished. Given that he was working on the website, this was not risky, but I do wonder if that altered github workflow execution in a strange way.

If it was concurrency related (in the roc codebase) it would be quite strange to fail 4 times, very close together in time and then never again before or after (that I'm aware of).

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 15:54):

Ok. Reverting the change to my PR. Hopefully it just passes and I can submit.

view this post on Zulip Brendan Hansknecht (Nov 07 2023 at 17:08):

Ok. Yeah, looks to be resolved in general now. Can someone give me a final review on #5955?


Last updated: Jul 06 2025 at 12:14 UTC