Ayaz Hafiz (Aug 23 2022 at 15:22): Ayaz Hafiz (Aug 23 2022 at 15:22):Similar to #contributing > generating a big Roc program for benchmarking, it would be helpful to have some mechanism for generating Roc programs with certain semantic properties, to check that the compiler is correctly validating or invalidating those properties. Here are a few examples of properties that would be useful for us to check:
We have a few algorithms for determining specializations of abilities whose correctness is assumed, but hasn't been verified to a significant extent. It would be helpful for us to generate programs that use a lot of abilities, and make sure that the compiler is able to resolve their specializations correctly (if it can't, that means it's possible to write programs that are impossible to compile - that's not good!)
In the past couple months we've run into a few holes in the exhaustiveness checker that we've patched. It would be nice to have a way to generate when expressions whose exhaustiveness status we know, and validate that the compiler checks their exhaustiveness the same way.
One of Roc's goals is to compile and run programs in the presence of type errors, at least as far as they can without getting stuck. However, today, a lot of programs with type errors cause the compiler to crash because those type errors break invariants in latter phases of the compiler. We have many known issues regarding such crashes, but later down the road, it could be useful to generate programs with type errors, and make sure Roc can compile them without crashing!
All of these are things I'm very interested in exploring more, so if anyone would be interested in helping explore them, I'd be more than happy to assist.
Brendan Hansknecht (Aug 23 2022 at 15:54):Just a note, we have some parser fuzzing setup, but I don't think it works anymore. I think it is easy to fix, but then you just hit panics in parsing super quick, so it isn't very useful.
Brendan Hansknecht (Aug 23 2022 at 15:55):So would have to be a lot smarter than fuzzing currently. Otherwise it will just find our current todos and other panics.
Ayaz Hafiz (Aug 23 2022 at 15:55):Yeah, ideally we would generate ASTs with certain properties
Brendan Hansknecht (Aug 23 2022 at 15:56):I remember looking at trying to generate an AST for fuzzing, but didn't want to deal with all of the bumpalo and references. Makes things a lot lot more annoying to generate.
Brendan Hansknecht (Aug 23 2022 at 15:57):Was looking at setting up random AST to formatter to parser and finally back to AST.
Ayaz Hafiz (Aug 23 2022 at 15:57):yes very true. It may be easier to generate the canonical AST, since that is a much smaller AST surface
Ayaz Hafiz (Aug 23 2022 at 15:57):though that doesn't really cover cases needed for the formatter
Last updated: Jul 26 2025 at 12:14 UTC