Aurélien Geron (Aug 04 2024 at 01:06): Aurélien Geron (Aug 04 2024 at 01:06):I'm working on adding a Roc track to exercism.org (a popular platform to learn new programming languages, see #contributing>exercism). One of their requirements is to provide a test runner that will test the user's code (in a docker container) and output a JSON file containing details about the errors in the user's code. This JSON file will then be used by the Exercism platform to nicely display the errors to the user.
Right now the output of roc test is meant for humans. I guess I could write a script to parse it, but that might be quite brittle. Alternatively, there could be an option such as roc test --json-output to output JSON (or some other machine-readable format). This would greatly simplify granular automated testing in Roc.
What do you think?
Brendan Hansknecht (Aug 04 2024 at 01:12):How do other languages generally do this?
Brendan Hansknecht (Aug 04 2024 at 01:12):I don't think many languages have test tools that emit JSON, but could be totally wrong
Ayaz Hafiz (Aug 04 2024 at 01:15):a ton of languages emit json or xml
Ayaz Hafiz (Aug 04 2024 at 01:15):junit xml is a popular one but there are other ones
Brendan Hansknecht (Aug 04 2024 at 01:20):Cool
Aurélien Geron (Aug 04 2024 at 01:24):For now I can parse the output, it's not hard, it's just that I fear that it's the kind of thing that can easily break in the future as new features are added.
Aurélien Geron (Aug 04 2024 at 01:27):Also note that Exercism does have the option to return a JSON file with a single big output message, without trying to split it into per-test outputs. I could start with that, but it wouldn't be a great experience for users on the Exercism platform.
If you're interested, here are the Exercism test-runner specs.
Richard Feldman (Aug 04 2024 at 01:45):in general I'd like to follow glue's example for things like this
Richard Feldman (Aug 04 2024 at 01:46):give you a way to write a roc script that takes roc data structures and outputs whatever format you like
Richard Feldman (Aug 04 2024 at 01:47):I like that because it doesn't tie us to any specific format (e.g. there was a time when xml was the obvious choice, today json is the obvious choice, in the future who knows what it'll be?) and also it's maximally flexible for the end user
Richard Feldman (Aug 04 2024 at 01:48):e.g. if you want JUnit XML vs TAP format vs something else, you can write a script (or get one from someone else) that outputs exactly what you want, and now you don't even need to run a separate program to transform the output of roc
Luke Boswell (Aug 04 2024 at 02:10):So roc test could take a .roc plugin which gets the results of the test run?
Luke Boswell (Aug 04 2024 at 02:11):Love it
Last updated: Jun 16 2026 at 16:19 UTC