Stream: contributing

Topic: Test discovery

Jonathan Schear (Mar 03 2024 at 22:54):

I'm interested in working on https://github.com/roc-lang/roc/issues/6490: "roc test platform/main.roc should test every file referenced in exposes and the respective imports of those files too" (and the related https://github.com/roc-lang/roc/issues/6491) and looking for a bit of guidance on the desired behavior when running roc test.

I think the first part of 6490 ("should test every file referenced in exposes") is already implemented -- when running roc test platform/main.roc for https://github.com/roc-lang/basic-webserver/blob/main/platform/main.roc, I see 60 tests executed, which appear to be 14 from Http.roc, 6 from Url.roc, and 40 from Utc.roc. But I'm not sure if the latter part ("and the respective imports of those files too") is functional yet -- working on a toy project to test that.

Is this specifying special behavior that should only apply to platforms, or in general should roc test file.roc run all expects within any modules that file.roc contains in its exposes and imports, recursively? (It feels like it might be nice to be able to run _only_ the expects in a certain file too, in cases where the transitive closure of all modules' expects could be a lot of tests.)

Richard Feldman (Mar 03 2024 at 23:30):

yeah, should work on any module!

Richard Feldman (Mar 03 2024 at 23:30):

thanks for taking this on! :smiley:

Jonathan Schear (Mar 04 2024 at 03:45):

For sure. A bit of a newcomer to Rust and the domain so we'll see how far I get :)

I'm seeing some code about expect-fx, sounds like those will eventually be for tests that may have side-effects?

Richard Feldman (Mar 04 2024 at 03:49):

yeah exactly!

Jonathan Schear (Mar 04 2024 at 05:51):

A few questions on the ideal output:

1. If we're listing each module & the number of tests that passed, failed, and the duration they took, should these messages be interspersed with any failures, or should they all be in a summary at the end?

2. If expects from only a single module are discovered, should we display the name of the module, or go with the current 0 failed and 2 passed in 129 ms. for simplicity?

Jonathan Schear (Mar 04 2024 at 05:52):

For 1, it would be the difference between:

❯ roc test ../roc-parser/package/main.roc
── EXPECT FAILED in ../roc-parser/package/String.roc ───────────────────────────

This expectation failed:

289│  expect parseStr parseNumbers "1,2,3" == Ok [1, 2, 4]
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

String.roc:
    1 failed and 12 passed in 43ms.

── EXPECT FAILED in ../roc-parser/package/HTTP.roc ─────────────────────────────

This expectation failed:

73│>  expect
74│>      actual = parseStr httpVersion "HTTP/1.2"
75│>      expected = Ok { major: 1, minor: 1 }
76│>      actual == expected

When it failed, these variables had these values:
...

HTTP.roc:
    0 failed and 1 passed in 46 ms.

and

~/dev/roc-parser main*
❯ roc test ../roc-parser/package/main.roc
── EXPECT FAILED in ../roc-parser/package/String.roc ───────────────────────────

This expectation failed:

289│  expect parseStr parseNumbers "1,2,3" == Ok [1, 2, 4]
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

── EXPECT FAILED in ../roc-parser/package/HTTP.roc ─────────────────────────────

This expectation failed:

73│>  expect
74│>      actual = parseStr httpVersion "HTTP/1.2"
75│>      expected = Ok { major: 1, minor: 1 }
76│>      actual == expected

When it failed, these variables had these values:

...

String.roc:
    1 failed and 12 passed in 43ms.
HTTP.roc:
    0 failed and 1 passed in 46 ms.

I lean towards the latter :man_shrugging:

Brendan Hansknecht (Mar 04 2024 at 06:27):

Printing all at the end (your later example) or early exiting on first failure (likely with a flag) sound like the way to go.

Aggregating that info is most useful.

Brendan Hansknecht (Mar 04 2024 at 06:27):

Interspersed is simply hard to follow

Anton (Mar 04 2024 at 10:06):

For 2. I think it's good to stick with the current approach. We can always change it if there is demand.

Jonathan Schear (Mar 04 2024 at 17:17):

I was able to get a test written for transitive imports, and it looks like they're working: roc test main.roc runs all three expects here:

# main.roc
package "transitive-tests"
    exposes [
        Direct,
    ]
    packages {}

# Direct.roc
interface Direct
    exposes [
        addAndStringify,
    ]
    imports [
        Transitive,
    ]

addAndStringify = \num1, num2 ->
    Num.toStr (Transitive.add num1 num2)

expect addAndStringify 1 2 == "3"

expect addAndStringify 3 4 == "7"

# Transitive.roc
interface Transitive
    exposes [
        add,
    ]
    imports []

add = \num1, num2 -> (num1 + num2)

expect add 1 2 == 3

Here's my general plan for implementing the reporting change, would love to hear if this makes any sense from anyone more familiar with the codebase:

• Currently expects and expects-fx are collected into VecMap<Symbol, Region>s on MonomorphizedModule, I'd collect them into maps keyed by module ID instead.
• I'd want to run each modules' expects separately in roc_repl_expect::run::run_toplevel_expects, so we can easily measure and report the elapsed time per module.

Alternatively, I guess I could leave MonomorphizedModule and friends as-is, and just do more bookkeeping around module ID, start time, and end time per test, then sum the elapsed time per module.

Richard Feldman (Mar 04 2024 at 18:05):

that proposed plan sounds great, this is awesome!!!

Last updated: Jul 06 2025 at 12:14 UTC