Brian Hicks (Nov 26 2021 at 18:17): Brian Hicks (Nov 26 2021 at 18:17):has anyone done any work on unit testing in Roc?
Brian Hicks (Nov 26 2021 at 18:20):made a Roc specification platform maybe? (rspec for short)
Anton (Nov 26 2021 at 18:45):Yes, Jim did in PR #1590
Brian Hicks (Nov 26 2021 at 19:01):niiiice
Brian Hicks (Nov 26 2021 at 19:02):know anything about fuzzing? (Actual fuzzing, not property testing)
Anton (Nov 26 2021 at 19:14):@Brendan Hansknecht started on fuzzing (in rust) in compiler/parse/fuzz. But I don't think anyone implemented fuzzing in roc.
Richard Feldman (Nov 26 2021 at 19:32):there's a design for baking testing into the language, but nobody's working on the implementation at the moment :big_smile:
Brian Hicks (Nov 29 2021 at 16:50):@Richard Feldman I think you maybe meant the Advent of Code topic :sweat_smile:
Richard Feldman (Nov 29 2021 at 17:25):ha, yup! fixed!
Dillon Kearns (Nov 30 2021 at 00:30):Richard Feldman said:
there's a design for baking testing into the language, but nobody's working on the implementation at the moment :big_smile:
Is that written down anywhere, or just an idea at this point? I got roc setup and the first thing on my mind is always testing, so I'd be curious to explore this more!
One of the things I'm curious about is if you picture the idea of a Platform being something that Roc's testing can swap in for data that it can make assertions against (rather than actually performing things in the platform). Not sure if it's as simple as that, but I'm very curious to discuss and contribute if I can!
Tim Whiting (Nov 30 2021 at 01:10):Interesting idea, essentially all effects happen through the platform, so it's a great boundary for testing.
Dillon Kearns (Nov 30 2021 at 16:54):Yeah, I think we can learn a lot from https://package.elm-lang.org/packages/avh4/elm-program-test/latest. It gives a way to simulate side effects. The way it's done is by wrapping Elm side-effects with an Effect type since currently Elm's Cmds and Tasks can't be inspected. Evan and Aaron were exploring having a way for tests to inspect those side-effects as data, which would allow for testing without creating an Effect type wrapper.
Brendan Hansknecht (Nov 30 2021 at 17:07):One of the things I'm curious about is if you picture the idea of a Platform being something that Roc's testing can swap in for data that it can make assertions against (rather than actually performing things in the platform). Not sure if it's as simple as that, but I'm very curious to discuss and contribute if I can!
I know we have discussed that exact idea a few times and I think it is definitely a goal, but I don't think anyone has really dug into it yet. Could theoretically even interact with your app while in some sort of "record" mode and then use that to generate test cases.
Dillon Kearns (Nov 30 2021 at 17:26):I like the idea of "record" mode, that's a nice way of putting it.
One of the biggest benefits of managed effects and pure functions is testability. A lot of the "TDD is Dead" talk from people like DHH are, in my opinion, reactions to the very painful experience of mocking side-effects and environment. So a great testing story is a killer feature. A lot of that comes for free because you don't need to mock pure functions.
But for the managed effects, there does need to be some way to swap things in and "record" effects. For elm-program-test, it's a more tailored way of recording the specific effects that can happen in the Elm platform. With Roc, since Platforms are swappable, it would need to be able to give a way to provide a test harness specific to a given platform.
Mario (Nov 30 2021 at 18:33):@Martin Stewart has done some pretty rad stuff with program-test, expanding it to a full stack version for Lamdera. He already demoed a "time travelling debugger" style replay, so seems logical to go a step further with some sort of record. I think whatever context that's applied in will be a boon to making it easier / more likely to "write" (read: record) acceptance tests.
jan kili (Aug 28 2022 at 07:53):I'm trying out the expect keyword for the first time, and I'm confused - what tests are running here?
$ roc test examples/hello-world/main.roc
0 failed and 5 passed in 729 ms.
I don't see any expects in the two .roc files involved...
Brian Carroll (Aug 28 2022 at 08:08):There are other .roc files involved too - all the ones in the standard library! There are three expects in Str.roc and two in Set.roc. At the moment there's nothing to filter out the ones in the std lib and only run the ones for the app. But longer term, we should implement that!
jan kili (Aug 28 2022 at 08:48):I see! Thank you for that insight.
jan kili (Aug 28 2022 at 08:50):New question, does/will roc test have any verbosity settings that could display the values of the left and right sides of an equality check? I'd like to know what the left side of this is, for example:
$ roc test 3.roc
── EXPECT FAILED ─────────────────────────────────────────────────────── 3.roc ─
This expectation failed:
24│ expect (calculatePrimeFactors 2) == [2]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1 failed and 7 passed in 781 ms.
We do have this old issue where the printing of intermediary values is discussed.
Brian Carroll (Aug 28 2022 at 09:05):I would think you'd need to implement expectEqual instead of expect in order to do this. The expect function can only really print its argument, which is True or False. Anything more than that requires lots of fancy compiler magic which that issue is describing in detail. If instead you had expectEqual it would have two arguments so it could print both.
But... then you also have to do expectGreaterThan and all that kind of stuff too, so instead of one or two magic functions, there's a whole family of them.
Although if you had something like Elm's Debug.log then you could presumably build expectEqual out of that and expect.
Brian Carroll (Aug 28 2022 at 09:07):The logging function would then have to be somehow prevented from being used in production, etc.
Brian Carroll (Aug 28 2022 at 09:08):I guess in that design you'd have a whole Expect module in the standard library.
Brian Carroll (Aug 28 2022 at 09:09):And expect already works differently in production than test (it's ignored)
Richard Feldman (Aug 28 2022 at 14:02):yeah I'd like to print the values of all the arguments to the outermost function call in the expect
Richard Feldman (Aug 28 2022 at 14:04):(e.g. expect 5 == 5 would desugar to expect Bool.isEq 5 5 so we'd print the two arguments to Bool.isEq)
Richard Feldman (Aug 28 2022 at 14:05):that would take care of all the comparisons you might want to do, not just ==
Richard Feldman (Aug 28 2022 at 14:05):nested function calls would be trickier I imagine, e.g. 1 == 1 || 2 == 3
Ayaz Hafiz (Aug 28 2022 at 14:12):Currently it's such that named variables in the expect expressions are printed
── EXPECT FAILED ─────────────────────────────── examples/hello-world/main.roc ─
This expectation failed:
6│> expect
7│> a = "f"
8│> b = "g"
9│> a == b
When it failed, these variables had these values:
a : Str
a = "f"
b : Str
b = "g"
That's wonderful, Ayaz! :party_ball:
Ayaz Hafiz (Aug 28 2022 at 18:43):well @Folkert de Vries did all this, ups to Folkert
Qqwy / Marten (Aug 28 2022 at 18:48):Folkert! :green_heart:
Folkert de Vries (Aug 28 2022 at 18:49):Lots of fun code went into making it too
jan kili (Aug 29 2022 at 00:45):I just went through my code and added a BUNCH of multi-line expects, and it actually caught some regressions from renamed builtins. Thanks, y'all! :smiley:
jan kili (Aug 29 2022 at 00:46):(and as an added bonus, that process also uncovered a formatter bug! https://github.com/roc-lang/roc/issues/3924#issuecomment-1229604782 haha)
jan kili (Aug 29 2022 at 00:56):What canonical formatting do we want for code like this?
computeFibonacciNumbersUpTo = \maximum ->
more = \olds ->
last2 = List.sublist olds { start: List.len olds - 2, len: 2 }
next1 = List.sum last2
news = List.append olds next1
if next1 > maximum then olds else more news
more [1, 2]
expect
fibs = computeFibonacciNumbersUpTo 5
fibs == [1, 2, 3, 5]
expect
fibs = computeFibonacciNumbersUpTo 20
fibs == [1, 2, 3, 5, 8, 13]
In a file with many function definitions, I want to add a few expects after each function definition, like above. However, this is what I expect the formatter to do, which looks weird to me:
computeFibonacciNumbersUpTo = \maximum ->
more = \olds ->
last2 = List.sublist olds { start: List.len olds - 2, len: 2 }
next1 = List.sum last2
news = List.append olds next1
if next1 > maximum then olds else more news
more [1, 2]
expect
fibs = computeFibonacciNumbersUpTo 5
fibs == [1, 2, 3, 5]
expect
fibs = computeFibonacciNumbersUpTo 20
fibs == [1, 2, 3, 5, 8, 13]
Do I just need to get used to the pre-return-value newline? #EmbraceTheWhitespace?
Richard Feldman (Aug 29 2022 at 02:02):personally the second code snippet looks right to me!
Brian Carroll (Aug 29 2022 at 06:33):To me the blank line feels quite excessive when there's only one let-binding above it. I would never do that manually and I don't like when the Roc formatter does it. It always feels to me like our formatter is crudely applying a rule that doesn't really fit this case because it's not smart enough. But maybe that's not the case and it's intended because different people have different preferences.
Some formatters allow you a certain degree of freedom with blank lines. I think the Rust formatter lets you insert one blank line in a function body but if you have two consecutive blank lines it merges them.
Brian Carroll (Aug 29 2022 at 06:47):Do we have any flexibility in ours? Could this become something that the user can control?
Brian Hicks (Aug 29 2022 at 11:07):are you talking about the blank line between fibs = … and fibs == …? That one actually makes sense to me in a testing context—I tend to arrange my tests in arrange/act/assert style and use whitespace to separate the stages.
Brendan Hansknecht (Aug 29 2022 at 14:26):I think the biggest issue for me with that formatting is the that if you add a spacing before the last line, you need at least 2 spaces before the next definition. That being said, i definitely prefer no extra space in this case.
A:
expect
fibs = computeFibonacciNumbersUpTo 5
fibs == [1, 2, 3, 5]
expect
fibs = computeFibonacciNumbersUpTo 20
fibs == [1, 2, 3, 5, 8, 13]
B:
expect
fibs = computeFibonacciNumbersUpTo 5
fibs == [1, 2, 3, 5]
expect
fibs = computeFibonacciNumbersUpTo 20
fibs == [1, 2, 3, 5, 8, 13]
C:
expect
fibs = computeFibonacciNumbersUpTo 5
fibs == [1, 2, 3, 5]
expect
fibs = computeFibonacciNumbersUpTo 20
fibs == [1, 2, 3, 5, 8, 13]
So in the examples, i would say, i dislike A. I think B is ok but still not great. I would definitely personally write C.
Maybe if there were more lines in the definition or if there were curly braces instead of tabs, i would feel better about B, but neither of those are the case here.
Brendan Hansknecht (Aug 29 2022 at 14:28):I guess also if there were explicit comments specifying the stages, then more spacing would also make sense to me.
Richard Feldman (Aug 29 2022 at 14:39):interesting! I personally prefer A but I'm open to any of them
Richard Feldman (Aug 29 2022 at 14:40):I'm curious what others think
jan kili (Aug 29 2022 at 14:47):Yeah, if we want a hyperconsistent rule (not allowing C in short cases, for example) then I prefer B
jan kili (Aug 29 2022 at 14:47):I always personally do double newline between top-level function definitions in other languages
Richard Feldman (Aug 29 2022 at 15:54):I think that's orthogonal to be honest - I think we either ought to have 1 or 2 spaces between all top-level declarations regardless of how we format expect
Richard Feldman (Aug 29 2022 at 15:55):relatedly, how do people feel about the same thing except when it has to do with defs?
Richard Feldman (Aug 29 2022 at 15:56):e.g.
foo =
x = y + 1
x * 2
vs.
foo =
x = y + 1
x * 2
do people prefer the def with or without the blank line?
Richard Feldman (Aug 29 2022 at 15:56):(and I guess relevantly, is that preference the same as your preference for expect, or different?)
Brendan Hansknecht (Aug 29 2022 at 18:56):For me, exact same comments I made about expect.
Brian Carroll (Aug 29 2022 at 20:05):Ditto. There's nothing special for me about expect, I was talking generally about the blank line before the expression value.
jan kili (Aug 31 2022 at 02:08):To be honest, this feels like a growing avalanche of reasonable incremental expansions... but at it's core these tests should not take up this much space.
jan kili (Aug 31 2022 at 02:10):Options A, B, and C above are 5x, 6x, and 4x (respectively) the line count of what I initially wrote:
expect computeFibonacciNumbersUpTo 5 == [1, 2, 3, 5]
expect computeFibonacciNumbersUpTo 20 == [1, 2, 3, 5, 8, 13]
(We could call this Option D, I suppose, but I don't have a holistic formatting algorithm yet)
jan kili (Aug 31 2022 at 02:12):Here's another example, to illustrate the value of a concise/terse test format:
isPrime = \n ->
when n is
1 -> False
_ -> (calculateFactors n) == [1, n]
expect isPrime 1 |> Bool.not
expect isPrime 2
expect isPrime 5
expect isPrime 19
expect isPrime 20 |> Bool.not
expect isPrime 13195 |> Bool.not
I feel like this is maybe going farther from the original question, but your formatting made me realize something. Realistically in more other languages, would write a table drive test for these examples.
Just a list of tuples (or anonymous structs) of arguments to the function and expected results. Then a simple for loop of expects. Can we do something similar in roc without too much verbosity? Do we have plans for it?
jan kili (Aug 31 2022 at 02:14):(This latest example doesn't require multi-line expects because it doesn't contain any ==s to want to analyze.)
jan kili (Aug 31 2022 at 02:15):(but in general I want to write several tests for each few-line function, and those tests will usually contain a comparison like == or > etc)
jan kili (Aug 31 2022 at 02:15):@Brendan Hansknecht Great point!
jan kili (Aug 31 2022 at 02:18):Maybe something like
expect [
(isPrime 1, `==`, False),
(isPrime 2, `==`, True),
(isPrime 5, `==`, True),
(isPrime 19, `==`, True),
(isPrime 20, `==`, False),
(isPrime 13195, `==`, False),
]
This still makes room for defs inside a multi-lineexpect (not shown here), but LoC scales sublinearly with test count
jan kili (Aug 31 2022 at 02:20):Maybe expects?
Brendan Hansknecht (Aug 31 2022 at 02:20):Why repeat the method and comparison on every line?
Brendan Hansknecht (Aug 31 2022 at 02:21):In this case I guess it is readable, but in other cases it may be quite verbose.
jan kili (Aug 31 2022 at 02:22):Hmm...
expectsOf isPrime [
((1,), False),
((2,), True),
((5,), True),
((19,), True),
((20,), False),
((13195,), False),
]
Yeah, i was thinking something like that
Brendan Hansknecht (Aug 31 2022 at 02:22):Where isPrime would be any function/lambda.
jan kili (Aug 31 2022 at 02:23):I love it
jan kili (Aug 31 2022 at 02:23):We could still make room for non-== comparisons, but I like this multiplicity concept
jan kili (Aug 31 2022 at 02:24):If we don't make test syntax concise/terse/efficient & readable, I predict that we'll end up with foo.roc & foo.tests.roc files - not a terrible fate, but counter to the stated goals of the expect syntax.
jan kili (Aug 31 2022 at 02:31):Because these incrementally-reasonable formatting algorithm decisions are leading to unreasonable formatting:
rangeExclusive = \low, high ->
if low == high then [] else List.range low high
expect
values = rangeExclusive 1 1
values == []
expect
values = rangeExclusive 1 2
values == [1]
expect
values = rangeExclusive 1 3
values == [1, 2]
expect
values = rangeExclusive 2 3
values == [2]
expect
values = rangeExclusive 3 3
values == []
vs.
rangeExclusive = \low, high ->
if low == high then [] else List.range low high
expectsOf rangeExclusive [
((1, 1), []),
((1, 2), [1]),
((1, 3), [1, 2]),
((2, 3), [2]),
((3, 3), []),
]
(to be fair, I'm shifting formatter problems onto the compiler by proposing new keywords/behavior, but that might be the best approach when there's no great formatter-only solution)
Brian Carroll (Aug 31 2022 at 07:20):Hmm but a lot of the improvement is coming from eliminating that unnecessary values variable.
You could write the first set of examples more concisely like this
expect rangeExclusive 3 3 == []
I know there can be more complex cases where you need lots of variables.. but then the expectsOfcase wouldn't be as neat either!
Brian Carroll (Aug 31 2022 at 07:23):With today's syntax you can implement the expectsOf operator using List.map and List.all.
expect List.map [
((1, 1), []),
((1, 2), [1]),
((1, 3), [1, 2]),
((2, 3), [2]),
((3, 3), []),
] \((a, b), out) -> rangeExclusive a b == out
|> List.all
Then you could extract that logic as a helper function
Brian Carroll (Aug 31 2022 at 07:29):testLotsOfThings = \f, testCases -> List.map testCases (\((a, b), out) -> f a b == out) |> List.all
I think the values variable is only needed for now so it can be printed on test failure but I don't think we plan to require these variables in the long term right?
Anton (Aug 31 2022 at 09:30):Given how often they will be used I do see the value in introducing expectsOf (or perhaps expectAll).
expect isPrime 20 |> Bool.not also does not read well. expectFalse isPrime 20feels very nice.
Brendan Hansknecht (Aug 31 2022 at 13:55):Note on the list.map version. It loses two important properties that I would hope expectof would have.
Brian Carroll (Aug 31 2022 at 15:25):Ah yes, good points!
Last updated: Jul 26 2025 at 12:14 UTC