Stream: API design

Topic: Tasks returning {}

drew (Apr 20 2024 at 14:03):

Has there been any discussion for more convenient backpassing syntax when the task returns {}? I find it odd to have to write _ <- Stdout.line "hello" |> Task.await. I'm sure you all know that Haskell "automatically" handles IO () in the do notation. I'm guessing this has been thought about and rejected?

Anton (Apr 20 2024 at 14:35):

Hi drew,
We created the ! syntax so you can avoid writing _ <-. I believe most of it is (very recently) implemented, I'm not sure what from the design doc is still missing.

drew (Apr 20 2024 at 14:37):

awesome thanks!

drew (Apr 20 2024 at 14:43):

oh wow, the ! design is really nice. i like that it means everything looks like the direct style.

drew (Apr 20 2024 at 14:46):

the with syntax feels slightly awkward, but i like that ! builds on with, and you'd _definitely_ want with for parsers imo.

drew (Apr 20 2024 at 14:46):

does with work for the entire following block? can you nest withs?

drew (Apr 20 2024 at 14:47):

one downside to removing backpassing is that if you want to interleave effects you need to use explicit callbacks, right?

Brendan Hansknecht (Apr 20 2024 at 14:55):

I think the current plan is to not add with unless it gains a really clear need.

Brendan Hansknecht (Apr 20 2024 at 14:55):

First we are making ! special to only tasks

Brendan Hansknecht (Apr 20 2024 at 14:56):

Next thing we would consider is making ! valid for any andThen style function on an opaque type (maybe a special case for result if needed). I don't think we can make it work with generic type aliases.

Brendan Hansknecht (Apr 20 2024 at 14:57):

After that, we would look into with if there is still need for more flexibility

Brendan Hansknecht (Apr 20 2024 at 14:57):

And yes, this means that callback need to be explicit otherwise. At least once back passing is removed

drew (Apr 20 2024 at 17:44):

got it, thanks.

drew (Apr 20 2024 at 18:05):

Brendan Hansknecht said:

Next thing we would consider is making ! valid for any andThen style function on an opaque type (maybe a special case for result if needed). I don't think we can make it work with generic type aliases.

how would this look for a non-Task type?

drew (Apr 20 2024 at 18:06):

does this mean that whatever the returned type is, if it could be subsequently passed to that module's andThen function, you could use !?

Brendan Hansknecht (Apr 20 2024 at 18:16):

This has been discussed some, but is not nailed down. A big question is if we want the complexity it could bring to the type system or simply to have a basic rewrite rule like what you mentioned by calling the module's andThen function.

drew (Apr 20 2024 at 18:26):

this couldn't be implemented directly via abilities because of higher-kinded types, yeah?

Brendan Hansknecht (Apr 20 2024 at 18:31):

Exactly

drew (Apr 20 2024 at 19:45):

just when you thought you were out, monads pull you back in

Brendan Hansknecht (Apr 20 2024 at 19:51):

Yeah, maybe. Though having a single monad for a very specific scope is very different from totally generic monad use cases.

Brendan Hansknecht (Apr 20 2024 at 19:51):

Though how ! and other syntax sugar work today, they are essentially macros. So they technically can skip past the type system and just generate something.

drew (Apr 20 2024 at 19:52):

totally agreed. i do think having ! work with user defined types seems better.

Brendan Hansknecht (Apr 20 2024 at 19:54):

+1

drew (Apr 20 2024 at 20:07):

the ! reminds me a bit of letops in ocaml

Richard Feldman (Apr 20 2024 at 20:27):

I'm curious to see what specific use cases (if any) remain for user-defined types once we have shadowing and can use that for random number seeds

Richard Feldman (Apr 20 2024 at 20:29):

the seed-passing style is very straightforward and composes trivially with tasks; the main ergonomics problem there is naming

Richard Feldman (Apr 20 2024 at 20:30):

whereas the "build up a big Generator" style clashes with tasks regardless of whether it has ! sugar - so I think once we have shadowing that will likely end up being the recommended style to use anyway

Richard Feldman (Apr 20 2024 at 20:31):

and I don't think ! would solve a significant pain point in parsers - e.g. Elm has neither that nor a backpassing equivalent and there doesn't seem to be demand for anything like that in Elm parsers

Richard Feldman (Apr 20 2024 at 20:32):

so if we have shadowing I'm not really sure what the specific use cases would be, what the delta in experience would be between having ! work and not work with those types, etc.

drew (Apr 20 2024 at 21:35):

interesting. what’s the “seed passing style”?

drew (Apr 20 2024 at 21:35):

and what do you mean by shadowing in this context?

Jasper Woudenberg (Apr 20 2024 at 22:02):

Richard Feldman said:

I'm curious to see what specific use cases (if any) remain for user-defined types once we have shadowing and can use that for random number seeds

I was thinking that if we have !-syntax for custom types, it might come in useful in the design of roc-ci. To replace the backpassing I'm using in the current design. This is an example pipeline definition in the current design:

buildAndTest : Ci.Job
buildAndTest =
    repoDetails <- Ci.step0 "setup git" Ci.setupGit
    binary <- Ci.step1 "build binary" buildBinary repoDetails
    testsPass <- Ci.step1 "run tests" runTests binary
    _ <- Ci.step2 "release" release binary testsPass
    Ci.done

This is using backpassing to do what would be called a writer monad in Haskell. Every CI step you create is recorded into some internal data structure, from which the CI runner can later extract it.

If the seed-passing-style is what I think it is, I imagine that might be an alternative here too. The internal data structure containing the steps would be passed explicitly around:

buildAndTest : Ci.Job
buildAndTest =
    job = Ci.emptyJob
    (repoDetails, job) = Ci.step0 job "setup git" Ci.setupGit
    (binary, job) = Ci.step1 job "build binary" buildBinary repoDetails
    (testsPass, job) = Ci.step1 job "run tests" runTests binary
    (_, job) = Ci.step2 job "release" release binary testsPass
    job

There is a downside to this approach though: it becomes possible to accidentally not record a CI step, particularly when you're not interested in the result of the CI step and so are _-ing part of the return value of that step anyway. For instance, this would be a version of the previous example where the release step at the end is dropped:

buildAndTest : Ci.Job
buildAndTest =
    job = Ci.emptyJob
    (repoDetails, job) = Ci.step0 job "setup git" Ci.setupGit
    (binary, job) = Ci.step1 job "build binary" buildBinary repoDetails
    (testsPass, job) = Ci.step1 job "run tests" runTests binary
    _ = Ci.step2 job "release" release binary testsPass
    job

Of course entirely different designs are possible too!

drew (Apr 20 2024 at 22:09):

One big outstanding question I have about roc is how application-wide config / data will be handled. I don’t love the Reader monad but I’m not sure what other options are available. Many use Task and rely on a specific effect from the platform?

Jasper Woudenberg (Apr 20 2024 at 22:20):

Another scenario where I think having ! for custom types could come in useful: defining a custom Task type wrapping the future builtin, with tracing support.

Suppose you want to create a platform that contains the following primitive for recording a span in a trace:

span : Str, Task val err -> Task val err

In words: this takes a task and a string description, and would record how long the task takes to run. The platform could then add support for sending this data to some tracing platform where you can look at a flamegraph of your code.

For this to work the Task would need to carry some metadata related to the current span:

• Either a unique ID for the current span or a mutable list of child spans, so the relation between spans in the trace tree is recorded.
• A timestamp representing the start time of the current span so we can calculate its duration when it ends.

In the current design where platforms define their own Task type, platforms can define it to carry whatever metadata they like. If this is no longer possible because Task becomes a builtin (I'm not sure about this), another approach might be to define a custom Task type by wrapping the builtin one, though if ! is not available for the custom Task type it would not be very ergonomic.

Richard Feldman (Apr 20 2024 at 23:34):

drew said:

One big outstanding question I have about roc is how application-wide config / data will be handled. I don’t love the Reader monad but I’m not sure what other options are available. Many use Task and rely on a specific effect from the platform?

for this use case I agree with Evan's advice for what to do in Haskell (where Reader and others are available): just pass it around!

drew (Apr 20 2024 at 23:35):

Richard Feldman said:

drew said:

One big outstanding question I have about roc is how application-wide config / data will be handled. I don’t love the Reader monad but I’m not sure what other options are available. Many use Task and rely on a specific effect from the platform?

for this use case I agree with Evan's advice for what to do in Haskell (where Reader and others are available): just pass it around!

Doesn't this mean passing something like a database connection to most non-pure functions in a large-ish web application?

Richard Feldman (Apr 20 2024 at 23:36):

yes, although with module params the passing doesn't have to be function-to-function, it can be module-to-module

Richard Feldman (Apr 20 2024 at 23:37):

I suspect that would be the more common way to pass it around in Roc

Richard Feldman (Apr 20 2024 at 23:37):

@Jasper Woudenberg could it make sense in a CI platform to use Task instead of Job?

Richard Feldman (Apr 20 2024 at 23:38):

alternatively, could an applicative-based API for Job make sense? Then you could use record builder syntax, and also the structure would be statically inspectable (so you could print out a build graph etc)

timotree (Apr 20 2024 at 23:55):

Jasper Woudenberg said:

There is a downside to this approach though: it becomes possible to accidentally not record a CI step, particularly when you're not interested in the result of the CI step and so are _-ing part of the return value of that step anyway. For instance, this would be a version of the previous example where the release step at the end is dropped:

buildAndTest : Ci.Job
buildAndTest =
    job = Ci.emptyJob
    (repoDetails, job) = Ci.step0 job "setup git" Ci.setupGit
    (binary, job) = Ci.step1 job "build binary" buildBinary repoDetails
    (testsPass, job) = Ci.step1 job "run tests" runTests binary
    _ = Ci.step2 job "release" release binary testsPass
    job

Would it make sense to catch this by linting _ =? I don't think there is much of a reason to write _ = in Roc if it means performing a pure computation and this discarding the result

Richard Feldman (Apr 20 2024 at 23:58):

yeah I think that's a reasonable warning in general :thumbs_up:

Brendan Hansknecht (Apr 21 2024 at 00:05):

could it make sense in a CI platform to use Task instead of Job?

Just thinking out loud of use cases that I expect to hit this (not that they need !, but they could use backpassing today and ! would be really nice):

1. Task
2. Any higher level abstraction that wraps Task
3. Result: at least in complex functions without effects/Task
4. Generators: sure rng can use shadowing and explicit state passing, but generators are nice. They also come in for arbitrary data generation from bytes like used in fuzzing.
5. Streams with manual continuations might also fit to some extent

Brendan Hansknecht (Apr 21 2024 at 00:07):

Not sure if there are any common higher level wrappers of Result that would benefit from this.

Richard Feldman (Apr 21 2024 at 00:27):

brief thoughts on those:

1. (of course Task is definitely going to support it no matter what :big_smile:)
2. "Any higher level abstraction that wraps Task" - in the past I've seen this used for simulating effects in tests, and logging effects, and in Roc we can do both of those without requiring wrappers. (Logging can be done by providing a function to the platform, which it can then call when executing the Task state machine.) I haven't heard of other use cases for wrapping Task that seem like a good idea, so I think the specifics matter on this one!
3. Result - neither Elm nor OCaml have sugar for this and it hasn't been an issue; I think it's fine if Roc doesn't have sugar for it either. Plus Result isn't opaque in Roc, so it can't get abilities, making it innately awkward for any kind of "make ! work with any type that has a particular ability" design. If we really want it just for Result we can always discuss a ? suffix in addition to !
4. Generators - to be clear, I think Generator is a good idea for reuse (and in that use case, it doesn't clash with Task), I'm specifically talking about the style of "running" the RNG where you wrap all the logic you want to do in a big Generator that you have no intention of reusing, just as a way of threading the seed around. That's the situation where you can absolutely clash with Task, and where using seed instead doesn't have that problem.
5. Streams - is that stream in the sense of "iterators," or in the sense of like "streaming I/O"?

Brendan Hansknecht (Apr 21 2024 at 00:36):

Logging can be done by providing a function to the platform, which it can then call when executing the Task state machine.

That may not give you the selective logging control you want. It also makes logging depend on the platform. It would be much nicer to build a logging library in roc that creates some form of logging task that simply has to know what effect to run for various kinds of logging. So I do think there is a rather large unlock here from allowing the wrapping.

I haven't heard of other use cases for wrapping Task that seem like a good idea

Yeah, not fully sure, the CI Job and general graph building + task style seems to be where this would most likely come up.

I'm specifically talking about the style of "running" the RNG where you wrap all the logic you want to do in a big Generator that you have no intention of reusing, just as a way of threading the seed around. That's the situation where you can absolutely clash with Task, and where using seed instead doesn't have that problem.

I was specifically think about where you define a generator per type and they are composable. So an outer generator composes many sub generators. I guess it probably could all be direct calls. Past hiding state, I'm not sure if there is a major difference in apis.

is that stream in the sense of "iterators,"

Kinda. I was more specifically thinking lazy data creation. Made by a chain of continuations.

drew (Apr 21 2024 at 01:28):

Richard Feldman said:

3. Result - neither Elm nor OCaml have sugar for this and it hasn't been an issue; I think it's fine if Roc doesn't have sugar for it either. Plus Result isn't opaque in Roc, so it can't get abilities, making it innately awkward for any kind of "make ! work with any type that has a particular ability" design. If we really want it just for Result we can always discuss a ? suffix in addition to !

letops are a syntax for this in ocaml that can work with user-defined types, right? and let-ppx was before letops were directly supported.

Richard Feldman (Apr 21 2024 at 04:39):

Brendan Hansknecht said:

Logging can be done by providing a function to the platform, which it can then call when executing the Task state machine.

That may not give you the selective logging control you want. It also makes logging depend on the platform. It would be much nicer to build a logging library in roc that creates some form of logging task that simply has to know what effect to run for various kinds of logging. So I do think there is a rather large unlock here from allowing the wrapping.

there's a deeper discussion to be had on the topic of logging, but briefly - given that platforms can already log all I/O (and give application authors a way to specify how they want each particular I/O operation to be logged), I'm skeptical that it's really beneficial to be able to say "okay this exact HTTP request should be at a different log level than this exact other HTTP request" as opposed to (for example) filtering the logs in some other way, or setting the level as a function of information contained in the the HTTP request itself (e.g. if the URL is going to some known third-party analytics domain, those requests can always be set to a lower log level)

Richard Feldman (Apr 21 2024 at 04:39):

regarding OCaml, I actually don't know anything about letops or let-ppx :sweat_smile:

Brendan Hansknecht (Apr 21 2024 at 04:57):

We'll have to see in practice once we have module params. My gut feel is that platform logging support will be adhoc and inconsistent. Including, I would guess that many platforms won't have proper logging at all. As such, it will be nicer to just call Stderr.line or File.write or Http.push to send logs. Then it will be nicer to wrap that up in a shared library that takes in the logging primitive as a module param task. I think that is the most likely way to get a really nice and consistent logging experience that can be attached to any platform with some sort of text output primitive.

Brendan Hansknecht (Apr 21 2024 at 04:59):

That said, I'm not sure that you will actually need any special form of wrapping task for that.

Brendan Hansknecht (Apr 21 2024 at 04:59):

So it may not need any sort of special !

drew (Apr 21 2024 at 16:23):

Richard Feldman said:

regarding OCaml, I actually don't know anything about letops or let-ppx :sweat_smile:

Here's a little example I put together to show how letops work -- https://gist.github.com/drewolson/5405ad83e5303026650d4015d185490c

drew (Apr 21 2024 at 16:24):

It feels worth explaining as letops actually feel pretty close to what you're describing with ! -- they work as basically "rewrites". They effectively make callback based code look like the direct style, using whatever definition of let* (or other let-style functions) are in scope.

drew (Apr 21 2024 at 16:25):

in the example I show using let* (direct-style bind) for both lists and options, just as an example

drew (Apr 21 2024 at 16:25):

the annoying downside is that you must explicitly bring into scope the particular implementation you want for a given function

drew (Apr 21 2024 at 16:25):

that said, at least it is explicit. it feels very similar to the proposal for with and ?.

drew (Apr 21 2024 at 16:29):

the same tools are used for applicative-style APIs via the conventionally named let+ and and+

drew (Apr 21 2024 at 16:41):

here's an example of an applicative-style parser using letops and the angstrom library https://gist.github.com/drewolson/1776047f9ac00d51df8a7a36deb2fda6

drew (Apr 21 2024 at 16:43):

the comparisons to ocaml feel useful as it is a language without ad hoc polymorphism at all :) but i get that the design goals of roc differ. just wanted it here as a comparison / prior art.

drew (Apr 21 2024 at 16:46):

Writing this small parser also made me remember how much i dislike full type inference in ocaml, related to one of the disadvantages of HKP in the FAQ. If I accidentally forget the labeled argument to Angstrom.parse_string, here's the helpful error message I get:

drew (Apr 21 2024 at 16:47):

File "bin/parse/main.ml", line 21, characters 4-11:

21 |   | Error _ -> print_endline "Error!"

         ^^^^^^^

Error: This pattern should not be a constructor, the expected type is

       consume:Angstrom.Consume.t -> (int * int, string) result```

drew (Apr 21 2024 at 16:47):

though i suppose this is actually related to currying :)

Jasper Woudenberg (Apr 21 2024 at 17:41):

Richard Feldman said:

Jasper Woudenberg could it make sense in a CI platform to use Task instead of Job?

Richard Feldman said:

alternatively, could an applicative-based API for Job make sense? Then you could use record builder syntax, and also the structure would be statically inspectable (so you could print out a build graph etc)

Yeah, I'd like to make the CI structure be statically inspectable, which is why I don't think using Task instead of Job would help. With regards to record builders, I've thought about it after getting that feedback before, but I'm not sure what record I'd want to build :sweat_smile:. Would the idea be to try represent the build graph as a set of nested records?

What I like about the design I have now is that it's already statically inspectable. on account of Job not coming with an andThen. And it allows you to write the project's CI almost like an ordinary function, with roc-ci being able to turn that into a pipeline (i.e., we can automatically figure out which bits to run in paralllel).

Richard Feldman (Apr 21 2024 at 17:46):

Jasper Woudenberg said:

What I like about the design I have now is that it's already statically inspectable. on account of Job not coming with an andThen. And it allows you to write the project's CI almost like an ordinary function, with roc-ci being able to turn that into a pipeline (i.e., we can automatically figure out which bits to run in paralllel).

ah! So ! desugars into an andThen, so it probably wouldn't be usable here even if it did support types other than Task :big_smile:

Jasper Woudenberg (Apr 21 2024 at 17:48):

So logging is one thing, but any thoughts about what it would take to have good tracing support? For me, traces have completely replaced logging in web applications, and I'd be happy never to have to think off log levels again :sweat_smile:. I think in terms of functionality required, while logging could be implemented by the platform providing some log function(s), for tracing the Task type needs to carry some context to allow a tree of spans to be created. Will that be possible with the language-provided Task type?

Jasper Woudenberg (Apr 21 2024 at 17:51):

Richard Feldman said:

ah! So ! desugars into an andThen, so it probably wouldn't be usable here even if it did support types other than Task :big_smile:

Ah, that makes sense, I guess that would never work then.

I'll probably go with the shadowing approach then, seems pretty straight-forward and I think it'd likely have better errors then the current backpassing approach anyway.

Richard Feldman (Apr 21 2024 at 18:17):

cool cool!

Richard Feldman (Apr 21 2024 at 18:18):

out of curiosity, how do you think you'd do it in Elm? :thinking:

Jasper Woudenberg (Apr 21 2024 at 18:57):

Elm uses Task so little, I'm not sure there'd be a ton of value in instrumenting it (given it'd leave most of the application out of scope). That said, if I had to, I'd probably introduce a custom Task type, give it the same API as the built-in task. That custom Task type could have a structure like this:

type TracingTask err val = TracingTask (TraceContext -> Task err val)

type alias TraceContext = { id: String, startTime : Time }

For a more full-formed example, we defined a Task type in a similar fashion around IO in nri-haskell:
https://github.com/NoRedInk/haskell-libraries/blob/trunk/nri-prelude/src/Platform/Internal.hs#L63

Richard Feldman (Apr 21 2024 at 18:58):

oh I meant modeling Job in Elm

Richard Feldman (Apr 21 2024 at 18:59):

since Elm doesn't have shadowing

Jasper Woudenberg (Apr 21 2024 at 19:05):

Richard Feldman said:

oh I meant modeling Job in Elm

Ah gotcha. Yeah, good question. Maybe define Job.map2 and Job.map3 and the like?

Nice question, I'll think a bit about this.

drew (Apr 22 2024 at 04:54):

in playing around with !, i've found that ending a program in the basic-cli with _two_ statements that both use ! works fine, but a single statement ending in ! to end the program produces a type error

drew (Apr 22 2024 at 04:54):

see https://gist.github.com/drewolson/635f4c39bd209338d3f6e5030720f1a9

drew (Apr 22 2024 at 04:55):

output from the non-working case:

drew (Apr 22 2024 at 04:55):

$ roc dev

── TYPE MISMATCH in main.roc ───────────────────────────────────────────────────

This 2nd argument to this function has an unexpected type:

22│              Stdout.line! "Hello, $(name)"

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The argument is an anonymous function of type:

    {} -> {}

But this function needs its 2nd argument to be:

    {} -> InternalTask.Task c *

Tip: Type comparisons between an opaque type are only ever equal if

both types are the same opaque type. Did you mean to create an opaque

type by wrapping it? If I have an opaque type Age := U32 I can create

an instance of this opaque type by doing @Age 23.

────────────────────────────────────────────────────────────────────────────────

1 error and 0 warnings found in 27 ms

.

You can run the program anyway with roc run```

Luke Boswell (Apr 22 2024 at 04:56):

Thank you, can you please log an issue. I'm pretty sure I know what is going wrong in this case

Luke Boswell (Apr 22 2024 at 04:56):

Just a link to the gist is all the description needed

drew (Apr 22 2024 at 04:56):

sure, make the issue in the roc repo?

drew (Apr 22 2024 at 04:57):

done, it's here https://github.com/roc-lang/roc/issues/6661

Last updated: Jul 06 2025 at 12:14 UTC