Stream: ideas

Topic: early return operator for Result

Sam Mohr (Jun 20 2024 at 22:55):

@Brendan Hansknecht I presume that the <- operator will only be removed if there's an ergonomic way to return early if an Err is returned from a Result, meaning there needs to be some version of ! available that does the equivalent of x <- res |> Result.try, right?

Sam Mohr (Jun 20 2024 at 22:56):

Also, is there a plan on how to implement that besides a special case that does Result.try / Result.andThen on [Ok _, Err _] tag unions?

Brendan Hansknecht (Jun 20 2024 at 23:16):

I presume that the <- operator will only be removed if there's an ergonomic way to return early if an Err is returned from a Result

Nope. At least not last I chatted with Richard. Cause if you look at ocaml and similar languages, they don't have sugar for result chaining and it doesn't tend to be a big issue in practice. I think this is mostly due to result chaining being way less common then task chaining. Also, with results, you can also use when is or handle them directly. This is not valid with tasks. That said, I'm sure there will be more discussion as we get closer to wanting to remove <-.

Brendan Hansknecht (Jun 20 2024 at 23:17):

besides a special case

Ah yeah, I always forget that Result isn't opaque. I think it would have to be a special case or separate symbol like the proposed ?.

Luke Boswell (Jun 20 2024 at 23:20):

Here is an example of how I am chaining a Result with Task's to exit early. This pattern has been working well for me.

Brendan Hansknecht (Jun 20 2024 at 23:21):

Yeah, works when in a merged world. The issue is really for pure libraries that use results.

Luke Boswell (Jun 20 2024 at 23:23):

For pure things, I like using a pipeline to combine them.

Brendan Hansknecht (Jun 20 2024 at 23:25):

Ah yeah, with result there is no need to nested Result.try you can just pipeline. Forgot that.

Brendan Hansknecht (Jun 20 2024 at 23:25):

Yeah, so still some clean solutions without <-

Luke Boswell (Jun 20 2024 at 23:25):

I would say it's early days though... the more I write Roc, the more of these things I'm discovering. So I'm sure there's a lot more to discover and maybe tune up a little

Sam Mohr (Jun 21 2024 at 02:08):

Nope. At least not last I chatted with Richard. Cause if you look at ocaml and similar languages, they don't have sugar for result chaining and it doesn't tend to be a big issue in practice.

I am just one dev, but I do write Rust for a living, and I believe even pure functions that I write in Rust are _much, much_ cleaner because I'm able to separate chained Result operations out into separate "definitions" (I know that the Rust ? works _basically_ like Roc's <-). It seems like a trap that a lot of newer FP devs run into (in my opinion) is being amazed at how much they can pipeline values into different functions, and then they'll do it with 10+ pipings of map and and_then. However, I think readability is greatly improved in many cases when I can see the significant intermediate steps that transformed values stop at.

Models.Session.isAuthenticated session.user |> Task.fromResult!

This kind of approach is a way to exploit the implementation of "!" for Task that is missing for Result, so it shows the value of providing "!" for Result.

For pure things, I like using a pipeline to combine them.

This definitely makes sense for short uses, but when we start getting big chains, it gets much more daunting IMO.

Sam Mohr (Jun 21 2024 at 02:13):

Not to pick on Luke, but I think this parser is made much more readable with the intermediate values pulled to definitions:

mayID =
    req.headers
    |> List.findFirst \reqHeader -> reqHeader.name == "Cookie"
    |> Result.mapErr \_ -> CookieHeaderNotFound
    |> Result.try \reqHeader ->
        reqHeader.value
        |> Str.fromUtf8
        |> Result.try \str ->
            str
            |> Str.split ";"
            |> List.findFirst \v -> v |> Str.trim |> Str.startsWith "$(cookieName)="
            |> Result.mapErr \_ -> CookieNameNotFound cookieName str
            |> Result.try \w ->
                w
                |> Str.split "="
                |> List.get 1
                |> Result.mapErr \_ -> NoEqualFound
                |> Result.try \v ->
                    v
                    |> Str.toU64
                    |> Result.mapErr \_ -> ValueNoInt v

and with exclams:

mayID =
    reqHeader = req.headers
        |> List.findFirst \reqHeader -> reqHeader.name == "Cookie"
        |> Result.mapErr! \_ -> CookieHeaderNotFound
    reqHeaderStr = Str.fromUtf8! reqHeader.value
    mayCookiePair = reqHeaderStr
        |> Str.split ";"
        |> List.findFirst \v -> v |> Str.trim |> Str.startsWith "$(cookieName)="
        |> Result.mapErr! \_ -> CookieNameNotFound cookieName reqHeaderStr
    mayCookieValue = mayCookiePair
        |> Str.split "="
        |> List.get 1
        |> Result.mapErr! \_ -> NoEqualFound

    Str.toU64 mayCookieValue
    |> Result.mapErr \_ -> ValueNoInt mayCookieValue

Richard Feldman (Jun 21 2024 at 02:16):

Sam Mohr said:

Nope. At least not last I chatted with Richard. Cause if you look at ocaml and similar languages, they don't have sugar for result chaining and it doesn't tend to be a big issue in practice.

I am just one dev, but I do write Rust for a living, and I believe even pure functions that I write in Rust are _much, much_ cleaner because I'm able to separate chained Result operations out into separate "definitions" (I know that the Rust ? works _basically_ like Roc's <-).

my experience is that ? for Result in Rust is almost always used for specifically handling I/O Results - which is basically the role that ! has in Roc

Sam Mohr (Jun 21 2024 at 02:17):

So I personally understand the argument that implementing ! just for Task is easier because we don't need an ability, and if we wanted to make an ability called AndThen that Task implemented it would still be hard to implement for Result because it's not an opaque type.

However, in my mind and experience, Task and Result are flip sides of the same result coin, one is simply the effectful version of the other. That's why "do notation" is used all over Haskell for Maybe and IO monads, not _just_ IO. I use both a good deal, and I think Roc code would be harder to read in large function contexts without it.

Sam Mohr (Jun 21 2024 at 02:18):

my experience is that ? for Result in Rust is almost always used for specifically handling I/O Results - which is basically the role that ! has in Roc

I don't see that, but I understand that it's hard for us to convince each other based on our personal experience.

Sam Mohr (Jun 21 2024 at 02:20):

I'm currently writing the equivalent of tax return software for my job, and handling all of the edge cases that come up is usually managed by returning appropriate errors and then ? them for the most part

Sam Mohr (Jun 21 2024 at 02:21):

If I had to manually bubble them up, especially early in the function definition, then most of my code would be "bubbling" code and not a series of recipe steps like "ensureThisIsValid" and "filterDownToUsefulData".

Sam Mohr (Jun 21 2024 at 02:25):

Guard statements are also something that are very valuable in Rust that aren't possible without some syntactical means for "early returns". In Roc, I can currently write:

ensureBarIsValid = \bar ->
    if isValidBar bar then
        Ok {}
    else
        Err BarIsInvalid

foo = \bar ->
    {} <- ensureBarIsValid bar
        |> Result.try

    # go ahead with bar

If I can't do that, I get code like:

foo = \nullableBar ->
    when nullableBar is
        Err BarIsNull -> ...
        Ok bar ->
            # 12 spaces now precede every line

Sam Mohr (Jun 21 2024 at 02:27):

Now, I don't mean to hijack this #beginners thread with rantings on a potentially beaten horse. I just find this feature a valuable add for readability and concision for what feels like an already-bitten bullet on beginner friendliness (since we already have ! for Task), and I'm not convinced it's worth forgoing.

Sam Mohr (Jun 21 2024 at 02:28):

If you'd rather forgo this discussion, or have it in an RFC, or something else, let me know.

Notification Bot (Jun 21 2024 at 02:30):

20 messages were moved here from #beginners > Task.attempt vs Task.result by Richard Feldman.

Richard Feldman (Jun 21 2024 at 02:30):

moved to a different topic!

Richard Feldman (Jun 21 2024 at 02:31):

so one of the ! proposal drafts mentioned the idea of a ? operator just for Result

Richard Feldman (Jun 21 2024 at 02:33):

Sam Mohr said:

I'm currently writing the equivalent of tax return software for my job, and handling all of the edge cases that come up is usually managed by returning appropriate errors and then ? them for the most part

this is pretty interesting, and I also think the indentation on your proposed parser example revision looks nicer for sure

Richard Feldman (Jun 21 2024 at 02:36):

back when the original document was proposed, we talked about the ? part of it, and decided that we should put that on the shelf in favor of discussing alternatives such as:

• just don't do it at all
• expand ! to be overloadable (which has downsides in terms of language complexity and which also necessarily raises separate design questions if using it for Result is a goal, because Result isn't opaque)

Richard Feldman (Jun 21 2024 at 02:37):

I was convinced for awhile that generalizing ! was the way to go, but having thought about it more, I'm currently thinking that's not the way to go after all

Sam Mohr (Jun 21 2024 at 02:37):

To give an example of my usage of the ? to filter data, I mean something like the following:

fn filter_valid_item(item: Item, today: Date) -> Result<ValidItem, ItemIsInvalid> {
    if item.start_date > today || item.end_date > today {
        return Err(ItemIsInvalid::InvalidDate);
    }

    let Some(quantity) = item.quantity else {
        return Err(ItemIsInvalid::MustHaveQuantity);
    };

    let price = item.calculate_average_price(today)?;

    Ok(ValidItem {
        id: item.id,
        quantity,
        price,
    })
}

This is a filter item to take a list of all "items" from our customer and provide the claimable list of items with their required data.

Sam Mohr (Jun 21 2024 at 02:38):

Richard Feldman said:

I was convinced for awhile that generalizing ! was the way to go, but having thought about it more, I'm currently thinking that's not the way to go after all

Yes, now having seen all the examples of ! in the wild for Roc, it seems like having it do Task.await and Result.try would be a bad mental overload.

Richard Feldman (Jun 21 2024 at 02:38):

for example, using ! for parsers and random number generators (Haskell-style) can be nice when using them on their own, but runs into problems as soon as you try to combine them with tasks (which is what leads to monad transformers in Haskell, and I extremely never want monad transformers to be a thing in Roc)

Richard Feldman (Jun 21 2024 at 02:39):

so an alternative stylistic option is to combine parsers without ! and pass seeds around for random number generation, which actually seems fine once we have shadowing

Sam Mohr (Jun 21 2024 at 02:40):

Yes, shadowing is a way to simplify the need for indentation arms races.

Richard Feldman (Jun 21 2024 at 02:40):

so I'm curious to revisit the original idea of ! for Task and ? for Result in light of these two use cases: lightweight parser Results (lightweight as in doing parsing without actually using an opaque Parser type - I do like the idea of most libraries that do parsing not needing to reach for a full-fledged Parser library) and the example use case of accounting software that deals with Result a lot

Sam Mohr (Jun 21 2024 at 02:42):

I'm actually always writing my own parser combinators because they're so simple, and it prevents consumers of my libraries from needing to download as many packages. This is a learning from all the Rust packages that boast "zero-deps".

Richard Feldman (Jun 21 2024 at 02:42):

in that world, the earlier parser example would look like this:

mayID =
    reqHeader = req.headers
        |> List.findFirst \reqHeader -> reqHeader.name == "Cookie"
        |> Result.mapErr? \_ -> CookieHeaderNotFound
    reqHeaderStr = Str.fromUtf8? reqHeader.value
    mayCookiePair = reqHeaderStr
        |> Str.split ";"
        |> List.findFirst \v -> v |> Str.trim |> Str.startsWith "$(cookieName)="
        |> Result.mapErr? \_ -> CookieNameNotFound cookieName reqHeaderStr
    mayCookieValue = mayCookiePair
        |> Str.split "="
        |> List.get 1
        |> Result.mapErr? \_ -> NoEqualFound

    Str.toU64 mayCookieValue
    |> Result.mapErr \_ -> ValueNoInt mayCookieValue

Sam Mohr (Jun 21 2024 at 02:42):

Yes, I would personally be _very_ happy with ! for Task and ? for Result, and NO other magic.

Sam Mohr (Jun 21 2024 at 02:42):

That looks awesome to me.

Richard Feldman (Jun 21 2024 at 02:43):

what do others think? I'd like to get some different perspectives on this!

Richard Feldman (Jun 21 2024 at 02:44):

I guess in this world, trailing ? would be optional just like trailing ! is?

Sam Mohr (Jun 21 2024 at 02:44):

I expect this wouldn't be that hard to implement, at least an MVP version.

Richard Feldman (Jun 21 2024 at 02:44):

so that last line could optionally be:

    Str.toU64 mayCookieValue
    |> Result.mapErr? \_ -> ValueNoInt mayCookieValue

Sam Mohr (Jun 21 2024 at 02:45):

Yes, I think whoever writes this could copy/duplicate the Task implementation, more or less.

Sam Mohr (Jun 21 2024 at 02:46):

I'll wait for someone(s) else to comment on this, and then I'll make a GitHub issue

Sam Mohr (Jun 21 2024 at 03:18):

Just to give another Roc example, I think this parsing code would be much less readable without the ability to return early via <- ... Result.try.

Brendan Hansknecht (Jun 21 2024 at 03:46):

Yeah, this is where I still quite like the power of <- even if it is much less common

Brendan Hansknecht (Jun 21 2024 at 03:47):

Like I would be totally happy keeping <- and just not teaching it until a super advanced tutorial.

Brendan Hansknecht (Jun 21 2024 at 03:47):

I really love using it for the long tail (like generators for fuzzing)

Brendan Hansknecht (Jun 21 2024 at 03:48):

Shadowing definitely makes it less needed, but I think it still can be nicer to not need to pipe the state arounf

Brendan Hansknecht (Jun 21 2024 at 03:48):

Though I guess passing state around with shadowing allows for more composibility.

Brendan Hansknecht (Jun 21 2024 at 03:50):

All this said, in certain code bases, I have seen a lot of ? for non io results in rust. I have also seen lots of code where early returning on error cases is the default for keeping code clean and easy to do follow. For pure code, that requires ? in roc. Cause we don't have early return equivalent otherwise.

Brendan Hansknecht (Jun 21 2024 at 03:51):

So I think we really should keep some for of solution for result.

Brendan Hansknecht (Jun 21 2024 at 03:51):

Probably simplest is ?.

Brendan Hansknecht (Jun 21 2024 at 03:54):

Also, I do generally question the idea of expanding ! at this point. I honestly think it might be better to keep <- then to expand ! to general use. Cause I think having a single mean may help with readability....that said, I would really have to see more example code with expanded ! to truly get a feeling for it.

Brendan Hansknecht (Jun 21 2024 at 03:56):

Anyway, that is my wall of thoughts. Feel like it is more meaning to just voice my general opinion than really vote one way or the other. Cause I think this is a story of nuance with limited code examples where we really would want to see each solution at scale before picking. All obviously will work, but none is clearly best.

The scale from simplicity to power.

Sam Mohr (Jun 21 2024 at 04:17):

The scale from simplicity to power.

This is the best encapsulation

Sam Mohr (Jun 21 2024 at 04:18):

I think the best thing about ? at this point in the discussion is that is gives us the benefits of <- without letting people do weird callback code exploits, in the same way that ! only working for Tasks prevents us from getting Monad'ed to death

Sam Mohr (Jun 21 2024 at 04:20):

And yeah, in the way that backpassing was introduced and seems to be on the way out after code was written enough to prove it wasn't needed, I think the fact that Roc doesn't even have numbered releases yet makes introducing ? non-committal

Norbert Hajagos (Jun 21 2024 at 15:09):

roc can be pretty heavy on indentation, so I used <- to reduce it, but mostly with Results, since it wasn't really readable with other constructs. I like the idea that ! and ? will do 1 thing.After extending !, you couldn't talk about Roc's simplicity, without a big BUT for that single operator.

I like passing around state as a beginner.
I was looking at a simple pseudo random package for haskell. Never wrote Haskell. The examples didn't make sense, because they used do notation and just called the random generator function repeatadly. If I knew less about haskell (namely that it is purely functional), I would just assume mutation of state after every call to get a new random value. Looked at a Roc example with explicit state passing, and there were no suprises there. Simple and easy to grok.
I know do is something for monads, but I only understood what it was doing after reading a Roc equivavelnt.

Long story short, I prefer separate ! and ? that do 1 thing.

Richard Feldman (Jun 21 2024 at 16:16):

I think it's important to note that <- is probably the hardest thing to learn in the language right now

Richard Feldman (Jun 21 2024 at 16:17):

whereas (given that ! is here to stay, one way or another) ? should be much easier to learn since it can be taught in terms of !

Richard Feldman (Jun 21 2024 at 16:17):

("! desugars to Task.await and ? desugars to Result.try")

Richard Feldman (Jun 21 2024 at 16:17):

whereas there is no helpful path to learn <- once you've already learned ! - as long as we have <-, it just continues to be the hardest thing to learn in the language

Richard Feldman (Jun 21 2024 at 16:18):

this surprises me, because I would not have guessed that <- would turn out to be as common a learning hurdle as it's turned out to be, but now that we know about that downside and also have !, I think the bar for keeping it has gone up a lot :big_smile:

Isaac Van Doren (Jun 21 2024 at 16:51):

I’m very in favor of adding ?. I use early returns all the time in the Java code I write at work and I think it makes it vastly more readable so I would like to be able to use the same style in Roc

Anton (Jun 21 2024 at 17:25):

I like ?, we should be able to re-use stuff from the ! implementation, so it seems low risk/cost as well.

Sam Mohr (Jun 21 2024 at 18:36):

Okay, it sounds like there's (at least some) community support. I'll make 2 issues later today:

• Add syntax sugar that translates ? to Result.try, in the same style as how ! desugars to Task.await.
  • Good error messages around type issues will be paramount
  • Are we planning on renaming Result.try?
• Deprecate the <-backpassing operator, I'm not seeing an existing issue for this. Let's do it in two phases:
  • After ? is implemented, then keep <- parsing and all for now, but raise a warning on every backpassing instance we find, saying "Backpassing will be removed from the language soon, please prefer ! for awaiting Tasks and ? for returning Err's early".
    • Look through GitHub for any Roc code that uses backpassing and raise an issue/contact the owner about removing it.
  • Once either enough time has passed or we are happy with the amount of backpassing we've removed, remove backpassing from the language. We can optionally keep a parsing error so that people that try to use it get a good message about why they can't do it.

Kilian Vounckx (Jun 21 2024 at 21:49):

Richard Feldman said:

I think it's important to note that <- is probably the hardest thing to learn in the language right now

I think record builder syntax is way harder to wrap my head around, but it is useful to have and can be taught separately to ! and ?. I feel the same about backpassing. It is useful in other situations then results and tasks and can also be taught separately. I think we should keep it, aside from ! and ?.

Gleam has use expressions, which are basically the same as backpassing. I don't know how it is received by beginners there, but I think it is a precedent for it not being a big problem if taught well.

Richard Feldman (Jun 21 2024 at 23:36):

record builder syntax is definitely confusing right now, but we already have a separate plan in progress to address that

Sam Mohr (Jun 22 2024 at 04:25):

https://github.com/roc-lang/roc/issues/6828 and https://github.com/roc-lang/roc/issues/6829 have been created

Kilian Vounckx (Jun 22 2024 at 06:55):

Richard Feldman said:

record builder syntax is definitely confusing right now, but we already have a separate plan in progress to address that

I see, where can I find more info about that?

Kilian Vounckx (Jun 22 2024 at 06:59):

Just to be clear, I know my last message was pretty negative, but I still really like the language. And I think ! and ? are really good additions. I get that removing backpassing creates a simpler and easier language, which is one of the main goals of roc. Not the trade-off I would have made, but totally understandable

Richard Feldman (Jun 22 2024 at 11:34):

Kilian Vounckx said:

Richard Feldman said:

record builder syntax is definitely confusing right now, but we already have a separate plan in progress to address that

I see, where can I find more info about that?

I've talked about it with @Agus Zubiaga but I don't think we have an actual #ideas post about it yet

Agus Zubiaga (Jun 22 2024 at 11:37):

I’ll post it in a few hours when I’m home :smile:

Kasper Møller Andersen (Jun 22 2024 at 11:59):

I haven't built up the experience to tell whether backpassing should stay or not, though it's definitely one of the more trippy concepts to understand. One thing which I think makes both backpassing and record building more complicated though, is that they overload the meaning of arrows in the language.

For context, I spend a bunch of my time in Scala, which has this problem too, though to a higher degree. That is, I've got to remember which kind of arrow to use, and also which direction it's pointing, and I still mess it up after 10 years of learning.

So I would take a good hard look at whether the sigil should actually be an arrow. My experience is that I tend to think of the arrow sigil as "input -> output", and both backpassing and record building use a somewhat-similar-but-really-different semantic, and just changing the direction of the arrow is not enough to make that clear.

Just to shake people out of their habits, here are some crazy suggestions:

foo -\ funcExpectingCallback
Str.concat foo " bar"

since the \ acts as a line where the stuff to the left and below are separate from that to the right above it.
And record building

    { aliceID, bobID, trudyID } =
        initIDCount {
            aliceID from incID,
            bobID from incID,
            trudyID from incID,
        } |> extractState

Because sometimes words are better than sigils :smiley:

Agus Zubiaga (Jun 22 2024 at 15:28):

FYI I just posted the record builders idea.

Daniel Schierbeck (Jun 25 2024 at 06:25):

Richard Feldman said:

for example, using ! for parsers and random number generators (Haskell-style) can be nice when using them on their own, but runs into problems as soon as you try to combine them with tasks (which is what leads to monad transformers in Haskell, and I extremely never want monad transformers to be a thing in Roc)

Just out of curiosity, what are the situations where you would need to combine these with tasks in a way that would cause issues? Like, I can imagine a setup like this:

Card : { rank : U8, suit :  [Clubs, Diamonds, Hearts, Spades] }

randomCard : Random Card
randomCard =
    rank = Random.range! 1 13
    suit = Random.pick! [Clubs, Diamonds, Hearts, Spades]
    { rank, suit }

randomCardTask : Task Card []
randomCardTask =
    card = Random.generate! randomCard
    Stdout.line! "card: $(card)"

I assume _this_ composition wouldn't be a problem, right? When would you run into issues?

Brendan Hansknecht (Jun 25 2024 at 06:39):

If you inline randomCard, I think it shows the issue.

Sam Mohr (Jun 25 2024 at 06:40):

a.k.a. who owns the exclam?

Brendan Hansknecht (Jun 25 2024 at 06:40):

You get some ! for Random and some ! for Task. That will lead to type errors and lack of composibility.

Richard Feldman (Jun 25 2024 at 11:19):

Daniel Schierbeck said:

Richard Feldman said:

for example, using ! for parsers and random number generators (Haskell-style) can be nice when using them on their own, but runs into problems as soon as you try to combine them with tasks (which is what leads to monad transformers in Haskell, and I extremely never want monad transformers to be a thing in Roc)

Just out of curiosity, what are the situations where you would need to combine these with tasks in a way that would cause issues? Like, I can imagine a setup like this:

Card : { rank : U8, suit :  [Clubs, Diamonds, Hearts, Spades] }

randomCard : Random Card
randomCard =
    rank = Random.range! 1 13
    suit = Random.pick! [Clubs, Diamonds, Hearts, Spades]
    { rank, suit }

randomCardTask : Task Card []
randomCardTask =
    card = Random.generate! randomCard
    Stdout.line! "card: $(card)"

I assume _this_ composition wouldn't be a problem, right?

it's not a problem if that Random.generate function returns a Task - the problem would be if you tried to use ! with both functions that return Random and functions that return Task in the same scope

Richard Feldman (Jun 25 2024 at 11:41):

of note, you could implement randomCard using a record builder; with the proposed new syntax it could look like:

randomCard : Random Card
randomCard =
    { Random.combine <-
        rank: Random.range 1 13,
        suit: Random.pick [Clubs, Diamonds, Hearts, Spades],
    }

Richard Feldman (Jun 25 2024 at 11:43):

(assuming Random.combine : Random a, Random b, (a, b -> c) -> Random c))

Daniel Schierbeck (Jun 25 2024 at 19:56):

I must say, the record builder syntax seems a bit exotic to me, but I could probably get used to it.

As for the mixing of types with ! – I think I would be OK with that being a compilation error? I mean, the whole point is to sequence two or more lines together, so it would make sense that they have to operate on the same types. Basically, the semantics of a! depends on the type of a, and the subsequent line must have the same type. Or am I misunderstanding? I think that can be adequately explained by error messages.

Daniel Schierbeck (Jun 25 2024 at 19:58):

Like, I would assume this would work as well:

randomCardTask : Task Card []
randomCardTask =
    card = Random.generate! (
        rank = Random.range! 1 13
        suit = Random.pick! [Clubs, Diamonds, Hearts, Spades]
        { rank, suit }
    )
    Stdout.line! "card: $(card)"

Brendan Hansknecht (Jun 26 2024 at 01:02):

Yeah, totally is fine to be a type error. The issue is that mixed flows comes up in practice and then you are stuck redesigning code. With state manually being passed around and shadowing, it never comes up. So it removes a class of complications.

This is a totally contrived example, but this is the type of flow that is impossible to represent (requires monad transformers or all random number generation to go through task).
You don't have a way to pipe your random state from the top to the bottom due to the tasks in the middle:

rank = Random.range! 1 13
Stdout.line! "The rank is $(Num.toStr rank). Please input x: "
x = Stdin.line!
randToX = Random.range! 0 (Str.toU64 x)

# use rank and randToX.

If instead, ! is only for Task and Random uses a shadowing api.
Everything just works.

(rank, rng) = Random.range rng 1 13
Stdout.line! "The rank is $(Num.toStr rank). Please input x: "
x = Stdin.line!
(randToX, rng) = Random.range rng 0 (Str.toU64 x)

# use rank and randToX.

Brendan Hansknecht (Jun 26 2024 at 01:05):

Does that make the issue clearer @Daniel Schierbeck

Daniel Schierbeck (Jun 26 2024 at 12:16):

I guess so, but it’s maybe also optimizing very much for reducing complexity in a language that I find at least somewhat inherently complex (but friendly!) Like, the whole business with having sized ints instead of just Int as also a tradeoff, I guess?

Could your example be written as:

rank = Random.range 1 13
    |> Random.generate!
Stdout.line! "The rank is $(Num.toStr rank). Please input x: "
x = Stdin.line!
randToX = Random.range 0 (Str.toU64 x)
    |> Random.generate!

Or am I misunderstanding how the semantics would work? (which I guess would argue against my proposal :D )

I don’t think it’s inherently a problem that you need to bubble up to a Task when you need an effect to happen; what I think ! does nicely is sequencing together same-type operations. Like, it should be possible to generate a playing card Random value in a simple way, because that’s a building block. But once you’re in an application tier that’s dealing with Tasks anyway, I don’t think it’s a problem that _everything_ is a task… but I probably haven’t seen as much production code as y’all. But isn’t that basically how Haskell apps are typically designed? The outermost layer is all IO, but you have building blocks that are pure, but oftentimes use monads. In my example, you’d also be able to generate _all_ possible playing cards with a List implementation that uses flatMap.

Brendan Hansknecht (Jun 26 2024 at 15:04):

Assuming Random.generate converts into a task. That works.

I think it is missing the core of what I am trying to get at though. I could have an 100% pure random number generator library in roc. It might not have a way to convert to a task. If it can't be converted into a task, then you're stuck here. Random doesn't have to be IO if it is seeded.

For Random.generate! to work:

1. The platform has to have random number generation primitives that are compostable to generate more complex types.
2. You have to give up the ability to control the RNG seed/state in roc. It is now a concept beyond the IO boundary.

Brendan Hansknecht (Jun 26 2024 at 15:05):

Cause Task doesn't have any special handling the pipe the RNG seed/state around. So it has to all be done behind the scenes by the platform. You just get stuck with whatever the platform implements instead of being able to pick from a wide swath of prng algorithms.

Brendan Hansknecht (Jun 26 2024 at 15:08):

So while forcing into the Task box can work with a random number generator (and the initial seed will always need to come from the platform), it imposes a lot of restrictions on usable prngs. RNG is also only one potential use of ! or state management with shadowing. While RNG might fit into task, others may not.

Daniel Schierbeck (Jun 26 2024 at 17:10):

I'm not sure that's true – generate could have the type Random a -> Task a [], which of course would require the platform to be able to provide a seed and maintain the "next" seed in state, but Random a could be defined basically as Seed -> (a, Seed); a non-Task version of generate with explicit seed passing should be quite simple to do, right? Random.generateFromSeed : Seed, Random a -> (a, Seed).

Brendan Hansknecht (Jun 26 2024 at 17:28):

In the example above, generate has to be the Task version. Otherwise, it would be type error.

Daniel Schierbeck (Jun 26 2024 at 18:01):

Definitely, but at some point, in order to generate random numbers you need _some_ Task to provide a seed, unless you hardcode it!

Daniel Schierbeck (Jun 26 2024 at 18:10):

And really, I don’t even think the platform would need to keep state; it should just use the native way to generate actual random numbers and pass that as the seed to the Random value. Applications that _want_ to manage the seed explicitly could just do that, and end with e.g. Task.ok (Random.generateWithSeed seed randomfoo) or something, right?

Brendan Hansknecht (Jun 26 2024 at 18:59):

I think this discussion is missing the point. The ! version is forcing it to be a Task. The shadowing version avoids that. RNG isn't the only chainable type with state. Other types will hit the same issue.

Looping back to where this discussion started. Allowing ! on more types leads to many cases where compatibility requires some sort of transformer function. There is no guarantee the transformer function is possible to write. As such, shadowing and explicitly passing around state for non-task types keeps the code consistent and simple without the need for transformers functions.

Daniel Schierbeck (Jun 26 2024 at 19:39):

OK – I think my understanding of the discussion was that it debated the merits of having ! _not_ only work for tasks, but for other types where sequencing of steps makes sense. Anyway, y'all are probably in a better position to evaluate the merits.

Brendan Hansknecht (Jun 27 2024 at 00:20):

I do think discussion of expanding ! is important. Probably deserves it's own ideas thread.

Brendan Hansknecht (Jun 27 2024 at 00:21):

I think it has a larger cost to add to the compiler, but totally is in long term consideration.

Brendan Hansknecht (Jun 27 2024 at 00:21):

Really depends how much friction is seen in practice

Brendan Hansknecht (Jun 27 2024 at 00:22):

Also, shadowing is slotted for addition, so we would want to add that and see if it elevates friction before jumping to expanding !.

Daniel Schierbeck (Jun 27 2024 at 06:44):

:+1:

Daniel Schierbeck (Jun 27 2024 at 06:45):

I was directed to this thread after opening this one https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/Expanded.20backpassing.20.2F.20.60!.60.20use.20cases

Ian McLerran (Jun 28 2024 at 01:07):

So I’m only newly familiar with the term/concept of “shadowing”, as I’ve been beginning to learn Rust, and I’ve seen it pop up a few times in discussions about back passing and the bang operator.

My understanding is that it allows you to redefine an existing definition (usually in an inner scope). I understand this can be convenient, but what problem does this specifically solve?

Luke Boswell (Jun 28 2024 at 01:39):

I think the example that has been mentioned as a typical use case is threading state through a series of functions, like a seed for a random generator.

For example, in the following we are adding an incrementing number to have unique identifier for each seed.

app [main] {
    cli: platform "https://github.com/roc-lang/basic-cli/releases/download/0.10.0/vNe6s9hWzoTZtFmNkvEICPErI9ptji_ySjicO6CkucY.tar.br",
    rand: "https://github.com/lukewilliamboswell/roc-random/releases/download/0.2.1/mJSD8-uN-biRqa6CiqdN4-VJsKXxY8b1eFf6mFTe93A.tar.br",
}

import cli.Stdout
import cli.Task exposing [Task]
import rand.Random

main =
    generator = Random.i32 0 100
    seed = Random.seed 1234

    {state:seed1, value:first} = generator seed
    {state:seed2, value:second} = generator seed1
    {state:_, value:third} = generator seed2

    values = [first, second, third] |> List.map Num.toStr |> Str.joinWith ","

    Stdout.line! "My random values are: $(values)"

With shadowing we can instead simplify this (and reduce the chance of copy-paste errors):

    {state:seed, value:first} = generator seed
    {state:seed, value:second} = generator seed
    {state:_, value:third} = generator seed

Brendan Hansknecht (Jun 28 2024 at 04:17):

Yeah, there is a lot of update centric roc code that can have many versions of the same variable. I think the dict source has some good examples (though the newer version is has less overall). Look at removeBucket. Editing it can be very bug prone when you use bucketIndex2 instead of bucketIndex3.

Otherwise many apis with state that gets threaded around like RNG hit this pretty bad.

Ian McLerran (Jun 28 2024 at 22:08):

Thanks guys, that makes sense. So to rephrase your answers shadowing will allow performing repeated “mutation” of a value, where for one reason or another piping cannot be applied.

Brendan Hansknecht (Jun 29 2024 at 17:06):

That is a solid way to look at it!

Kiryl Dziamura (Jul 02 2024 at 14:35):

not sure if it was discussed, but what about ? in the statement position? Should it early return?

and = \resultA, resultB ->
    resultA? # return resultA if it's Result.err
    resultB? # otherwise return resultB

and = \resultA, resultB ->
    Result.try resultA \{} ->
        resultB

Brendan Hansknecht (Jul 02 2024 at 14:38):

I assumed it works work exactly like? just so that it is consistent

Richard Feldman (Jul 08 2024 at 22:12):

Kilian Vounckx said:

Gleam has use expressions, which are basically the same as backpassing. I don't know how it is received by beginners there, but I think it is a precedent for it not being a big problem if taught well.

here's a data point on that: https://erikarow.land/notes/using-use-gleam

Recently, a colleague checked out Gleam’s language tour. They liked what they saw, but they were confused by Gleam’s use syntax.

edit: lobste.rs discussion about that post: https://lobste.rs/s/mmje1n/using_use_gleam

Richard Feldman (Jul 08 2024 at 22:14):

TIL Gleam also chose the name Result.try for that function!

Kiryl Dziamura (Jul 08 2024 at 23:16):

https://github.com/gleam-lang/gleam/issues/1709#issuecomment-1236297281

I don't know. Maybe backpassing will become a common feature in languages? Maybe the main problem is unfamiliarity? For me, it’s not very clear what’s fundamentally different between backpassing and pipeline operator. One scary and the other not? The biggest roadblock in learning backpassing for me was loads of discussions about how special it is and how confusing it is for newcomers. This mystification distracted me a lot. When I tried it twice - I couldn’t imagine how to write code a different way. It felt natural very quickly.
Sorry for the offtopic.

Daniel Schierbeck (Jul 09 2024 at 08:28):

I think the reason I found it confusing might be that it’s the same operator used in Haskell’s do syntax, but it doesn’t leverage monads in the same way, so it doesn’t “just work” on values? There’s an argument that being explicit is “simpler”, but there’s also an argument that it’s _not_ :D

Sam Mohr (Jul 09 2024 at 08:59):

I'd say that monads and do notation are "simpler" in the way that they are a single, generic concept that can be applied to a massive variety of domains, but ? is "simpler" in that it's easier to teach and grok for new users because they only have a single use case, meaning you just have to think "? propagates errors kind of like a throw"

Sam Mohr (Jul 09 2024 at 08:59):

So do we aim for conceptual simplicity, or simple to understand? The Roc approach in general seems to be aim for simple to understand

Sam Mohr (Jul 09 2024 at 09:00):

So you're right that ? is a special case in a way, meaning it's less conceptually simple, but that's okay with the team

Sam Mohr (Jul 09 2024 at 09:01):

And also, a big benefit of using ? instead of monads is that we avoid allowing really complex code to be written with Roc, meaning it's easier for any developer to engage with any Roc code base (a la golang) without needing to understand a different style of handling complexity

Sam Mohr (Jul 09 2024 at 09:04):

An example would be how gleam's use operator allows for list comprehensions, which are convenient, but are now more confusing. I'd say they're more confusing in specific because ? and ! only affect control flow in the "do I return early from this function?" way, whereas list comprehensions now emulate a loop, which Roc doesn't really have the concept of (ignoring List.walk, which is different IMO)

Brendan Hansknecht (Jul 09 2024 at 15:30):

Meanwhile, here I am working on an encoder, just created an tryEncode and definitely will miss <-. Cause it also works for my own try style methods (Also really useful for simple wrapper methods like Encode.custom)

Alex Nuttall (Jul 09 2024 at 16:06):

I think the problem, such as it was, with back-passing and the language's learning curve, was it being used in very introductory 'hello-world' exercises, which just print something to the console. But that has already been resolved by the bang operator and it should now be encountered in slightly more advanced contexts where it immediately provides benefits to the user as well as posing a (not massive) learning challenge

Speaking as a beginner myself, by the way

Anton (Jul 09 2024 at 16:09):

That's an interesting way to think about it, once you know !, <- may indeed be easier to understand.

Romain Lepert (Sep 06 2024 at 06:54):

Kiryl Dziamura said:

https://github.com/gleam-lang/gleam/issues/1709#issuecomment-1236297281

I don't know. Maybe backpassing will become a common feature in languages? Maybe the main problem is unfamiliarity? For me, it’s not very clear what’s fundamentally different between backpassing and pipeline operator. One scary and the other not? The biggest roadblock in learning backpassing for me was loads of discussions about how special it is and how confusing it is for newcomers. This mystification distracted me a lot. When I tried it twice - I couldn’t imagine how to write code a different way. It felt natural very quickly.
Sorry for the offtopic.

The symmetry between back passing and the pipe operator is a good point ! I don’t think |> trips people much. Maybe the back passing can be <| instead to make that connection clearer. The conceptual leap is small and it almost feels like learning just two sides of the same coin

Brendan Hansknecht (Sep 06 2024 at 15:24):

I think that would likely be more confusing in the common cases. It does something quite different from piping and it is often used with piping which will lead to

someVar <| someFunc a b c |> Result.map
...

Sky Rose (Sep 07 2024 at 01:27):

Mhm. Pipes are a new way to _call_ a function. Backpassing is a new way to _define_ a function (and also do something with it). Defining a function is already more complex than calling, so any trip hazards with backpassing will cause more problems than piping.

Last updated: Jun 16 2026 at 16:19 UTC