Stream: compiler development

Topic: map2 to mapN

Anton (Feb 16 2024 at 12:24):

For a question form github; once we have map2 to execute tasks in parallel I expect we'll be able to provide a mapN as well right for any number of tasks?

Folkert de Vries (Feb 16 2024 at 12:35):

yes you can just recursively use map2 to map over N tasks

mainrs (Jun 03 2024 at 14:18):

Folkert de Vries said:

yes you can just recursively use map2 to map over N tasks

A follow-up question:

This would still mean that only 2 tasks get executed in parallel, is that correct? This just makes it easier to hand off multiple tasks to the runtime without oneself having to do the recursive call.

Agus Zubiaga (Jun 03 2024 at 16:12):

No, they could all get executed in parallel

Agus Zubiaga (Jun 03 2024 at 16:13):

The platform can still gather them and run them in parallel because they are not chained

Agus Zubiaga (Jun 03 2024 at 16:29):

To be clear, the functions (passed to map2) that combine the results of 2 tasks would not run in parallel, but the actual tasks in the platform would

Brendan Hansknecht (Jun 03 2024 at 16:49):

    R
   / \
  S1  S2
 / \  / \
A   B C  D

Root spawns S1 and S2 which are just spawner tasks. S1 and S2 spawn A, B, C, and D which can all run in parallel. So you have 4 tasks running in parallel but only use map2.

mainrs (Jun 06 2024 at 14:22):

Would a mapN then make sense? It receives a list of tasks, and subdivides them into calls to map2?

For me at least, running the tasks in parallel is the feature I am looking for. That the combination does not happen in parallel is totally fine for me.

Anton (Jun 06 2024 at 15:24):

Would a mapN then make sense?

I think so

Richard Feldman (Jun 06 2024 at 15:27):

I think the name should communicate whether it's running the tasks in parallel or sequentially

Richard Feldman (Jun 06 2024 at 15:29):

e.g.

Task.sequence : List (Task ok err) -> Task (List ok) err
Task.concurrent : List (Task ok err) -> Task (List ok) err

EDIT: I realized the return type should be Task (List ok) err - previously I had Task ok err

Richard Feldman (Jun 06 2024 at 15:30):

something like that

Richard Feldman (Jun 06 2024 at 15:31):

like if you're doing a list of "print to standard out" tasks, you probably don't want those to run concurrently :big_smile:

Anton (Jun 06 2024 at 15:31):

parallel may be an easier word compared to concurrent

Brendan Hansknecht (Jun 06 2024 at 15:33):

Obviously, Task.hopefully_parallel_but_maybe_only_concurrent. Just to clarify things.

Richard Feldman (Jun 06 2024 at 15:38):

yeah the problem is it couldn't be guaranteed to be parallel

Richard Feldman (Jun 06 2024 at 15:39):

because that would depend on the platform

Richard Feldman (Jun 06 2024 at 15:39):

so parallel wouldn't be accurate sometimes

timotree (Jun 06 2024 at 20:27):

What would Task.concurrent do if one of the tasks errored?

Richard Feldman (Jun 07 2024 at 01:32):

that's an excellent question! Personally I'd expect all the tasks in the list to run to completion no matter what, so I think that at the end if any of them failed, the returned Task would have errored out with the error of whichever one failed first

Richard Feldman (Jun 07 2024 at 01:32):

I could also see an argument for the return type being -> Task (List ok) (List err) and it accumulates all the errors of all the ones that failed

Richard Feldman (Jun 07 2024 at 01:33):

or maybe even -> Task (List ok) (err, List err) since there would always be at least one err if it ended up failing

Brendan Hansknecht (Jun 07 2024 at 01:58):

You could also give it a folder function

timotree (Jun 07 2024 at 04:02):

Task (List ok) (List err) only lets you return either a list of errors or a list of success values. If some tasks failed and others succeeded you want both

Brendan Hansknecht (Jun 07 2024 at 04:07):

Yeahs should be technically Task (List Result ok err) otherErr when otherErr is for failure to spawn tasks or something else platform related.....though I guess you could just leave the OtherError empty and put it in the list

Brendan Hansknecht (Jun 07 2024 at 04:12):

I guess looking at rust for API inspiration. They have

Task (List ok) err which returns the first error and cancels all tasks on failure

Brendan Hansknecht (Jun 07 2024 at 04:13):

I would expect this is actually wanted user API 85% of the time

Brendan Hansknecht (Jun 07 2024 at 04:14):

Oh wait, actually no, important caveat. Rust you would spawn them all in a loop and join separately and that makes a huge difference

Brendan Hansknecht (Jun 07 2024 at 04:15):

Would be as if you had Task.spawn : Task ok err -> Task (Future ok err) [] and then a separate way to join the handle. Either manually or all in a list.

Brendan Hansknecht (Jun 07 2024 at 04:16):

Yeah, I think Task.concurent is missing that you likely want to be able to spawn tasks and later query them. Or run code to decide what to span linearly. So I think we want some sort of intermediate Future type for this flow.

Brendan Hansknecht (Jun 07 2024 at 04:21):

I just realize that a Task and Future are the same thing. So a task that spawns a thread and return a future to join back the results later would be:

Task.spawn : Task ok err -> Task (Task ok err) []

Does that make sense.....

Last updated: Jul 06 2025 at 12:14 UTC