Stream: ideas

Topic: recording and replaying effects

Richard Feldman (May 14 2023 at 14:14):

I'm working on some designs for testing Tasks, and one thing I'm exploring is something like this:

$ roc run --record-fx MyEffects.rfx

when you run, this would save all the raw inputs and outputs to all the Tasks in your program - in other words, the actual data that goes to/from the systems-level functions in the platform's host

Richard Feldman (May 14 2023 at 14:15):

so, suppose we had that - there's this file (I made up the file extension .rfx but it could be something else) with all the inputs and outputs of a Roc program that's been run

Richard Feldman (May 14 2023 at 14:15):

what interesting things could we do with that?

Richard Feldman (May 14 2023 at 14:15):

one idea that comes to mind is that, given a .rfx file and the same Roc code base which generated it, you could re-run a debugging session on exactly what happened in that program without actually needing it to redo the effects

Richard Feldman (May 14 2023 at 14:16):

like it wouldn't redo any actual network requests or disk I/O, but the Roc code would all do exactly what it did on that run

Richard Feldman (May 14 2023 at 14:16):

so that seems pretty obviously useful in a future world where we have a debugger for Roc, but what other use cases might there be for having this?

Richard Feldman (May 14 2023 at 14:18):

another idea is having some way to generate tests from a .rfx file, like "this test takes a Task and verifies that if I run my Roc program on it, it will result in the same data getting sent to the host as before, and then assuming the host subsequently returns the same data back as it did for those inputs on the run we recorded, we can continue verifying with the next task, and so on and so forth"

Richard Feldman (May 14 2023 at 14:18):

that would be a pretty brittle test though

Richard Feldman (May 14 2023 at 14:18):

it would be useful for making sure refactors of effectful code didn't actually change anything in terms of the I/O

Richard Feldman (May 14 2023 at 14:19):

but there would be a lot of scenarios where a change that's harmless to the behavior of the I/O code in there would break the test, and I'd have to go manually fix it every time, which might end up being more time-consuming than the value of the test itself :sweat_smile:

Richard Feldman (May 14 2023 at 14:20):

anyway, so I'm looking for ideas based on this idea...if we had something like this, what could we do with it?

Richard Feldman (May 14 2023 at 14:21):

I'm not looking for polished proposals or anything, I'd just like to explore what the possibility space looks like!

Ayaz Hafiz (May 14 2023 at 14:22):

seems useful for unit tests, along the lines of Ruby's VCR: https://github.com/vcr/vcr

Richard Feldman (May 14 2023 at 14:23):

I've never used vcr - what's the experience like?

Richard Feldman (May 14 2023 at 14:24):

oh yeah, this is a very good consideration to be aware of:

Screen-Shot-2023-05-14-at-10.23.57-AM.png

Ayaz Hafiz (May 14 2023 at 14:35):

It’s pretty nice, I hadn’t used it either until a few months ago. Say you have some code under test that calls out to an external API (for example IMDB but you don’t want to rack up an API call bill in test). It matches outgoing network requests to recorded network responses.

Richard Feldman (May 14 2023 at 14:36):

oh interesting

Richard Feldman (May 14 2023 at 14:36):

does order matter?

Richard Feldman (May 14 2023 at 14:37):

or is it like "any time I send these exact bytes, assume this is the response"

Richard Feldman (May 14 2023 at 14:37):

I guess order would have to matter to some extent, in that sending the same bytes more than once over the course of the recorded run could result in different responses :big_smile:

Ayaz Hafiz (May 14 2023 at 14:43):

it does matter

Ayaz Hafiz (May 14 2023 at 14:43):

@Tommy Graves is an SME here :)

Richard Feldman (May 14 2023 at 17:29):

https://twitter.com/dan_manges/status/1657799462058598400 - Dan makes a good point about it being valuable to be able to simulate failure scenarios

@rtfeldman In Ruby, StringIO and VCR are easy to use and give confidence that the happy path of the code will work when actually performing I/O. They could do more to simulate failure scenarios though.

- Dan Manges (@dan_manges)

Richard Feldman (May 14 2023 at 17:29):

this is something I wonder about: starting from an exact recording and then wanting to tweak it

Richard Feldman (May 14 2023 at 17:30):

for example to be like "everything works the same as it did when I ran it, except at this one particular point there's a failure instead of a success, and then here's what I expect to happen after that..."

Anton (May 15 2023 at 10:27):

what interesting things could we do with that?

It could be useful to quickly inspect open source software for privacy concerns.

Richard Feldman (May 15 2023 at 11:09):

coooool

Richard Feldman (May 15 2023 at 11:13):

apparently this exists for JS: https://mswjs.io/

Nikita Tchayka (Aug 11 2023 at 17:14):

Replayability comes with a lot of cool advantages, for example, the Haskell framework Transient uses replayability to make transparent network transport and connection: https://github.com/transient-haskell/transient-stack

You can write code like:

foo = initNode do
  x <- locally do
    putStrLn "I'm on the main node"
    return 42

  y <- onRemote do
    putStrLn "I'm on the secondary node"
    return 10

  let result = x + y
  return x + y

And run the same program on two different processes. The main node (configured in some way, dont recall how to set a node to be the main one) would run all the locally blocks, recording the results of the effects, and then onRemote will block the main node so it awaits for a result.

On the secondary node, locally blocks the node, awaiting for the recorded result of that block, while onRemote will run locally on the secondary node, sending the result back to the main node when it finishes, unblocking the main node.

Paul M (Aug 11 2023 at 18:38):

Adding my two cents

Something semi-similar is ROS bag. Which is a recording and playback facility for the ROS robotic operating system. This collects data streams from cameras, 3d sensors, motors etc, then allows play back though though a software algorithm stack for testing software algorithims without physical hardware attached.

This a very very high level logging application, but what might be interesting to take inspiration from is the GUI. This has a lot of data to mange and simplifying and making this usable & easy is what I think this tool does well.

My inspiration that might be useful are:
1) ROS collects data & datatypes together, allows different "visualisation tools" to be bolted on for different use cases e.g X,Y graphs etc.

2) The gui can display multiple named datasources on a graph or overlay, so you can correlate events etc. So mabye having logging to different named topics/output stream/logging variable might be a killer features that could allow easier debugging of log files etc. (e.g log to log.motor_speed and log.motor_temperature are both in the log file, but the gui/other apps can separate out the log data streams)

3) It has a fastfoward/rewind ability - so from a long stream of data, you can jump to the important bits, and go though, you can also jump back and replay a bit without exiting/resetting everything. I think the visually jumping forward might be good, but connecting to rewind might be an insane challenge.

Here is the demo video of user experience https://www.youtube.com/watch?v=pwlbArh_neU

Bonus - here is the bit of the video of unit testing robotic hardware..........its very very pretty so worth a look for geeking out! https://www.youtube.com/watch?v=pwlbArh_neU&t=18s

Last updated: Jun 16 2026 at 16:19 UTC