timer in repl · ideas · Zulip Chat Archive

I am curious about are there any timer function or other way for me to see hong long it takes a specific operation to happen in The Roc REPL. I am trying to evaluate the improvement in efficiency for compiling the same piece of code in Roc compared with another language.

Richard Feldman (Mar 28 2023 at 18:31):

Notification Bot (Mar 28 2023 at 18:31):

Brendan Hansknecht (Mar 28 2023 at 18:32):

If you move beyond the repl, we have a flag for timing the building of an application. It breaks the timing into many pieces.

Richard Feldman (Mar 28 2023 at 18:35):

» :time List.walk foo bar baz

Richard Feldman (Mar 28 2023 at 18:36):

but I could see that not being quite what you want; if this is being used for benchmarking, you probably want something closer to what hyperfine does: run the expression several times and report median execution time, min and max execution time, etc.

dank (Mar 28 2023 at 18:37):

ipython's got
%timeit for inline, and %%timeitfor blocks
which are pretty convenient

Richard Feldman (Mar 28 2023 at 18:38):

» :benchmark List.walk foo bar baz

and it runs the given expression several times (we can tweak that formula in the future) and then reports those things

Richard Feldman (Mar 28 2023 at 18:38):

dank (Mar 28 2023 at 18:39):

oh yea that's what timeit does. runs it like 1000 times, gives you min max and avg

Brendan Hansknecht (Mar 28 2023 at 18:41):

» :benchmark (List.walk foo bar baz) (List.walk foo bar maybeFasterBaz)

Though if repl is always debug builds and one of these functions ends up causing copying, it may not be that useful for performance benchmarks.

Brendan Hansknecht (Mar 28 2023 at 18:41):

Richard Feldman (Mar 28 2023 at 18:48):

hm, really? I read it the other way, but I guess either interpretation could make sense!

Audrey Gao (Mar 28 2023 at 18:58):

» :benchmark List.walk foo bar baz
That sounds pretty good! But I guess I cannot have the opportunity to include it in my current senior thesis which is gonna be due recently. I am trying to use the timer information for benchmarking the improvements in translating Hazel to Roc and utilizing WebAssembly.

Brendan Hansknecht (Mar 28 2023 at 19:03):

I would advise avoiding the repl for something like that. Just generate the final wasm and run it in a browser with profiler (or local wasm runtime with hyperfine for benchamarking). Be able to compare the perf of multiple different versions of the code, loop enough time to get good performance number details.

Brendan Hansknecht (Mar 28 2023 at 19:03):

Since repl is always debug builds, you will get much worse performance in many cases.

Richard Feldman (Mar 28 2023 at 19:12):

but that's not available in our web repl because we don't want to bundle all of LLVM in there

Richard Feldman (Mar 28 2023 at 19:13):

so yeah, generating the wasm and running it in the browser sounds like a better idea for this project :thumbs_up:

Audrey Gao (Mar 28 2023 at 19:18):

Thanks so much for the advice! I will try to generate the wasm and run it in the browser then.

Brendan Hansknecht (Mar 28 2023 at 19:24):

Oh, one quick tip with benchmarking in the browser, either have dev tools closed when you run the benchmark or run the benchmark while the profiler is running. Otherwise all of the devtool hooks will really mess up the performance and you will get wrong results.

Richard Feldman (Mar 28 2023 at 19:29):

also if you don't mind, I think we'd all be curious to see the results of your project! :smiley:

Richard Feldman (Mar 28 2023 at 19:29):

so it would be awesome if you could share them (or even the whole thesis, if you're comfortable) - either on Zulip, or DM one of us in private if that would be preferable

Audrey Gao (Mar 28 2023 at 19:38):

Thank you for sharing that helpful tip! I will definitely keep that in mind when benchmarking in the browser. I would be happy to share my thesis once it is finished.

Richard Feldman (Mar 28 2023 at 20:35):

Audrey Gao (Mar 29 2023 at 05:08):

I tried to compile with the commend "roc build --target=wasm32 main.roc --prebuilt-platform=false", but got the following error messages: image.png

Luke Boswell (Mar 29 2023 at 05:45):

Luke Boswell (Mar 29 2023 at 05:46):

Anton (Mar 29 2023 at 07:40):

Audrey Gao (Mar 29 2023 at 14:34):

I have pulled the latest version of roc and put this link into the packages session of the header but still got the same error message. :face_exhaling: I don't know if I have used the cli release link correctly. The terminal cannot run "roc dev" for the test file now. By the way, I am using MacOS and maybe this can be a problem?

Anton (Mar 29 2023 at 14:38):

Hmm, I don't think basic-cli ever had a host.zig so this error is pretty strange, can you share the header of your main.roc file

Audrey Gao (Mar 29 2023 at 14:45):

Anton (Mar 29 2023 at 14:49):

Yeah, nothing wrong there.
Can you share the entire file?
Are you on macos 12 or 13?
Are you on an intel or apple silicon mac?

Audrey Gao (Mar 29 2023 at 14:55):

I am on macOS 11.6 (Big Sur) and an intel mac. Should I update the version of MacOS? The entire file is also quite trivial. image.png

Anton (Mar 29 2023 at 14:56):

Oh I think I see the problem, I believe --target=wasm32 requires a host.zig file

Anton (Mar 29 2023 at 14:57):

Audrey Gao (Mar 29 2023 at 15:01):

Anton (Mar 29 2023 at 15:13):

Anton (Mar 29 2023 at 15:36):

basic-cli 0.3.1 works for me but 0.2.0 does not (as expected). Did you accidentally use the old nightly for this latest test? You can check with roc version

Audrey Gao (Mar 29 2023 at 15:46):

"roc version" returns "roc built-from-source" and I am currently up to date with 'origin/main'.
image.png

Anton (Mar 29 2023 at 15:54):

Hmm strange, can you try with the latest nightly so I can eliminate some variables?

tar xf roc_nightly-macos_x86_64-<VERSION>.tar.gz
cd roc_night<TAB TO AUTOCOMPLETE>
./roc path_to_your_file.roc

Audrey Gao (Mar 29 2023 at 16:00):

Anton (Mar 29 2023 at 16:00):

Using curl -OL https://github.com/roc-lang/roc/releases/download/nightly/roc_nightly-macos_x86_64-2023-03-29-6714ada.tar.gz to download the archive should prevent that

Audrey Gao (Mar 29 2023 at 16:06):

Anton (Mar 29 2023 at 16:17):

Audrey Gao (Mar 29 2023 at 16:27):

Sorry for my mistake! I forgot to change the basic-cli version. It now works with 0.3!

Anton (Mar 29 2023 at 16:38):

No problem, the error message you got was not very helpful. I'll make an issue to improve it.

Anton (Mar 29 2023 at 16:40):

I am pretty confident the /Users/Audrey/roc/target/debug/roc binary was not up to date though.

Anton (Mar 29 2023 at 18:09):

Thanks for sticking with me through the debugging journey @Audrey Gao :)
I assume you still want to do a wasm build? I don't think we intend to support wasm with basic-cli, this is a minimal wasm platform. What kind of programs did you want to build?

Audrey Gao (Mar 30 2023 at 03:05):

Yes, I still want to do a wasm build. I want to write a program to benchmark Roc code against similarly-optimized programs in Hazel, but haven’t decided the content of the program yet. Are there any sample program that can facilitate such benchmarking?

Brendan Hansknecht (Mar 30 2023 at 04:17):

Brendan Hansknecht (Mar 30 2023 at 04:18):

Brendan Hansknecht (Mar 30 2023 at 04:21):

Started based on someone else's performance benchmarks of functional languages in the browser

Brendan Hansknecht (Mar 30 2023 at 04:22):

Everything in implementations/roc-effectful should be a standalone platform with roc app that can be imported by including a js file that will load everything

Audrey Gao (Mar 31 2023 at 05:46):

Audrey Gao (Mar 31 2023 at 23:03):

I find it hard to benchmark roc in the browser using the old code. Is it possible for me to get to the REPL codebase and use the Instant() in Rust to keep track of the time between hitting the "return" button and getting the evaluated output?

Brian Carroll (Apr 01 2023 at 04:17):

It's possible but not implemented.
Any Rust function that wants to know the time, has to ask the operating system, by making a system call. In Wasm that compiles to a WASI system call.
But WASI is specifically for non-browser environments and we are in a browser! So for the REPL we have a mock WASI implementation in JavaScript. They're mock functions because we don't need any syscalls normally.

We could find a WASI polyfill library for the browser and integrate it into the REPL. Or implement the specific ops for timing.

This can be done by inserting performance.now calls into the right places in repl.js

Brian Carroll (Apr 01 2023 at 04:23):

In repl.js depending on where you put those calls, you can benchmark the Roc compiler or the Roc application or the pretty printing of the output. Those 3 things happen in Wasm but are connected together through JS.

Brian Carroll (Apr 01 2023 at 04:23):

The Roc app in this case is a concatenation of all the lines you typed into the REPL

Audrey Gao (Apr 01 2023 at 04:26):

Can you point me to the gitHub repository for REPL so that I can make a try for the implementation?

Brian Carroll (Apr 01 2023 at 04:26):

Brian Carroll (Apr 01 2023 at 04:27):

Brian Carroll (Apr 01 2023 at 04:28):

Brian Carroll (Apr 01 2023 at 04:29):

Audrey Gao (Apr 01 2023 at 04:33):

Brendan Hansknecht (Apr 01 2023 at 04:54):

Though that will only give dev backend wasm, right? So it will be slower than what LLVM would generate and may not be the most useful for benchmarks.

Brian Carroll (Apr 01 2023 at 05:12):

Brian Carroll (Apr 01 2023 at 05:38):

Yeah so in general I have found that the hardest part of Wasm tends to be integrating it with JS to get a useful end-to-end system, and that is what we are seeing here. @Audrey Gao I'm afraid it's going to be hard to get a ready made platform that just does what you want. You might end up having to invest a bit of time into this platform/setup stuff. But you're welcome to ask questions here and we'll help out!

Audrey Gao (Apr 01 2023 at 05:51):

So the dev backend wasm will be even slower than the current command line compiler?

Brian Carroll (Apr 01 2023 at 06:01):

The dev back end does no optimizations.
That means the compiler finishes its job sooner but the resulting binary is unoptimized and will run slower.
I'm not sure which time you were asking about.

Audrey Gao (Apr 01 2023 at 06:09):

Will it be considered run time or the sum of compile time and run time if my counting start with hitting the "return" button and getting the evaluated output?

Brian Carroll (Apr 01 2023 at 06:13):

Audrey Gao (Apr 01 2023 at 06:25):

So does that mean creating a new platform also include doing optimization? I am actually quite confused about building a new platform. Are there any documentation about how to use this codebase: https://github.com/bhansconnect/functional-mos6502-web-performance? I tried to run "make" but failed with the following error: image.png I have installed the latest idris2 version.

Brian Carroll (Apr 01 2023 at 06:34):

The REPL has a Wasm build of the Roc compiler embedded into the web page. It is a special version of the compiler that cannot do optimizations because we want it to be as small as possible.
But if you are not using the REPL then you can use the normal Roc compiler from the command line to compile your Roc code to a Wasm binary, and use optimizations if you want.

Brian Carroll (Apr 01 2023 at 06:36):

So yes if you create a new platform it will enable you to do optimizations. The only place you cannot have optimizations is in the REPL.

Audrey Gao (Apr 01 2023 at 06:43):

I think I roughly understand now. Actually I have another question regarding how to locally run REPL. When I try to do "crates/repl_wasm/build-www.sh" from the project root directory, I got the following error message: image.png

Brian Carroll (Apr 01 2023 at 07:20):

I'm not sure what's happening there. Looks like there is some setting that assumes you are only ever building for your own CPU, which apparently is an Intel "Ice Lake". But you are building for Wasm so that doesn't work.

Brian Carroll (Apr 01 2023 at 07:21):

To get started you could download the pre-built binary. This is what we do in the website build script.

Richard Feldman (Apr 01 2023 at 11:17):

one way to think about this is that the wasm repl is optimized for minimizing (compile time + run time + pretty printing time)

Richard Feldman (Apr 01 2023 at 11:17):

if it used LLVM as its backend instead, then run time would improve but compile time would degrade - and for the types of things people usually evaluate in repls, the compile time hit would almost always dramatically outweigh the runtime performance improvement

Richard Feldman (Apr 01 2023 at 11:19):

if you want to get a sense for how much LLVM costs, compare how fast roc repl is at printing answers to how fast https://www.roc-lang.org/repl (the wasm web repl) is at doing the same thing

Richard Feldman (Apr 01 2023 at 11:20):

the online repl is typically way faster because it's using the dev backend instead of the LLVM backend

Richard Feldman (Apr 01 2023 at 11:20):

(there are also linker differences, but I think LLVM accounts for most of the difference)

Richard Feldman (Apr 01 2023 at 11:21):

so yeah, I think if you want to optimize for (compile time + run time + pretty printing time) being as low as possible, and you're running in the browser, then I would want to do exactly what the wasm web repl is doing :big_smile:

Audrey Gao (Apr 01 2023 at 12:10):

I have run the command "cp -r www/public www/build" for a minimal build which doesn't give me any output in the terminal, so I assume it run successfully. Does that mean I should run "www/build.sh" for a full build instead?

Audrey Gao (Apr 01 2023 at 12:15):

Thanks a lot for the thorough explanation! So I guess I can still try to use performance.now calls to count for the (compile time + run time + pretty printing time) of wasm web repl of the dev backend, and get a rough benchmark?

Brian Carroll (Apr 01 2023 at 12:24):

For small programs, the compile time might be 100x the run time, so if you are mainly interested in the runtime then this is much worse than just "rough"! I assume what you are interested in is the runtime of the Roc program? Is that right?
Actually, could you remind me what your goal is? I think I missed the beginning of this whole conversation.

Brian Carroll (Apr 01 2023 at 12:25):

If the runtime of the Roc program is what you care about then you can insert performance.now calls into repl.js to only measure that. You are not limited to only measuring (compile time + runtime + printing time) because you are editing the code that connects those pieces together.

Brian Carroll (Apr 01 2023 at 12:25):

But it is an unoptimized build. It might be a factor of 2 or 3 slower than an optimized build, I don't know, I haven't actually benchmarked it myself.

Brian Carroll (Apr 01 2023 at 12:31):

Audrey Gao (Apr 01 2023 at 12:44):

I'm actually interested in measuring (compile time + runtime + printing time) as a whole.

Audrey Gao (Apr 01 2023 at 12:48):

I tried to do "www/build.sh" first, it passed the command for downloading the pre-built binary, but got stuck at "cargo run --release --bin roc run www/generate_tutorial/src/tutorial.roc -- www/generate_tutorial/src/input/ www/build/tutorial/" Is there other "build.sh" files that I should run beforehand?
image.png

Brian Carroll (Apr 01 2023 at 12:57):

Brian Carroll (Apr 01 2023 at 12:58):

Brian Carroll (Apr 01 2023 at 12:59):

Brian Carroll (Apr 01 2023 at 13:02):

You will need that version.
There are instructions in BUILDING_FROM_SOURCE.md for how to install everything.
It's OK to have two versions of Zig on your machine as long as you set the environment variable ROC_ZIG to the path of the 0.9.1 executable

Brendan Hansknecht (Apr 01 2023 at 15:41):

Yeah, it is outdated and based on some random persons blog post code, so it has a lot of random dependencies. If you wanted, I could help wip up a simple wesm benchmarking platform. That said, i guess if you actually want dev compile time + dev runtime the web repl may actually be the best option. Just won't really represent anything real world except for repl use of short snippets of code.

Audrey Gao (Apr 01 2023 at 17:55):

I tried to manually install zig 0.9.1 by directly downloading, since "brew install zig@0.9.1" is not working. However, even though I added the PATH "export PATH=$PATH:/Users/audrey/zig-macos-x86_64-0.9.1/zig" to the .zshrc file and ran source ~/.zshrc, I still got the error "zsh: command not found: zig." Is there another way to install zig 0.9.1? Also, when I try to run zig directly, it returns the info that "zig” cannot be opened because the developer cannot be verified." Is the problem laying here?

Anton (Apr 01 2023 at 18:03):

Audrey Gao (Apr 01 2023 at 18:04):

I actually installed all dependencies with nix first, but the zig version is 0.10.1 instead of 0.9.1.

Anton (Apr 01 2023 at 18:09):

Audrey Gao (Apr 01 2023 at 18:12):

Yes. I haven't installed zig before and when I check its version after installing with nix, it is shown as 0.10.1

Anton (Apr 01 2023 at 18:13):

Audrey Gao (Apr 01 2023 at 18:13):

Anton (Apr 01 2023 at 18:14):

Audrey Gao (Apr 01 2023 at 18:15):

Anton (Apr 01 2023 at 18:16):

Anton (Apr 01 2023 at 18:21):

I have to go but your flake.lock file should be identical to the one I linked.
If that's the case it should not be possible to install the wrong version of zig with nix.
I recommend double checking, so at the root of the repository:

nix develop
zig version
which zig

Audrey Gao (Apr 01 2023 at 18:22):

Audrey Gao (Apr 01 2023 at 18:55):

I pulled again from the main branch and run "nix develop", the zig version is correct now. But I encountered a new issue regrading the version of MacOS that nix is using. When I try to run "www/build.sh" it failed to compile roc_load with the warning that "object file (/Users/audrey/roc/target/release/deps/libring-ccd1460ef92b19a2.rlib(aesni-x86_64-macosx.o)) was built for newer macOS version (11.0) than being linked (10.12)"

Brendan Hansknecht (Apr 01 2023 at 20:08):

Audrey Gao (Apr 01 2023 at 20:27):

Audrey Gao (Apr 02 2023 at 03:01):

Sorry for asking so many questions. I encountered a new issue regarding printing info into the console. When I inserted several console.log() inside the "repl.js" and rebuilt the repl, I typed some expressions and hit enter, but no message was shown in the browser's console.

Brian Carroll (Apr 02 2023 at 13:48):

Audrey Gao (Apr 02 2023 at 13:50):

When I looked into the source files of the repl running on my localhost, there are no console.log() commands inserted. So I guess the rebuild doesn't effect js file. I just followed instructions in this rm: https://github.com/roc-lang/roc/tree/main/crates/repl_wasm.

Brian Carroll (Apr 02 2023 at 14:14):

I think that readme might have out of date paths in it from before we reorganized things. But basically the repl.js file is just copied into the build directory. You should be able to figure out the correct directory from reading the build script.

Audrey Gao (Apr 02 2023 at 14:40):

Thanks for the help! Calling "./www/build.sh" fixed the problem. If I understood correctly, "outputText = await roc_repl_wasm.entrypoint_from_js(inputText)" should be the compile time + runtime ? image.png

Brian Carroll (Apr 02 2023 at 15:13):

Oh, the await reminds me, it also includes the time the browser takes to initialize the Wasm module.

Brian Carroll (Apr 02 2023 at 15:14):

What are you going to compare this number to? What kind of conclusions are you hoping to draw from the analysis?

Audrey Gao (Apr 02 2023 at 15:58):

I'm evaluating whether translating Hazel (another functional programming language) to Roc and utilizing the WebAssembly backend would result in greater efficiency compared to compiling with Hazel's existing compiler. I am comparing the time of running the code with the same functionality on https://hazel.org/build/dev/ with the translation time of the transpiler plus the compile time and run time in ROC REPL.

Stream: ideas

Topic: timer in repl

Audrey Gao (Mar 28 2023 at 17:46):

Richard Feldman (Mar 28 2023 at 18:31):

Notification Bot (Mar 28 2023 at 18:31):

Brendan Hansknecht (Mar 28 2023 at 18:32):

Richard Feldman (Mar 28 2023 at 18:35):

Richard Feldman (Mar 28 2023 at 18:36):

dank (Mar 28 2023 at 18:37):

Richard Feldman (Mar 28 2023 at 18:38):

Richard Feldman (Mar 28 2023 at 18:38):

dank (Mar 28 2023 at 18:39):

Brendan Hansknecht (Mar 28 2023 at 18:41):

Brendan Hansknecht (Mar 28 2023 at 18:41):

Richard Feldman (Mar 28 2023 at 18:48):

Audrey Gao (Mar 28 2023 at 18:58):

Brendan Hansknecht (Mar 28 2023 at 19:03):

Brendan Hansknecht (Mar 28 2023 at 19:03):

Richard Feldman (Mar 28 2023 at 19:12):

Richard Feldman (Mar 28 2023 at 19:12):

Richard Feldman (Mar 28 2023 at 19:12):

Richard Feldman (Mar 28 2023 at 19:13):

Audrey Gao (Mar 28 2023 at 19:18):

Brendan Hansknecht (Mar 28 2023 at 19:24):

Richard Feldman (Mar 28 2023 at 19:29):

Richard Feldman (Mar 28 2023 at 19:29):

Audrey Gao (Mar 28 2023 at 19:38):

Richard Feldman (Mar 28 2023 at 20:35):

Audrey Gao (Mar 29 2023 at 05:08):

Luke Boswell (Mar 29 2023 at 05:45):

Luke Boswell (Mar 29 2023 at 05:46):

Anton (Mar 29 2023 at 07:40):

Audrey Gao (Mar 29 2023 at 14:34):

Anton (Mar 29 2023 at 14:38):

Audrey Gao (Mar 29 2023 at 14:45):

Anton (Mar 29 2023 at 14:49):

Audrey Gao (Mar 29 2023 at 14:55):

Anton (Mar 29 2023 at 14:56):

Anton (Mar 29 2023 at 14:57):

Anton (Mar 29 2023 at 14:57):

Audrey Gao (Mar 29 2023 at 15:01):

Anton (Mar 29 2023 at 15:13):

Anton (Mar 29 2023 at 15:36):

Audrey Gao (Mar 29 2023 at 15:46):

Anton (Mar 29 2023 at 15:54):

Audrey Gao (Mar 29 2023 at 16:00):

Anton (Mar 29 2023 at 16:00):

Audrey Gao (Mar 29 2023 at 16:06):

Anton (Mar 29 2023 at 16:17):

Audrey Gao (Mar 29 2023 at 16:27):

Anton (Mar 29 2023 at 16:38):

Anton (Mar 29 2023 at 16:40):

Anton (Mar 29 2023 at 18:09):

Audrey Gao (Mar 30 2023 at 03:05):

Brendan Hansknecht (Mar 30 2023 at 04:17):

Brendan Hansknecht (Mar 30 2023 at 04:18):

Brendan Hansknecht (Mar 30 2023 at 04:21):

Brendan Hansknecht (Mar 30 2023 at 04:21):

Brendan Hansknecht (Mar 30 2023 at 04:22):

Audrey Gao (Mar 31 2023 at 05:46):

Audrey Gao (Mar 31 2023 at 23:03):

Brian Carroll (Apr 01 2023 at 04:17):

Brian Carroll (Apr 01 2023 at 04:23):

Brian Carroll (Apr 01 2023 at 04:23):

Audrey Gao (Apr 01 2023 at 04:26):

Brian Carroll (Apr 01 2023 at 04:26):

Brian Carroll (Apr 01 2023 at 04:27):

Brian Carroll (Apr 01 2023 at 04:28):

Brian Carroll (Apr 01 2023 at 04:29):

Audrey Gao (Apr 01 2023 at 04:33):

Brendan Hansknecht (Apr 01 2023 at 04:54):

Brian Carroll (Apr 01 2023 at 05:12):

Brian Carroll (Apr 01 2023 at 05:38):

Audrey Gao (Apr 01 2023 at 05:51):

Brian Carroll (Apr 01 2023 at 06:01):

Audrey Gao (Apr 01 2023 at 06:09):

Brian Carroll (Apr 01 2023 at 06:13):

Audrey Gao (Apr 01 2023 at 06:25):

Brian Carroll (Apr 01 2023 at 06:34):

Brian Carroll (Apr 01 2023 at 06:36):