Ryan Barth (Jul 04 2024 at 05:25): Ryan Barth (Jul 04 2024 at 05:25):Since the value of an expect expression must be a bool and Tasks do not implement the Equality ability is there any way to use roc test on code with effects?
Luke Boswell (Jul 04 2024 at 05:26):There's a section in the Module Params design proposal which talks about the design for Simulation Tests, and Effect Recordings.
Ryan Barth (Jul 04 2024 at 05:31):So the practical answer for today is to go wire up some external test harness that actually calls the compiled binary?
Sam Mohr (Jul 04 2024 at 05:33):Yeah, you could look at how the basic-cli CI tests work for an example of that
Ryan Barth (Jul 04 2024 at 05:33):Perfect, thanks!
Sam Mohr (Jul 04 2024 at 05:33):https://github.com/roc-lang/basic-cli/blob/main/ci/expect_scripts/echo.exp
Ryan Barth (Jul 04 2024 at 05:36):Since I have your attention has anyone tried implementing some of the more popular benchmarking suites out there? I was playing around with implementing the Techempower web framework suite and Roc might be the fastest non-manual-memory-managed language on there. And I havent even gotten to performance tweaking...
Brendan Hansknecht (Jul 04 2024 at 05:38):Really? that surprise me. As in, today, roc has a lot of sharp edges that I would expect to mess up perf on something like that without tweaking. Totally makes sense when thinking about where roc should land in the long term (somewhere around go hopefully)
Ryan Barth (Jul 04 2024 at 05:42):Maybe its cheating because I am using sqlite and not incurring the TCP stack latency, but i was hitting low 300k on the score for the single query perf.
https://www.techempower.com/benchmarks/#hw=ph&test=db§ion=data-r22&o=4
Ryan Barth (Jul 04 2024 at 05:43):I will run the full suite when i am done to get the apples to apples comparison since those results are run on a VPS
Ryan Barth (Jul 04 2024 at 05:43):But still, that screams...
Sam Mohr (Jul 04 2024 at 05:45):That would be good to check! Though you're correct that Roc should have a pretty good performance edge by default over the other languages there (JIT perf improvements ignored) once Roc is ready, there are two big blockers I can think of. First is as Brendan said, we have a long way to go before Roc performs as well as its design allows for. Second is that we haven't figured out how to make Roc communicate efficiently between applications and platforms for shared memory.
Luke Boswell (Jul 04 2024 at 05:46):I expect JSON decoding will be much slower than encoding. We have ideas for writing a fast pure roc decoder based on ideas from the simdJSON library but I haven't had a change to really get into it. Maybe with the rewrite from the new Encoding/Decoding API's that Trevor is working on we could try it out.
Luke Boswell (Jul 04 2024 at 05:48):Basically break into two stages, first is a fast pass through the input bytes using branchless code to segment, and then parsing.
Luke Boswell (Jul 04 2024 at 05:59):For that test, it looks like everyone else is using Postgres. I wonder how roc scores if we include roc-pg?
Ryan Barth (Jul 04 2024 at 06:02):I was getting there :smile: . I was going to make basic-webserver + roc-pg a separate stack. First try is all just basic-webserver (and roc-json)
Ryan Barth (Jul 04 2024 at 06:11):I think it will be nice to start tracking some of these numbers internally. Eventually I think it will make a nice promotional piece :smile:
Ryan Barth (Jul 04 2024 at 06:15):That's a first :shock:
$ roc check
thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
zsh: IOT instruction (core dumped) roc check
whoa! Can you post a reproduction somewhere?
Ryan Barth (Jul 04 2024 at 16:54):https://github.com/roc-lang/roc/issues/6867
Brendan Hansknecht (Jul 04 2024 at 17:53):oh and the overflow is in roc check...super intriguing.
Agus Zubiaga (Jul 05 2024 at 21:32):Ryan Barth said:
I was getting there :smile: . I was going to make basic-webserver + roc-pg a separate stack.
Note: roc-pg decoding isn’t as fast as it could yet. I have some plans to improve that later.
Ryan Barth (Jul 05 2024 at 21:45):For any one who is interested in the outcome here are the initial results.
Couple caveats. I took this on to get practice writing roc. I thought this would be a good "platform tour" for me to learn to work with basic-webserver. I wrote this as though I was going to have to maintain it, not for bit twiddling performance. I am sure it can go faster.
Next, this initial version is using sqlite because that is what is built into basic-webserver. Because of that I did have to disable some of the database checks in the TFB test suite to allow the benchmark to run. Because of that the results are not formally verified by the benchmark framework. This is a limitation of the framework not supporting sqlite. That said I manually validated the behavior and I believe have made an honest implementation. I will be able to write a formalized version if I do one targeting roc-pg.
Personally I think this is a really strong showing for such a nascent project, and I learned a ton!
Richard Feldman (Jul 05 2024 at 22:01):really cool! cc @Folkert de Vries in case you want to try out Nea on this benchmark
Ryan Barth (Jul 05 2024 at 22:01):One cool bit that does not show up in the results page (but should!) is the implementation size. The benchmark counts the code size for each implementation. My roc implementation was the smallest in the suite I measured by a factor of 3.
* Rust/actix: 1537
* CSharp/aspnetcore: 2751
* Roc/basic-webserver: 87
* Python/django: 409
* C++/drogon: 2767
* Go/echo: 292
* JavaScript/express: 1338
* Python/fastapi: 853
* Go/gin: 591
* PHP/laravel: 1431
* Elixir/phoenix: 1121
* Ruby/rails: 553
* Java/spring: 665
* Zig/zap: 577
Sample count command:
Running "cloc --yaml --follow-links . | grep code | tail -1 | cut -d: -f 2" (cwd=/FrameworkBenchmarks/frameworks/Rust/actix)
I’m curious how much optimization the others are doing (vs just writing a typical-looking implementation)
Richard Feldman (Jul 05 2024 at 22:02):this is a situation where supporting interpolation in string patterns would undoubtedly improve performance
Ryan Barth (Jul 05 2024 at 22:03):The suite categorizes implementations into "Realistic" and "Stripped". These values are self reported and I would have to go read some to give you an answer to how honest the implementations are.
Richard Feldman (Jul 05 2024 at 22:04):like if we could do "/users/$(userId)" -> ... that would perform better (and read better imo) than ["users", userId] -> ...
Folkert de Vries (Jul 05 2024 at 22:10):yeah trying this with nea would be really cool. I'll try to make some of the basic-webserver url/db stuff work with nea
Ryan Barth (Jul 05 2024 at 22:12):What is nea?
Folkert de Vries (Jul 05 2024 at 22:12):https://github.com/tweedegolf/nea
Folkert de Vries (Jul 05 2024 at 22:13):in short, another roc webserver platform that does some cool things with allocators
Folkert de Vries (Jul 05 2024 at 22:13):which in theory gives better, but certainly more predictable performance
Ryan Barth (Jul 05 2024 at 22:13):Oh I have seen this! I already had it starred :laughing:
Folkert de Vries (Jul 05 2024 at 22:19):yeah so specifically what needs work I think is the database (exposing some TCP socket, I think). the rest I think is just a matter of copying the roc code and that should work fine
Ryan Barth (Jul 05 2024 at 22:24):From writing it I would like to point out 2 places I think basic-webserver made it hard to have a 100% conformant implementation.
Date header to the response. Particularly the day of week calculation. Brendan Hansknecht (Jul 05 2024 at 22:27):We also run the sqlite database with a mutex. Meaning only one thread can interact at a time. I bet that hurts perf (though it might just add back in the equivalent of the database latency)
Richard Feldman (Jul 05 2024 at 22:41):concurrency is a work in progress
Richard Feldman (Jul 05 2024 at 22:41):concurrent tasks, that is
Folkert de Vries (Jul 05 2024 at 22:41):but individual requests can run independently right?
Folkert de Vries (Jul 05 2024 at 22:42):at least in nea I think they can as far as the platform is concerned
Ryan Barth (Jul 05 2024 at 22:42):I am glad I didn't just miss it by being dumb / new :laughing:
Ryan Barth (Jul 05 2024 at 22:44):Most of the tests require the database (all except json and plaintext). From what @Brendan Hansknecht said and the fact you are running tokio / hyper under the hood I would imagine the answer is yes, except you are stuck waiting for that mutex.
Folkert de Vries (Jul 05 2024 at 22:47):but, why is this
We also run the sqlite database with a mutex
why is exclusive access needed?
Ryan Barth (Jul 05 2024 at 22:48):Using something like RWLock would probably help. Many of the tests are read only.
Brendan Hansknecht (Jul 05 2024 at 22:48):It isn't need. That's just how it's currently inplemented
Ryan Barth (Jul 05 2024 at 22:49):But the way the API is setup the platform cant make guarantees about whether the given query is mutating or not.
Folkert de Vries (Jul 05 2024 at 22:49):ah ok but I'll need to do some custom things anyway for nea. So perhaps we can lift that restriction in nea
Folkert de Vries (Jul 05 2024 at 22:49):isn't dealing with mutation the DBs job?
Brendan Hansknecht (Jul 05 2024 at 22:49):The correct implementation would be to set sqlite to multi threaded mode and spawn one connection per thread. Then access it through that. Also would be great to use prepared queries with longer lifetimes instead of compiling the query on every call.
Brendan Hansknecht (Jul 05 2024 at 23:10):I'll definitely take a look at multi threading for sqlite as I work on updating the API. I want to make that nicer
Brendan Hansknecht (Jul 06 2024 at 01:04):Only just starting to work on things, but the sqlite-improvements branch now uses an sqlite connection per thread. For a super basic local test with a 4 core server, got a 17% speed increase. Honestly less than I expected, but still nice.
Brendan Hansknecht (Jul 06 2024 at 01:05):Just benchmarked with examples/sqlite3.roc with the print commented out.
Last updated: Jul 06 2025 at 12:14 UTC