Karl (Jul 13 2026 at 02:41): Karl (Jul 13 2026 at 02:41):https://github.com/grayrest/roc-tower-platform
I decided to spend this weekend making a platform for Tokio and the Tower middleware stack. This is mostly LLM generated so heads up to the people that are annoyed by that. This isn't intended to be a production platform. I did it to check on Roc's progress and am sharing as a way of providing feedback to Richard and the other contributors. The overall experience was good and much nicer than my last go-round with the language before the big rewrite decision.
The initial plan was to do it Axum style but I couldn't get the IntoResponse pattern working on strings so I decided to instead work through what re-frame would look like as a web framework. The result is a platform with pure handlers plus a system for managing effects on the input and output sides.
I did a RealWorld implementation as a sanity check of the API design and the design worked and it's reasonably fast–45k req/s for articles on my Macbook M1, 5k req/s on the lists–so I'm pretty pleased on the whole. The sqlite stuff is kind of bad–I'm getting 2.7ms of framework overhead on the list requests which seems excessive–but that wasn't the focus. I'll be following up with related questions/comments.
Karl (Jul 13 2026 at 02:43):The blocker for the IntoResponse plan was that where only works on nominal types and there's no equivalent to Rust's orphan rule so strings/numbers can't be covered.
Karl (Jul 13 2026 at 02:50):Claude decided to box two things going across the host boundary: the handlers and the per-request Model. I'm not familiar enough with this sort of thing to know if the type erasure/boxing is necessary and the model, in particular, results in a number of unboxing calls and the Rust side doesn't invoke it so it seems not so great.
The bug report I have for the rust glue is that generating the glue with a capitalized Model crashes the glue generator. I can have the LLM generate a minimal test case if needed.
Luke Boswell (Jul 13 2026 at 02:52):It's by design that anything crossing the boundary to the host that isn't concrete must be boxed -- as the platform host is prebuilt it cannot know what shape or layout that type will be when an app provides an implementation.
Richard Feldman (Jul 13 2026 at 02:54):super cool! :smiley:
Richard Feldman (Jul 13 2026 at 02:54):yeah please share the repro of the Model crashing glue generator :thumbs_up:
Karl (Jul 13 2026 at 02:55):I'm okay with how Cofx.withN landed but I'm used to languages with metaprogramming or compile time generation so it's more hands-on than I'm used to. Not sure if it could be done better.
One thing I'm not showing off in the examples is how the record version works:
{
model: Req.model,
now: Req.now!
}.Cofx
->Cofx.resolve(|{model, now}| ...)
also, I have one example (rocci bird) where Model is boxed but we optimize its updates such that in practice it only gets allocated once at the beginning of the program, and then all of its subsequent updates get made in-place to the existing heap allocation anyway (as long as it remains unique) so it's 1 heap allocation for the whole life of the program instead of per change of the model
Richard Feldman (Jul 13 2026 at 02:55):that said, the optimization hasn't been battle-tested on other code bases, so I'm curious whether it applies in yours!
Richard Feldman (Jul 13 2026 at 02:56):(it just landed, and we haven't pushed a nightly build since it landed, so you'd need to build the compiler from source to try that out right now :sweat_smile:)
Karl (Jul 13 2026 at 02:56):I'm on source from yesterday at like 2PM
Richard Feldman (Jul 13 2026 at 02:57):yeah if you rebuild with current main you should have that optimization
Karl (Jul 13 2026 at 02:59):One unexpected footgun I ran into is that the Roc compiler isn't built with ReleaseFast so I was getting like 65s for glue generation and 13s for specialization. Switching to release was a 150x and 300x, respectively, improvement.
Karl (Jul 13 2026 at 03:00):That's everything I have notes to mention. Experience was pretty good on the whole.
Luke Boswell (Jul 13 2026 at 03:01):Yeah our debug build has extremely defensive checks on everything -- and these are zero cost in release builds.
Karl (Jul 13 2026 at 03:26):LLM summary of the Model crash
Richard Feldman (Jul 13 2026 at 12:10):Luke Boswell said:
Yeah our debug build has extremely defensive checks on everything -- and these are zero cost in release builds.
yeah in addition to not having llvm optimizations, the debug build is like "wait, are any compiler invariants violated right now? If so, panic" all the time and it's very costly (but helps with debugging the compiler, of course) :smile:
Anton (Jul 13 2026 at 12:20):Karl said:
One unexpected footgun I ran into is that the Roc compiler isn't built with ReleaseFast so I was getting like 65s for glue generation and 13s for specialization. Switching to release was a 150x and 300x, respectively, improvement.
You may already know this but nightlies are built with ReleaseFast and when building from source you can use zig build build-release.
Karl (Jul 13 2026 at 12:27):The issue was mostly just knowing to do it. When I hit this I was trying to implement IntoResponse and thought I'd hit some pathological corner case to get a 15s build and realized I was on debug when looking into it. I'm used to the 10x difference from LLVM optimizations but 300x was well beyond my expectations.
Anton (Jul 13 2026 at 12:36):People always underestimate how fast Roc can be :smile:
Karl (Jul 19 2026 at 05:50):I've been poking at this all week and in the process I ran across a number of bugs that I traced back to source and vibecode patched which are now reported but it means the project needs my Roc fork to actually build.
In the interest of making the sql experience better, I pulled in the analysis code from sqlx and so the platform generates Roc structs from ## @sql tagged strings. The RealWorld example was updated to use this functionality.
I then did an extended perf tour. Got about 50% more base perf for ~70k req/sec for article and 11k for list and automatic cache invalidation for 120-140k req/sec across the board.
The initial attempt was to use sqlite's JSON features to build the output JSON in the database and then send the byte directly to the kernel. This worked reasonably for single entity endpoints but the serialization of aggregate queries was mediocre. I suspected sqlite's json serializer wasn't especially tuned and thought that Roc's was more likely to be tuned if not now then in the future since I expect it to be heavily used.
Profiling showed allocation as the hotspot with malloc/free at ~30% and memcopy at ~20%. The data needs to be copied once out of sqlite's btree and a second time into the kernel to get to the socket but I had the "clever" idea to use one of Turso's APIs to get a stable row buffer, write the correct bytes for Roc lists/structs, and cast them like Abomonation to bypass the allocation/marshaling of individual rows/values. In the process of fighting Claude's displeasure with doing this I had it put in an arena allocator as Roc's allocator and that turned out to drop about 95% of the struct CPU thrashing and I can do the free in a reaper thread after the request was sent. So I picked up Turso support but my casting idea turned out to be bad. It also turned out that Roc was faster on serialization bound workloads even with the memory pressure and a serialization isn't currently that much faster while using more cpu because contention in sqlite is bad.
Turso as a backing option works fine for simple requests where roughly even for the no-join endpoints but the complex queries (e.g. realworld list view is a 4 table join) were 2-4x faster in sqlite just due to engine maturity. Left it in because syncing databases is still a neat feature.
In profiling all this I also noticed that returning json was eating 5% cpu on utf8 checking. The encoder produces a List(U8) which is a utf-8 encoded string which then goes through Str.from_utf8_lossy (5% cpu) to go through the platform boundary and out. Monkeypatching a separate method to bypass the utf8 check confirmed it and I did the rest of my benchmarking with it my monkey patch but removed it just before pushing. There's a comment in the source mentioning reusing the input list (what my monkeypatched alternate method did) but I also think most of the CPU overhead can be recovered without the need for completely bypassing by doing the SIMD ascii prefix 0 scan optimization.
With all the reasonable straightforward perf avenues explored I thought "I have a framework with pure handlers and full control of every possible input, caching is just memoization" and after three days of fighting Claude's insistence that it was impossible, managed to get full and automatic cache invalidation for unauthenticated requests on the platform.
It turns out that sqlite has a hook that will provide the exact accessed tables for any query (including triggers, etc) and when combined with the sqlx analysis lets the platform invalidate affected caches on write. It's aggressive so if sqlx analysis fails for whatever reason it invalidates everything derived from every table touched. All other data comes from the request so it's just a matter of ensuring that any extracted piece is tied to the cache key. The platform has an http fetch feature that poisons the request but it'd be straightforward to cache the result and tie the output cache time to the request cache time. It's currently unauthenticated only because Claude insists caching authenticated requests a security risk and I haven't thought through the security implications in detail.
The request extraction isn't completely sound. It's possible for a coeffect extractor to access a field using .and_then without adding a corresponding entry to the cache key. This earlier feature request should close the hole.
I also wanted to key the cache entries to only the exact extracted values in the body so I monkeypatched access to the hasher. I presume it's locked away for some reason but that's my use case for it.
Luke Boswell (Jul 19 2026 at 08:34):The hasher isnt exposed so we can change the implementation in future and it doesnt break anyone.
Last updated: Jul 23 2026 at 13:15 UTC