Stream: performance

Topic: SQLite owned buffer

Richard Feldman (Sep 02 2026 at 21:07):

Karl said:

the SQLite buffer is (obviously) host side and getting it into Roc requires a copy

:thinking: why would that require a copy?

Richard Feldman (Sep 02 2026 at 21:08):

it's just memory, right? you can make a List in the host with reserved capacity for the buffer and then hand that directly into ordinary Roc code, which should be able to write into it as long as its refcount remains at 1

Karl (Sep 02 2026 at 21:09):

you can make a List in the host with reserved capacity for the buffer

SQLite owns the memory. As far as I know there isn't a way to do a slice on a host owned buffer

Richard Feldman (Sep 02 2026 at 21:10):

hm, "owns the memory" in what sense? :thinking:

Richard Feldman (Sep 02 2026 at 21:10):

like it gives you an address, but it reserves the right to write to it or something?

Karl (Sep 02 2026 at 21:10):

This is SQLite's cursor/results buffer. Owns as in allocates/deallocates.

The encoders+decoders are arranged so that the bytes go from SQLite's cursor to the output buffer with no intermediate allocation.

Karl (Sep 02 2026 at 21:13):

Let me see if it's compiling/working and push it to github.

Richard Feldman (Sep 02 2026 at 21:14):

I see, so it sounds like there are two missing pieces here:

1. We don't currently have a way to say "encode into this buffer" or "decode into this buffer" (I think this would be easy to add, but I haven't thought about it before)
2. if it's a List, we can't allow it to be resized because that would reallocate and SQLite wouldn't be okay with that

Richard Feldman (Sep 02 2026 at 21:14):

does that sound right?

Karl (Sep 02 2026 at 21:36):

Trying for a coherent explanation:

I have a web platform that's mostly a sandbox for ideas. After last month's discussion about encoders/decoders I decided to see how far I could push the encoder/decoder APIs. The one that's relevant for this discussion is that I'm implementing a decoder for SQLite query row results wired up to the encoder for Json. For a query SQLite copies data from its btree into (my term) the results buffer and provides a cursor for walking through the rows. The data also needs to be formatted/copied into the send buffer since MacOS doesn't have something like io_uring. So my goal is to get down to the minimum so the response is faster. I have it working but it required some massaging.

Problems:

All non-small RocStr/RocList are refcounted and the code for decref is:

alloc_ptr = get_allocation_ptr()        // == bytes, for a big (non-slice) string
rc = *(alloc_ptr - 8)                    // the refcount word
if rc == 0 { return }                    // REFCOUNT_STATIC_DATA: never free
if fetch_sub(rc) == 1 { host.dealloc(alloc_ptr - 8) }

The alloc_ptr on the host is in SQLite's memory space and control so there's no refcount at that position. There's a rc=0 variant that's used for static strings but that also requires the byte at ptr-8 to be the right value. There would need to be some variant of List/Str that knows it's host owned and doesn't do the refcounting.

The other factor for this particular problem is that the memory is only valid until the cursor moves so there needs to be some sort of lifetime feature. I have a hacky non-effectful host function sql_col_str_json_into that copies the bytes from the cursor into the output.

Luke Boswell (Sep 02 2026 at 21:38):

can you use a seamless slice? from the hosts sqlite buffer

Karl (Sep 02 2026 at 21:39):

seamless slice sets requires a backing Roc allocation and shares its refcount

Karl (Sep 02 2026 at 21:42):

It's hacky but the perf is super fast. Total roc portion of the handler time is 16.4µs instead of 39.2µs.

Richard Feldman (Sep 03 2026 at 02:07):

I think we had

Karl said:

seamless slice sets requires a backing Roc allocation and shares its refcount

this part is actually fine - we designed for this situation (although the lifetime is a separate question) - the three fields in a str or list are:

• bytes_ptr
• capacity_or_alloc_ptr
• length

so what you can do is set the alloc ptr to be totally unrelated to where the bytes_ptr is, and just have your refcount be right there - e.g. you allocate 16B yourself, and use it to store (refcount, NULL) and just have capacity_or_alloc_ptr point to the NULL, so the refcount is right in front of it

Richard Feldman (Sep 03 2026 at 02:08):

(I think you can maybe even get away with not bothering to allocate the NULL, but there might be some super weird edge case where you're right at the end of the address space or something, and I also think most allocators will allocate a minimum of 16B anyway so it's probably zero-cost anyway)

Richard Feldman (Sep 03 2026 at 02:10):

anyway, that doesn't address the lifetime part of it

Karl said:

The other factor for this particular problem is that the memory is only valid until the cursor moves so there needs to be some sort of lifetime feature.

when does the cursor actually move? like is the platform fully in charge of when it moves, or do you expose the cursor to the app author to move directly?

Karl (Sep 03 2026 at 02:16):

I'll give the seamless slice a go. Not an approach I would have come up with.

The cursor moves by a call to sql_step_row! which returns a token that has to be passed to sql_col_str_json_into which asserts the allocation is valid is the lifetime soundness of the system.

Karl (Sep 03 2026 at 02:18):

I'm not committed to any particulars for this design. I'm just trying to eliminate copies via encoders/decoders.

Richard Feldman (Sep 03 2026 at 02:18):

yeah I think @Brendan Hansknecht thought to design seamless slices for this use case a couple years ago - full credit to him for the payoff here! :smile:

Karl (Sep 05 2026 at 01:38):

I've been thinking about this some more. The slice trick is working well for me so I'm thinking about lifetimes. If there was a way to force a clone on incref (not literally, my mental model of the mechanism) I believe this could could be done in a Cursor.each!(cursor, |row| ...) pattern, set up a decoder for the rows that produces seamless slices, and have zero copy on field reads and just the encoded copy for sending over the wire.

Richard Feldman (Sep 05 2026 at 02:20):

interesting - so basically the original one has a known lifetime, and if you want to keep your own version around, it will be copied

Richard Feldman (Sep 05 2026 at 02:21):

a refcount of "immortal" (as in, "this data lives in the readonly static data section of the binary and so you can't even change the refcount") is already special-cased, so we could add a second sentinel refcount number for this

Karl (Sep 05 2026 at 02:28):

It's a slice into ephemeral data and as long as access is scoped then it should be safe but the problem is escape which the refcount guards against.

Brendan Hansknecht (Sep 05 2026 at 02:33):

Do you have the ability to let the empheral data be controlled and freed by roc and make new empheral data if the SQL access continues? Or is the empheral data totally controlled by SQL?

Brendan Hansknecht (Sep 05 2026 at 02:35):

Even if SQL controls the ephemeral storage, you could still have a defensive check if the refcount. If the data has not yet been freed by roc, you could return an error.

Though really the best pattern I have seen for this kinda of scoped lifetimes is callbacks that get access to the data and must return before you grab the next data.

Though maybe there is a better pattern now.

Karl (Sep 05 2026 at 02:46):

This is the SQLite cursor buffer, where SQLite copies btree data into result rows. It's only valid until the cursor moves.

Brendan Hansknecht said:

Though really the best pattern I have seen for this kinda of scoped lifetimes is callbacks that get access to the data and must return before you grab the next data.

This is basically what I want to do but there needs to be some sort of protection against putting the data in a struct.

Richard Feldman (Sep 05 2026 at 02:57):

right, the trouble with that is if you just return the thing with the troublesome lifetime from the lambda :sweat_smile:

Richard Feldman (Sep 05 2026 at 02:57):

also hi @Brendan Hansknecht! :smiley:

Richard Feldman (Sep 05 2026 at 02:59):

you might need to do both actually :thinking:

Richard Feldman (Sep 05 2026 at 03:00):

as in have a "slice with clone on incref" and then provide that to a callback closure - and if you try to return something from the closure it will result in an incref and clone

Richard Feldman (Sep 05 2026 at 03:03):

because otherwise, if it's still unique within the roc application when you go to advance the cursor, then the incref happens in the host - at which point it's too late to have cloned the one in the roc application

Richard Feldman (Sep 05 2026 at 03:05):

but overall, the "slice with clone on incref" idea is appealing to me because its characteristics seem to be:

• as long as you use it as intended, there is no clone and approximately no runtime cost compared to what you'd do with the memory in C
• if you don't use it as intended, the penalty is a clone rather than memory unsafety

Richard Feldman (Sep 05 2026 at 03:06):

off the top of my head, I suspect the perf of adding a second sentinel could be fine because:

• we already branch on the first sentinel value
• if you've already paid the misprediction cost on that, a second branch on the same integer right afterwards should be ~free

Brendan Hansknecht (Sep 05 2026 at 03:45):

Can you force clone on incref? I don't think roc supports that. Roc will only trigger the clone path on attempt of mutation.

Karl (Sep 05 2026 at 03:46):

No, it's a feature request.

Brendan Hansknecht (Sep 05 2026 at 03:46):

This is definitely a case where defense is easy (assert roc freed), but active niceness is hard.

Brendan Hansknecht (Sep 05 2026 at 03:47):

Another option would be some sort of double indirection where the host the can swap out the reference

Brendan Hansknecht (Sep 05 2026 at 03:47):

So if roc retains a clone, the platform could duplicate the data and swap out the reference.

Brendan Hansknecht (Sep 05 2026 at 03:48):

Definitely a pattern that should get a proper support story. Mutable host data that you really just want to reference once for encode is hard with current apis.

I feel like it should be doable, but might take a trick. Probably could be made more simple.

Last updated: Sep 24 2026 at 15:59 UTC