Oskar Hahn (Sep 07 2024 at 16:04): Oskar Hahn (Sep 07 2024 at 16:04):Hi, this is a followup question from here I watched the Video from the April Meetup, but there are still open questions. The background is, that I have to create a RocList implementation for Go.
My first question is, where is the refcount (RC) and where is the original len counter(OL). When I understand the zig code correctly, then it is easy, when the alignment of the list elements is lower or equal then usize. In this case, the RC is always the usize bytes before the data and OL (if it exists) is always the usize bytes before the RC. But if the alignment of the list elements is bigger, then you have to go "one alignment" before the data. If an OL exists, then you find the OL there and the RC usize-bytes afterwards. If no OL exists, then you find the RL "one alignment" before the data. Is this correct?
My second question is, how to know, if an OL exists. Is it correct, that an OL exist, when the elements of the list are refcounted? How do I know, if this is the case? I think, this depends on there type. Is there a trick to know, which types are refcounted? For example for an List Str, I would guess, that the elements are refcounted and therefore an OL exists? But for a List U64, I would guess the elements are not refcounted, so the RC is 8 bytes before the data, even on a wasm build, where RC is only 4 bytes long. Is this correct as well?
I can not find the place in the zig code, where the OL is set. I followed the code from RocList.fromSlice(). The space for OL gets allocated here. And the RC get set here. But where is the code, that sets the OL? I think it happens in RocList.incref(), But this function is never called in RocList.fromSlice()
When roc returns a List to my host, I have to reduce the refcounter. What does this mean?
a) I have to check, if the list elements are refcounted to find out, if there is an OL
b) Then I have to find the place of the RC in memory. I have to consider, if the List is a seamless slice, to do this
c) If RC is greater then 1, I just reduce it by one and everything is done.
d) If RC is 1, I have to deallocate the list and reduce the RC of each elements (if they are refcounted) and deallocate them as well, if they have a RC of 1. To know, how many elements there are, I have to look into OL. To dealocate the List data, I also have to deallocate the RC and the OL.
Is all of this correct?
Brendan Hansknecht (Sep 07 2024 at 16:45):This is off the top of my head and some of it could be wrong
Also, man, I really wonder if we can wrap the zig code in a reasonable generic way such that implementing in various languages is easier and less bug probe. Like at least the core of the zig library would be great to share via cffi.
Oskar Hahn (Sep 07 2024 at 18:34):Brendan Hansknecht said:
- Rc should always be at data pointer minus usize width. Original Len should always be directly before that if it exists. Totally possible we have code handling this wrong cause larger than usize alignment is very rare and probably not thoroughly tested.
You are right. After carefully reading the zig-code again, that does the allocation, I can see now, that RC is always the usize-bytes before data. But when the alignment is greater then usize (for Example List U64on a wasi build), then alignment-bytes get allocated before the data. This has to be considered, when dealocating the List. You can not just dealocate from RC. You have to dealocate from data - (alignment of elements), even when there is no OL.
- It is only set when incrementing the refcount from unique to shared. It is not set any other time. Unique lists don't need to have it set. This enables unique lists to mutate length without worrying about the len on the heap..it is only required once shared and a seamless slice might need it.
How do I now, if a list is unique or shared? Is the following statement correct? If it is a seamless slice, then it is always shared. If is not a seamless slice then it does not matter. I just look at the len-attribute.
So in conclusion, when dealocating a seamless slice of refcounted elements with a RC of 1, you have to read the OL. When RC is higher or the elements are not refcounted or it is not a seamless slice, then ignore it. But when dealocating the data of the slice, you have to know if the OL exists, to deallocate it.
Also, man, I really wonder if we can wrap the zig code in a reasonable generic way such that implementing in various languages is easier and less bug probe. Like at least the core of the zig library would be great to share via cffi.
That would be so helpful. I was already thinking about compiling the builtin and link it to my go code. I have to use cgo anyway. But the builtins have so many code, that I don't need, that I don't want to include it.
Brendan Hansknecht (Sep 07 2024 at 23:28):Yeah, that sounds correct. Basically freeing a seamless slices uses that original length stored on the heap to free the underlying elements of the original list. A regular list will always point to a full list. So it can safely use the length on the stack for deallocating.
Oskar Hahn (Sep 07 2024 at 23:33):Puh. After hours of doing stupid mistakes, I think I finally have a working version of RocList in Go that does not crash when sending lists to roc or dealocating lists returned from roc.
Thank you for the help. I think, I understand the memory layout. (Lets see how long this statement holds :sweat_smile: )
Brendan Hansknecht (Sep 07 2024 at 23:55)::party_ball: Awesome
Last updated: Jul 05 2025 at 12:14 UTC