jan kili (Dec 13 2024 at 20:02): jan kili (Dec 13 2024 at 20:02):JanCVanB said:
... Tomorrow I'll revisit the memory zeroing bug...
I've created a minimal bug reproduction (using just a few additions to basic-cli@main) of this confusing obstacle I've been hitting in my roc-on project (before and after switching to purity inference). I get this output, which demonstrates the zeroing I'm also seeing with all tag union discriminants (seemingly because discriminant is the second field in the union).
jan@framey:~/_code/JanCVanB/basic-cli$ roc version
roc nightly pre-release, built from commit 7495495 on Do 12 Dez 2024 09:43:52 UTC
jan@framey:~/_code/JanCVanB/basic-cli$ ./bug_repro___run.sh
+ '[' -z '' ']'
+ echo 'Warning: ROC environment variable is not set... I'\''ll try with just '\''roc'\''.'
Warning: ROC environment variable is not set... I'll try with just 'roc'.
+ ROC=roc
+ roc build --lib ./platform/libapp.roc
0 errors and 0 warnings found in 143 ms while successfully building:
./platform/libapp
+ cargo build --release
Finished `release` profile [optimized] target(s) in 0.05s
+ '[' -n '' ']'
+ cp target/release/libhost.a ./platform/libhost.a
+ roc build --linker=legacy build.roc
0 errors and 0 warnings found in 287 ms while successfully building:
build
🦅 happy: {bar: 22, baz: 33, foo: 11}
🦀 sad: ZeroedStruct { bar: 22, baz: 0, foo: 0 }
jan@framey:~/_code/JanCVanB/basic-cli$
I'm happy to create a github issue for this, but I'm unclear which repo is responsible - maybe a roc-lang/roc/crates/glue/src/RustGlue.roc bug?
Brendan Hansknecht (Dec 13 2024 at 20:32):Yeah, known c abi bugs
Brendan Hansknecht (Dec 13 2024 at 20:33):Rust is expecting {bar: 22, baz: 33, foo: 11} to be passed in a single register.
Brendan Hansknecht (Dec 13 2024 at 20:33):Cause it is 3 U8s
Brendan Hansknecht (Dec 13 2024 at 20:33):We are probably passing it in 3 different registers
Brendan Hansknecht (Dec 13 2024 at 20:34):Thus only the first value is correct (cause it is in the first register)
jan kili (Dec 13 2024 at 20:59):Oh, okay! I'm so glad this is known and clear to you.
jan kili (Dec 13 2024 at 21:00):Can I debug it by telling Rust to expect multiple registers? (no idea where to start, but happy to look up things)
Brendan Hansknecht (Dec 13 2024 at 21:01):If you make rust recieve 3 u8s instead of a struct, I think that should work
Brendan Hansknecht (Dec 13 2024 at 21:02):This is a major issue we have with small types. We really need to revamp how we generate c-abi and deal with all the various cases.
jan kili (Dec 13 2024 at 21:03):Do you mean [u8]? Is there a better one-to-one container? Since I have dozens of nested struct types with lots of U8s, I'm looking for structure.
jan kili (Dec 13 2024 at 21:05):Or would ballooning to U32/64 minimum sizes and then tossing head bits magically fix it?
jan kili (Dec 13 2024 at 21:07):I infer that this approach would be a PR to RustGlue.roc to fix the representations it chooses.
Brendan Hansknecht (Dec 13 2024 at 21:10):For this specific case: https://godbolt.org/z/xj87ajszh
Brendan Hansknecht (Dec 13 2024 at 21:11):If all of your structs/tags were big enough to avoid fitting in two register, it would probably generate correct calls that match what rust expects.
Brendan Hansknecht (Dec 13 2024 at 21:12):Or would ballooning to U32/64 minimum sizes and then tossing head bits magically fix it?
So yeah, this would probably would in most cases.
jan kili (Dec 13 2024 at 21:16):Ballooning results:
U8s
🦅 happy: {bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66}
🦀 sad: ZeroedStruct { bar: 22, baz: 0, foo: 0, qux1: 0, qux2: 0, qux3: 0 }
U16s
🦅 happy: {bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66}
🦀 sad: ZeroedStruct { bar: 22, baz: 1951, foo: 32766, qux1: 0, qux2: 33, qux3: 0 }
U32s
🦅 happy: {bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66}
🦀 sad: ZeroedStruct { bar: 66, baz: 32766, foo: 3966136302, qux1: 22066, qux2: 1945484424, qux3: 32766 }
U64s
🦅 happy: {bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66}
🦀 sad: ZeroedStruct { bar: 140724510355272, baz: 94609053062050, foo: 140724510355320, qux1: 8, qux2: 139842509156127, qux3: 94609075901416 }
:blushing:
Brendan Hansknecht (Dec 13 2024 at 21:19):ballooned on both the roc and rust side?
jan kili (Dec 13 2024 at 21:21):yeah both, via naive regenerated glue
Screenshot_20241213_141958.png
Screenshot_20241213_142026.png
Screenshot_20241213_142045.png
Luke Boswell (Dec 13 2024 at 21:25):My "workaround" is to make the type larger by adding in U64's or something until it gets passed by pointer, and then just ignore the unused parts.
jan kili (Dec 13 2024 at 21:25):Should I add a pointer dereference to this sextet of U64s? (And is 6 "enough" to trigger that passing by pointer?)
Brendan Hansknecht (Dec 13 2024 at 21:26):Luke Boswell said:
My "workaround" is to make the type larger by adding in U64's or something until it gets passed by pointer, and then just ignore the unused parts.
That's probably an easier general workaround
jan kili (Dec 13 2024 at 21:27):Sneaky pointer confirmed: I wonder if it's already passing by pointer:
Re-running U64s
🦅 happy: {bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66}
🦀 sad: ZeroedStruct { bar: 140732166601912, baz: 93831138048418, foo: 140732166601960, qux1: 8, qux2: 140691827044127, qux3: 93831156052968 }
and again
🦅 happy: {bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66}
🦀 sad: ZeroedStruct { bar: 140721497110504, baz: 94174084974690, foo: 140721497110552, qux1: 8, qux2: 139823797833503, qux3: 94174114378728 }
or just unwiped ram noise, I suppose
jan kili (Dec 13 2024 at 21:30):Does this mean that glue code always requires nontrivial handwriting after generation? (when numbers are involved)
jan kili (Dec 13 2024 at 21:32):Luke, do you have an existing example of that pointer+ignore workaround that I can adapt? Like from roc-ray?
Luke Boswell (Dec 13 2024 at 21:35):Sure, here's how I get UUID's across to the host in roc-ray
RawUUID : {
upper : U64,
lower : U64,
zzz1 : U64,
zzz2 : U64,
zzz3 : U64,
}
And the glue for that
#[derive(Clone, Copy, Default, Debug, PartialEq, PartialOrd, Eq, Ord, Hash)]
#[repr(C)]
pub struct PeerUUID {
pub lower: u64,
pub upper: u64,
pub zzz1: u64,
pub zzz2: u64,
pub zzz3: u64,
}
roc_refcounted_noop_impl!(PeerUUID);
impl From<matchbox_socket::PeerId> for PeerUUID {
fn from(peer_id: matchbox_socket::PeerId) -> PeerUUID {
let (upper, lower) = peer_id.0.as_u64_pair();
PeerUUID {
lower,
upper,
zzz1: 0,
zzz2: 0,
zzz3: 0,
}
}
}
impl From<&PeerUUID> for uuid::Uuid {
fn from(roc_uuid: &PeerUUID) -> uuid::Uuid {
uuid::Uuid::from_u64_pair(roc_uuid.upper, roc_uuid.lower)
}
}
impl From<&PeerUUID> for PeerId {
fn from(roc_uuid: &PeerUUID) -> matchbox_socket::PeerId {
let uuid = uuid::Uuid::from_u64_pair(roc_uuid.upper, roc_uuid.lower);
matchbox_socket::PeerId::from(uuid)
}
}
Gotcha, so stuff in fluff then ignore it as much as possible!
jan kili (Dec 13 2024 at 21:40):What are the implications/ripples of passing by pointer? Would my
pub extern "C" fn roc_fx_sendZeroedStruct(sad: glue2::ZeroedStruct) -> RocResult<(), RocStr> {
change to
pub extern "C" fn roc_fx_sendZeroedStruct(sad: &glue2::ZeroedStruct) -> RocResult<(), RocStr> {
?
Luke Boswell (Dec 13 2024 at 21:40):Yeah, I'm hoping with the ROAR stuff we've been discussing, that would clean all the C ABI issues up.
jan kili (Dec 13 2024 at 21:40):...otherwise I don't see why the ballooning above wouldn't be working already.
Luke Boswell (Dec 13 2024 at 21:41):JanCVanB said:
What are the implications/ripples of passing by pointer? Would my
pub extern "C" fn roc_fx_sendZeroedStruct(sad: glue2::ZeroedStruct) -> RocResult<(), RocStr> {change to
pub extern "C" fn roc_fx_sendZeroedStruct(sad: &glue2::ZeroedStruct) -> RocResult<(), RocStr> {?
Yeah I think so.. here's an example getting PeerUUID from roc
extern "C" fn roc_fx_sendToPeer(bytes: &RocList<u8>, peer: &glue::PeerUUID) {
Just adding the &:
🦅 happy: {bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66}
🦀 sad: ZeroedStruct { bar: 22, baz: 33, foo: 11, qux1: 44, qux2: 55, qux3: 66 }
:woman_facepalming: I should be dancing right now but I'm not.
Luke Boswell (Dec 13 2024 at 21:43):Congratulations, you've basically found all the known issues with platform development :smiley:
jan kili (Dec 13 2024 at 21:43):This is fantastic news. For now I don't care that my programs will consume 100x memory.
jan kili (Dec 13 2024 at 21:44)::point_up: Best case scenario is that I'm quoting this comment in a month.
jan kili (Dec 13 2024 at 21:48):Okay, now I'm dancing: 3 U8s + 4 U64s triggers passing by pointer!
🦅 happy: {bar: 22, baz: 33, fluff1: 0, fluff2: 0, fluff3: 0, fluff4: 0, foo: 11}
🦀 sad: ZeroedStruct { fluff1: 0, fluff2: 0, fluff3: 0, fluff4: 0, bar: 22, baz: 33, foo: 11 }
THANK YOU ALL
jan kili (Dec 13 2024 at 21:51):Hmm, now how to apply this logic to tag unions, which currently generate two-field structs (payload+discriminant)...
jan kili (Dec 13 2024 at 21:52):I'll try fluffing every tag payload
Brendan Hansknecht (Dec 13 2024 at 21:53):Yeah, I'm hoping with the ROAR stuff we've been discussing, that would clean all the C ABI issues up.
they are kinda unrelated
Brendan Hansknecht (Dec 13 2024 at 21:53):ROAR is for the dev backend
Brendan Hansknecht (Dec 13 2024 at 21:53):llvm backend is failing to follow c abi in these cases
jan kili (Dec 13 2024 at 22:00):JanCVanB said:
LET'S
GOOOOOOOOOOOOOOOOOO
🦅😶 main is sending custom MIDI: {format: SingleTrack, timing: (Metrical 100 0 0 0 0)} 🦀😶 roc_fx_echoMidiSmallish received custom MIDI: Header { timing: Timing::Metrical(Timing_Metrical { f1: 0, f2: 0, f3: 0, f4: 0, f0: 100 }), format: Format::SingleTrack } 🦀😊 roc_fx_echoMidiSmallish casted that to Midly: Header { format: SingleTrack, timing: Metrical(u15(100)) } 🦀😊 roc_fx_echoMidiSmallish casted that back too: Header { timing: Timing::Metrical(Timing_Metrical { f1: 0, f2: 0, f3: 0, f4: 0, f0: 100 }), format: Format::SingleTrack } 🦅😶 main received an echo back: {format: SingleTrack, timing: (Metrical 100 0 0 0 0)} 🦅😊 main sees no difference!
jan kili (Dec 13 2024 at 22:05):
jan kili (Dec 13 2024 at 22:11):You've heard of whitespace - now introducing whxtxspxcx:
# This is to help avoid hitting a known C ABI bug
# by ensuring small values are passed by pointer.
# See https://roc.zulipchat.com/#narrow/channel/463736-bugs/topic/Struct.20memory.20partially.20zeroed.20between.20Roc.20and.20Rust/near/488928494
Fluff : U64
X : Fluff
FpsTiming : [Fps24 X X X X, Fps25 X X X X, Fps29 X X X X, Fps30 X X X X]
Format : [SingleTrack X X X X, Parallel X X X X, Sequential X X X X]
Timing : [Metrical U16 X X X X, Timecode FpsTiming U8 X X X X]
Header : {
format : Format,
timing : Timing,
}
JanCVanB has marked this topic as resolved.
Last updated: Jul 06 2025 at 12:14 UTC