Stream: beginners

Topic: Roc Serialization

Ryan Barth (Nov 05 2024 at 01:37):

Hey folks! I have been working on a Roc implementation of Postcard, a binary serialization protocol. I am about 90% of the way there, but I have hit a wall due to some combination of limitations with Roc, the fact the protocol is non-self-describing, or my own ignorance.

The few issues I have hit so far are:

• No ability to get a list of variants of a Tag, or a discriminant
• No ability to get a list of fields on a record, or their definition order

I looked at the builtin record decoder. This looks like something that requires the compiler internals and it would not be possible to create from within Roc, preventing an in-roc alternate decoding framework / ability.

Here is the repo with the progress I have made so far. I would love if anyone could suggest a way forward.

Considered alternatives:

• Hashing field names when encoding so the wire format is a tuple of alternating key hashes and field values. This would deviate from the spec (further than I already have, see the note on variants) and make the representation less compact.
• Code gen with an external program?

Brendan Hansknecht (Nov 05 2024 at 16:42):

Probably needs the work discussed at the top of this thread: https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/Revamped.20Encode.20and.20Decode

Brendan Hansknecht (Nov 05 2024 at 16:42):

Encode and decode are too limited currently

Brendan Hansknecht (Nov 05 2024 at 16:42):

Sadly, that work is blocked by a compiler bug last I checked

Brendan Hansknecht (Nov 05 2024 at 16:44):

Though maybe you can do most of what you want with the current version of decode: https://github.com/roc-lang/roc/blob/main/crates/compiler/builtins/roc/Decode.roc

Brendan Hansknecht (Nov 05 2024 at 16:45):

But I think current decode may require self descriptive formats in many cases

Ryan Barth (Nov 05 2024 at 17:15):

Thanks for the reply @Brendan Hansknecht ! That first link had exactly what I would need to solve the variant problem. Is the type erasure stuff what you meant by the compiler bug that was blocking that work?

But I think current decode may require self descriptive formats in many cases

That was what I was afraid of. From what I gleaned from the json implementation it seemed like you needed the parser to drive the decoder. That is you need the name of the fields, then call stepField state fieldName. Do you know if there is an issue I can reference for any planned changes for the record decoding specifically?

I also think given the current limitations I am better off just using your MsgPack package for what I need :smile: . I am bummed I couldn't finish my first roc package.

Brendan Hansknecht (Nov 05 2024 at 17:22):

The msgpack package is where I was testing out the new encoder and decoder abilities

Brendan Hansknecht (Nov 05 2024 at 17:22):

So it is not actually functional.

Brendan Hansknecht (Nov 05 2024 at 17:23):

That said, msgpack could be made to work with current encode and decode

Brendan Hansknecht (Nov 05 2024 at 17:24):

As for records, I was messing around with the interface in the msgpack repo. I don't think it has a solo bug, just something I wanted to fold into the new encode and decode

Brendan Hansknecht (Nov 05 2024 at 17:25):

The bug was an issue with lambdaset unification if I recall correctly. Essentially, the compiler getting confused by the recursion and captures that are generated by the ability.

Last updated: Jul 05 2025 at 12:14 UTC