Stream: ideas

Topic: CBOR serialization

notramo (Sep 25 2026 at 14:51):

The builtin parser_for/encode_for is seriously limited. It's basically only usable for JSON.
For example, there's no way to configure whether a List(U8) should be emitted as bstr (bytes blob) or [* uint] (list of U8 — the * means in CDDL that it can have an arbitrary length). https://github.com/roc-lang/roc/issues/11660

notramo (Sep 25 2026 at 14:52):

See the issue for more details on the problem.

notramo (Sep 25 2026 at 14:54):

This can be worked around by decoding the CBOR into an AST, then manually filling in the struct fields.

notramo (Sep 25 2026 at 14:54):

But this wastes memory, because the AST is stored separately, instead of directly decoding data into the struct fields.

Jonathan (Sep 25 2026 at 15:46):

Still trying to get my head round the parser api myself, so - partially for my own understanding - couldn't this be implemented with a wrapper type for the byte blob around List(U8)? The wrapper type would have to have a custom parser_for and encoder_for, which would expect from the 'generic' format some Encoding.parse_blob and Encoding.encode_blob respectively, which return and take List(U8). In practice the format wouldn't be generic because you would use it exclusively with your Cbor format, in which you would implement encode_blob/parse_blob.

notramo (Sep 25 2026 at 15:54):

This solution would require unpacking it everywhere in the program. An ideal serialization library would not require modifying the program everywhere in order to configure the serialized output.

Jonathan (Sep 25 2026 at 15:57):

Yeh I get that for sure. I mean, the unpacking would be free computationally, if not ergonomically?

notramo (Sep 25 2026 at 16:02):

Yes, but it's really a bad idea to write my program code differently because some serialization library needs it that way.

If this is the case, even a manual serialization is better, since that's concentrating the added complexity at the parse/encode function, and not creeping into the whole program.

Jonathan (Sep 25 2026 at 16:05):

Does it creep into the whole program though? I imagine ser/de typically being at the shell of your code, and only at that periphery do you need to convert your specific List(U8) into Bytes so that the Cbor format can dispatch to the right encoder/decoder

notramo (Sep 25 2026 at 16:10):

It's possible, but then I would need a different struct type, with the only difference being the given field.

Jonathan (Sep 25 2026 at 16:13):

Yeh that's what I arrived at too. It would be more ergonomic if there was a way to update records but also change the type like : record_for_cbor = {..internal, data: internal.data |> Blob.from_list()}. For more nested records it would be less pleasant.

Steve Howell (Sep 25 2026 at 16:24):

Is https://www.roc-lang.org/examples/EncodeDecode/README the relevant doc here?

I'm kinda curious how you would use CBOR data in a Roc program. If you know the data types in advance, then I think you could use nominal types like the README says? If you don't know the data types in advance, then I don't know what advantages you get from using Roc.

Jonathan (Sep 25 2026 at 16:29):

Also https://github.com/roc-lang/roc/blob/main/docs/langref/static-dispatch.md#parsing-and-encoding

notramo (Sep 25 2026 at 16:34):

I'm kinda curious how you would use CBOR data in a Roc program. If you know the data types in advance, then I think you could use nominal types like the README says? If you don't know the data types in advance, then I don't know what advantages you get from using Roc.

MyType := {
  blob_field: List(U8) # This stores a blob, e.g. a binary file.
  list_of_ints: List(U8) # This stores actual numbers.
}

How would you achieve that blob_field gets serialized as bstr, while list_of_ints gets serialized as [* uint]?

Steve Howell (Sep 25 2026 at 16:39):

notramo said:

How would you achieve that blob_field gets serialized as bstr, while list_of_ints gets serialized as [* uint]?

I think opaque types are one option.

Blob := { bytes : List U8 }

notramo (Sep 25 2026 at 16:48):

That's what I want to avoid. Why change my entire program just to configure an encoder?

Steve Howell (Sep 25 2026 at 16:54):

notramo said:

That's what I want to avoid. Why change my entire program just to configure an encoder?

I don't know your use case or your programming style well enough to answer that question. But I think you'd be wise to just model Blobs as Blobs with or without encoding considerations. Presumably you're doing something useful with the Blob, so even Blob is kinda low-level. I don't understand why you would avoid nominal types in a language where that's a major selling point.

notramo (Sep 25 2026 at 16:55):

The problem is that a Blob in a CBOR library would not be the same as a Blob in another library, despite the same name. And there's a reason why Roc don't have a builtin Blob.

Steve Howell (Sep 25 2026 at 16:58):

When you say "another library" what kind of library are you referring to? A Roc library?

notramo (Sep 25 2026 at 16:59):

Yes.

Steve Howell (Sep 25 2026 at 17:07):

I still don't understand your difficulty. If your other library needs the Blob represented as U64s or whatever, write a function to convert your representation before calling the library. I must be missing something here.

notramo (Sep 25 2026 at 17:10):

Encapsulation.

The problem is that this bstr problem is just one thing. CBOR is complex, because it can represent every data layout and type that is needed most of the time. This is just one thing out of many. I don't want to restructure my types and program logic just because a serialization library can't keep up.

I would rather write manual serialization, which can be encapsulated, then write the rest of the program normally.

Jasper Woudenberg (Sep 26 2026 at 08:18):

I think you could do an Elm-style encoder/decoder library for CBOR. Then you have full control about how particular types are encoded and decoded.

notramo (Sep 26 2026 at 09:20):

I already said that:

I would rather write manual serialization, which can be encapsulated, then write the rest of the program normally.

But then why have encoder_for/parser_for if everything has to be manually serialized? And it's not just CBOR, but a bunch of other formats too.

Jasper Woudenberg (Sep 26 2026 at 10:15):

notramo said:

But then why have encoder_for/parser_for if everything has to be manually serialized?

As a convenience for other use cases where a default serialization is good enough.

notramo (Sep 26 2026 at 10:15):

So basically JSON only?

Jasper Woudenberg (Sep 26 2026 at 10:15):

Why would it be JSON only?

notramo (Sep 26 2026 at 10:16):

What other serialization format do you know that the builtin can properly handle?

notramo (Sep 26 2026 at 10:17):

Ziggy? No.
TOML? No.
YAML? No.
CBOR? No.
Gura? No.

notramo (Sep 26 2026 at 10:17):

JSON is the simplest one, and basically every serialization format has more features than JSON.

Jasper Woudenberg (Sep 26 2026 at 10:21):

Having a default encoding/decoding strategy for any of those formats is valuable even if the encoding doesn't fully exercise every feature of the serialization format. For many use cases that default will be good enough. Anyone is free to define custom encoders/decoders if they feel that for their particular use case the juice is worth the squeeze.

Steve Howell (Sep 26 2026 at 10:25):

I can understand why @notramo is a bit frustrated here. It should be as simple as adding one more field to the API where you can indicate the type. Is the default encoding/decoding strategy so simple that you don't want to taint it with any extra moving parts?

notramo (Sep 26 2026 at 10:27):

The only problem with it is that the encoder does not get enough info about the encoded data. I would add a config option to the encoder, for example, to encode e.g. User.public_key as a blob, but it does not know that it's currently encoding User.public_key.

Jasper Woudenberg (Sep 26 2026 at 10:28):

The current API allows encoders/decoders to be configurable. For instance: consider configuring the JSON encoder to tell it whether to produce pretty or compact JSON.

But if you have a custom datatype in which you want to encode some fields one way and another field of the same type a different way, the example @notramo brought at the start of this thread, then I think a custom encoder is your only option. Though maybe the custom encoder can call on the default encoder for parts of the data structure that do not require special handling.

Steve Howell (Sep 26 2026 at 11:41):

@notramo I had Claude take a stab at this. See what you think about https://github.com/showell/roc-apps/tree/master/cbor-pets (updated after further discussion)

notramo (Sep 26 2026 at 11:52):

Thanks. I like the simplicity of the parser/encoder, but how does it plug into the default serialization system?

notramo (Sep 26 2026 at 11:54):

An ideal API would be something like this:

profile : Profile
profile = Cbor.parse(serialized_bytes)

Steve Howell (Sep 26 2026 at 12:04):

It doesn't plug into the default serialization system, which I think is fine.

I don't understand how you would expect the code below to make sense in Roc. For Cbor.parse to make sense, it has to return a very generic type. The compiler should complain if you're assigning arbitrary data to a concrete type like Profile.

profile : Profile
profile = Cbor.parse(serialized_bytes)

notramo (Sep 26 2026 at 12:05):

The compiler should complain if you're assigning arbitrary data to a concrete type like Profile.

No, the default parsing mechanism derives the expected shape of the parseable data from the destination type.

This is how JSON parsing works in the stdlib too.

notramo (Sep 26 2026 at 12:06):

If the CBOR blob do not match the data structure, parse() returns an error that tells that it expected e.g. a string, but found an integer in the encoded CBOR.

Jonathan (Sep 26 2026 at 12:06):

I think with the caveat that it would be Cbor.parse(...)?, if that's the mixup Steve?

Steve Howell (Sep 26 2026 at 12:06):

notramo said:

The compiler should complain if you're assigning arbitrary data to a concrete type like Profile.

No, the default parsing mechanism derives the expected shape of the parseable data from the destination type.

This is how JSON parsing works in the stdlib too.

Do you have an example of that? That's where I wish the docs were a bit more clear.

notramo (Sep 26 2026 at 12:07):

Sorry, the result should be a Try:

profile : Try(Profile, _)
profile = Cbor.parse(serialized_bytes)

Jonathan (Sep 26 2026 at 12:07):

https://github.com/roc-lang/roc/blob/main/docs/langref/parsers.md

notramo (Sep 26 2026 at 12:08):

Also:
https://www.roc-lang.org/examples/Json/README

Steve Howell (Sep 26 2026 at 12:08):

Jonathan said:

I think with the caveat that it would be Cbor.parse(...)?, if that's the mixup Steve?

I don't understand your question. I'm asking how Roc type-checks profile = Cbor.parse(serialized_bytes) when you say in the line above that profile has to be the type of Profile.

notramo (Sep 26 2026 at 12:09):

Roc's type inference is two-way. (Hindley-Millner)

notramo (Sep 26 2026 at 12:09):

A function can basically ask "What type I'm expected to return?"

notramo (Sep 26 2026 at 12:11):

The type can be inferred from later uses of a variable.
For example:

# It has to be something that has a .name field, because later it's accessed.
my_var = ...

my_var.name

Jonathan (Sep 26 2026 at 12:11):

Steve Howell said:

Jonathan said:

I think with the caveat that it would be Cbor.parse(...)?, if that's the mixup Steve?

I don't understand your question. I'm asking how Roc type-checks profile = Cbor.parse(serialized_bytes) when you say in the line above that profile has to be the type of Profile.

Sorry Steve, that wasn't clear - I just meant that the parse returns a Try(type, err), hence my adding a ? to parse, which is where the parser can complain if your type doesnt match the expected shape.

notramo (Sep 26 2026 at 12:13):

I recommend pointing an LLM at the Roc repo and ask how the Json.parse uses the two-way type inference to extract the desired type from the return type.

notramo (Sep 26 2026 at 12:14):

It's a bit complex, but an LLM can explain it quite well.

Steve Howell (Sep 26 2026 at 12:25):

From Claude:

It also shows exactly where his problem sits: the type is the only instruction the format receives. A List(U8) field always arrives as parse_list_start followed by parse_u8 per element, and there is no parse_bytes hook. So pet_ages and a picture stored as plain List(U8) look identical to the format.

Your example is the exception, because the picture is already an ImageBlob. ImageBlob could define its own parser_for/encoder_for whose where clause asks the format for parse_bytes/encode_bytes; this is Jonathan's suggestion from the thread. JSON would then fail to compile for Profile, which is arguably correct, and CBOR would work. That case is covered; notramo's objection is specifically about two plain List(U8) fields.

Steve Howell (Sep 26 2026 at 12:45):

Ok, https://github.com/showell/roc-apps/blob/master/cbor-pets/ImageBlob.roc shows Claude's new approach.

notramo (Sep 26 2026 at 12:51):

Yes, this works, but it requires a custom type, ImageBlob in this case.

But it's very limited in flexibility.

Steve Howell (Sep 26 2026 at 12:54):

What use case can't it handle? I don't think requiring an ImageBlob class is a workaround here. Do you have some kind of use case where you're hurling blobs around without caring about their actual semantics?

notramo (Sep 26 2026 at 12:54):

How to serialize a List(U8) field in a type that is from a 3rd party package, so I can't extend it?

Steve Howell (Sep 26 2026 at 12:55):

Can you give a more concrete example? What if my Profile were in a 3rd party package? Are you claiming you couldn't use it then?

Steve Howell (Sep 26 2026 at 13:09):

FYI I opened #11727 and #11728 (from Claude) just doing this exercise.

notramo (Sep 26 2026 at 13:18):

Imagine if Profile were in a 3rd party package, and the image was a List(U8).

Steve Howell (Sep 26 2026 at 13:21):

I would avoid 3rd party packages that don't distinguish images from ordinary lists of integers. In fact, I don't love that my Profile class models pet ages as List U8 instead of adding a nominal Age type.

I know that sounds like a cop-out, but do you actually have a 3rd party package in mind?

Steve Howell (Sep 26 2026 at 13:22):

To be a little more pragmatic, if I have a third party package that does a lot of good stuff under the hood, it's almost always the case that I write some kind of edge layer anyway. So that's what I would do here.

notramo (Sep 26 2026 at 13:26):

adding a nominal Age type.

Yes, that's good practice.

I know that sounds like a cop-out, but do you actually have a 3rd party package in mind?

No specific library. I think, instead of searching for the perfect CBOR library, I'll try a few approaches, and see what's the most usable in practice.

Richard Feldman (Sep 26 2026 at 20:47):

so I did some cursory research which seems to confirm my intuition: in practice, CBOR APIs use bstr almost exclusively and "array of uint8" and "array of int8" are technically supported but basically never used in practice

Richard Feldman (Sep 26 2026 at 20:50):

assuming that's true, personally what I'd do is:

• for auto-derived serialization, always choose bstr - since the whole point of auto-deriving is to save you time in common cases, and this is the common case
• if someone needs the uncommon array of uint8 or int8, they can do explicit serialization instead

Richard Feldman (Sep 26 2026 at 21:14):

as an aside, I know that a lot of languages have "per-field annotations for serialization" to address this, and I'm really resistant to adding a feature like that to Roc.

in general, serialization formats have different types they support natively. Programming languages have their own types they support natively. So these are never going to be 1-to-1 in the general case; you're always going to have this problem where "the language has multiple ways to represent ____ but the serialization format only has one way to represent that, so how do you know which to parse it as?" or (like in this thread) "the language has only one way to represent _______ but the serialization format has multiple ways, so how do you know which to serialize it as?"

Richard Feldman (Sep 26 2026 at 21:14):

the "annotate fields" approach to solving this creates a combinatoric explosion of complexity

Richard Feldman (Sep 26 2026 at 21:20):

like suppose Roc had a way to annotate fields - there are at least 3 ways to address this innate problem using such a feature:

1. stringly-typed annotations (like what Go does): you annotate it with a string like "cbor-serialize: bstr" and then CBOR libraries know to look for that magic string, parse it, and use it to determine what strategy to use. This has the usual downsides of stringly-typed designs. :sweat_smile:
2. annotations specific to a serialization format (e.g. a specific CBOR implementation)
3. annotations specific to a serialization library and general serialization library (e.g. in Rust this would be cbor-serde or similar, where the combinatorics include both the CBOR package and the serde serialization library)

so if you want to solve this problem for N different serialization formats, you have to add at least N different annotations, and possibly more depending on which of these strategies you choose (e.g. two different Roc CBOR implementations would likely have separate annotation types, so you'd need to specify it separately for both of them)

Richard Feldman (Sep 26 2026 at 21:23):

the downside of what I'm recommending here, namely that for auto-derived serialization the CBOR package author picks defaults (give or take passing in global options like the "convert camel case fields" arg to Json.parse) is that it offers less granular customization compared to field-specific annotations, but of course if you really need granular customizations you can do that with something other than auto-derived ones

Richard Feldman (Sep 26 2026 at 21:28):

but the upsides are that it has neither stringly-typed footguns nor combinatorics problems (no matter how many different CBOR packages there are, you never have to annotate fields) and it also works with structural types as well as nominal types, whereas any annotation-based strategy can only possibly work with nominal types.

Richard Feldman (Sep 26 2026 at 21:30):

It also keeps the language simpler; it's not like you just "add field annotations" and you're done - there has to be a way to access the annotations, it has to integrate with third-party packages at the same time as type information from the annotation, it has to participate in auto-deriving, etc.

notramo (Sep 28 2026 at 19:08):

@Richard Feldman, thanks for the feedback.

Matthieu Pizenberg (Sep 30 2026 at 11:48):

Just to note that I was excited to see someone looking into this ^^. I’ve rewrote from scratch a CBOR codec library in elm in the past, and will likely be very interested in looking into cbor+roc in the future too :)
CBOR is a very cool format, because it’s binary, compact, and self-described (no need for schema ahead of time). But it’s a very annoying format to support right for multiple reasons. You can encode things in various ways, both with different types and with different shapes.

For example, you can encode an array either with a fixed length, given at the beginning of the array. Or with a dynamic length, where the decoder waits for a special byte marking the end of the array. The fixed length is much more convenient for full element decoding/encoding, but the dynamic length can be super convenient for streaming use cases for example.

I’ve personally interacted with protocols where every bit of encoding decision is important, both for types and shapes, and I believe CBOR is particularly well suited for these use cases due to its flexibility. So I wouldn’t trust an LLM summary saying one way is almost always used over other ways @Richard Feldman

Last updated: Oct 15 2026 at 17:56 UTC