Niklas Konstenius (Oct 10 2024 at 14:27): Niklas Konstenius (Oct 10 2024 at 14:27):What is the Roc idiomatic way of encoding Dicts to json?
I'm trying to solve the rest-api excercise on Exercism and my first try was to just use the Encoding ability of the platform to encode a Dict as JSON.
Getting this error though:
I can't generate an implementation of the Encoding ability for
List User
In particular, an implementation for
Dict Str F64
cannot be generated.
My next idea was to use some json encoding primitives like in Elm but this does not seem to be an idiomatic way to do it in Roc? I could not find any Elm like encoding functions anyway.
Is string concatenation really my only option to build json objects with dynamic field names in Roc right now?
Brendan Hansknecht (Oct 10 2024 at 15:40):Encode and Decode before Dict and Set where really proper types. They need to be expanded
Brendan Hansknecht (Oct 10 2024 at 15:42):We actually have a proposal and partial implementation of a much more flexible implementation. Sadly, I think it is currently stuck in some lambaset bugs.
Brendan Hansknecht (Oct 10 2024 at 15:42):That said, the current version of encode and decode definitely could be expanded to support dict and set directly.
Brendan Hansknecht (Oct 10 2024 at 15:45):New encode and decode idea discussed in #ideas > Revamped Encode and Decode.
I should probably circle back to the partial implementation to see if I can push it farther
Niklas Konstenius (Oct 10 2024 at 16:05):Thanks for the link to the Revamped Encode and Decode discusson. Interesting read indeed, hope you will manage to finish it.
Do you know if there is any other (possibly Elm style) Roc json-libraries in the wild? I really like the simplicity of Elms approach to json encoding / decoding and wouldn't mind using the same approach in Roc.
The Roc Encode / Decode stuff is still a bit magical to me. :-)
Brendan Hansknecht (Oct 10 2024 at 16:19):I don't know if any other library for json
Richard Feldman (Oct 10 2024 at 16:22):it would definitely be possible to port elm-json-decode-pipeline to Roc!
Richard Feldman (Oct 10 2024 at 16:22):you could even use record builder syntax to make it even nicer
Niklas Konstenius (Oct 10 2024 at 17:29):Cool, I need to read up on that record builder syntax now. :-)
Sam Mohr (Oct 10 2024 at 18:19):That one is in the tutorial
Sam Mohr (Oct 10 2024 at 18:19):Feel free to ask anyone about it, but I in particular have good experience building interfaces with them
Agus Zubiaga (Oct 11 2024 at 03:45):I think this would be very useful in code generated from an API contract. Like type-safe GraphQL or OpenAPI clients. You don’t want those to use the derived Encode/Decode abilities.
Brendan Hansknecht (Oct 11 2024 at 05:30):Couldn't GraphQL just generate the correct records and then use decode and encode?
Niklas Konstenius (Oct 11 2024 at 06:32):From my perspective I don't really think that decoding JSON => Record is very useful except for very simple cases. Optimizing for simple decoding of json into record structures only means (if I understand it correctly) that I need to have a "simple" record typ that represents the structure of the json document 1-to-1 and then parse this json-record into my domain types. For this more advanced case its just much simpler to write a elm-style decoder from json directly to my domain model imho.
There could of course be something I'm missing in this Encode/Decode architecture. I must confess that I don't have all the concepts totally clear in my head right now. :)
Nathan Kramer (Oct 11 2024 at 06:56):I personally struggled a bit with elm JSON decoders in terms of the learning curve.
Definitely powerful but a bit of a mind bending experience for me.
Niklas Konstenius (Oct 11 2024 at 07:22):I struggled with json decoders in Elm too. :-)
I do think that json => record is very good for beginners and simple usecases. But the moment you want something that is not easily represented 1-to-1 as a record it starts to get more complicated to not use elm style decoders imho.
A good example is sum-types and how the "tag" are represented in the json structure.
Agus Zubiaga (Oct 11 2024 at 11:00):Yeah, exactly. I think this applies primarily to clients, servers are more likely to match because they own the structure.
Agus Zubiaga (Oct 11 2024 at 11:07):When you’re building a client, it’s quite common that (even if it’s within the same company) the structure defined in the API is inconvenient for your use case. Things might be too nested, too general, have confusing names, or abuse primitives.
Agus Zubiaga (Oct 11 2024 at 11:11):Withdillonkearns/elm-graphql, I can deal with all of that complexity at the border right where I define the query selection.
Agus Zubiaga (Oct 11 2024 at 11:17):APIs become messy over time as they evolve, but a client is often relying on a subset or a specific point in time where certain properties hold true. This is the mismatch that warrants this approach.
Agus Zubiaga (Oct 11 2024 at 11:21):Another thing this applies to is database schemas, which is why I designed the roc-pg’s selection builder after elm-graphql’s. You can rename columns, map them, or even change the structure at the query.
Agus Zubiaga (Oct 11 2024 at 11:33):All that said, that applies at a certain level of complexity and depth. There’s so many clients that only need to work with a few fields in mostly flat structures, and the derived abilities would work great :grinning:
Brendan Hansknecht (Oct 11 2024 at 15:20):I'd be curious which of these critiques apply to serde from rust. It is used very comprehensively in the rust ecosystem. I think people rarely reach for something else. (Graphql maybe being special due to flexible queries). But for most things, I think it full covers everything. So encode and decode should theoretically as well.
Isaac Van Doren (Oct 11 2024 at 16:13):The jackson library is very popular in Java and it is similar in that your json is serialized and deserialized based on the fields on your objects so usually the structure of your json and objects is very closely related. That being said, there is a lot of customization you can do that makes it more flexible. The kind I end up using most is field level customizations like specifying how a date should be formatted.
Niklas Konstenius (Oct 11 2024 at 16:56):I would argue that the way json serialization are done in Java (with for example the Jackson library) more has to do with tradition and Javas background in OOP than it has to do with being a "good" way. :smile:
I have seen (and produced myself) lots of Java code with tons and tons of annotations polluting the carefully crafted domain objects just to configure how the objects properties should serialize to json. The rationale for all this noise is that its "just configuration" and somehow configuration is always better than code...
Imho, simple Elm-style transformation functions that maps the domain objects to it's json representation often is way more compact, more flexible and easier to read than all the annotation mess and I really wish these functional approaches would find its way into Java land. Tradition is hard to change though...
Since Roc aims to be a functional language I really hope Roc will take the more functional approach, or at least have good support for both approaches.
Brendan Hansknecht (Oct 11 2024 at 17:29):I'm sure both approaches can work and will be supported. I think with decode instead of annotations it would be specified in the types. Occasionally with custom abilities impls.
Like you might have a Nullable I64 json field that allows for the field to be missing.
I'm mostly just curious where decode will fall short and why. Is it personal preference or something more fundamental. I probably should look more into where serde falls short cause decode is based on serde.
Richard Feldman (Oct 11 2024 at 17:56):I don't think either approach is more or less functional than the other :big_smile:
Richard Feldman (Oct 11 2024 at 17:57):one is more flexible but also more repetitive in common cases
Richard Feldman (Oct 11 2024 at 17:57):that's the main tradeoff I see in terms of usage
Richard Feldman (Oct 11 2024 at 17:57):there's also a major learning curve gap between them for common cases though
Niklas Konstenius (Oct 11 2024 at 18:20):I'm mostly just curious where decode will fall short and why. Is it personal preference or something more fundamental.
For me it's a personal preference of wanting to have a clear separation of the domain model and it's representation as json. That's why I don't usually find generic json => record mappings (and the other way around) that useful. In Java that means I often end up with a set of simple classes representing the wanted json-structure (which is serialized/deserialized using jackson or similar) and then a set of transformation functions to and from the domain-model.
Since the json-model is just something thats needed by Jackson its just boilerplate and I would rather use an Elm style decoder/encoder pattern to do the mapping to/from json directly. For me this is a way more ergonomic / flexible pattern than the traditional style using annotations and reflection through jackson.
But I do understand that for simple usecases and beginners this may be a big hurdle.
Niklas Konstenius (Oct 11 2024 at 19:14):Like you might have a
Nullable I64json field that allows for the field to be missing.
This is actually a good example of what I DON'T want : smile:
I do not want my domain types be polluted by stuff that only makes sense in the context of how it´s represented as json.
Brendan Hansknecht (Oct 11 2024 at 20:16):Fair enough. I think the standard for more complex inputs and something like serde is to represent the input exactly, map it into the happy land of a typed language with all the worts like Nullable (this guarantees it is fully represented and subsequent logic handles all cases), map it to a nice struct for domain use and without worts.
Brendan Hansknecht (Oct 11 2024 at 20:17):So you still have a domain type and logic. It is just split into two parts.
Niklas Konstenius (Oct 12 2024 at 16:24):I've been thinking a bit and realised that the only thing I really need is a way to Decode / Encode json-text into a general ADT like this:
Json : [
True,
False,
Null,
Number F64,
String Str,
Array (List Json),
Object (List (Str, Json)),
]
Do you think this is something that could be achieved using the Roc Decode/Encode functions and the existing Json library? Or maybe something as general as this is out of the scope for the Decode/Encode feature?
Brendan Hansknecht (Oct 12 2024 at 17:00):With a custom implementation of the ability, that should be doable. Then you could encode/decode any json.
Brendan Hansknecht (Oct 12 2024 at 17:02):That said, may be faster in many cases to lazily decode json. That way you only load the parts of the message you care about. That would not be possible with encode/decode.
Niklas Konstenius (Oct 12 2024 at 17:11):Not sure about what you mean by lazy but being able to just extract specific parts of a json-document is really valuable and something I do a lot. I have not realised that this could be a problem with Decode though.
Do you mean that the record structure must match the json structure exactly (like all fields must be included) to be able to decode it?
Brendan Hansknecht (Oct 12 2024 at 17:14):What I mean is Decode will parse the entire json document into a roc record. If the document is 100MB and you want to access 1 specific field, that is a huge waste of time parsing the entire document.
Brendan Hansknecht (Oct 12 2024 at 17:15):A lazy parser could do the minimal work to grab the one field. Then if you need a second field, it would again do minimal work to grab the second field.
Really depends on the use case though. If you are always gonna access most fields, this won't matter.
Brendan Hansknecht (Oct 12 2024 at 17:17):In tag terms, it might represent json as:
Json : [
True,
False,
Null,
Number F64,
String Str,
Array (List Json),
Object (List (Str, Json)),
Lazy ({} -> Json),
]
This way, it can stop parsing part way instead of parsing everything.
Niklas Konstenius (Oct 12 2024 at 17:20):I'm not an expert on parsing but I'm not sure how one could parse a JSON file without reading the complete document? I mean, if just the last closing curly bracket is missing the document is invalid.
Brendan Hansknecht (Oct 12 2024 at 17:23):2 main options:
For example, if I want to get foo.bar, I can still do so from this:
{
"baz": []
"foo": {
"something": "abc",
"bar": 42,
"other": null,
Still not sure how one could do 2 without proper parsing but as I said, I'm not an parsing expert. :smile:
To me it seems like there is an opportunity to fill a missing space in the Roc ecosystem by writing a generic JSON parser library ( json text => tag union ADT) that can be used as a building block for other libraries like for example Elm style json-decoder / encoder.
I'm tempted to have a go at it to be honest. Would be a great way to learn more about Roc as a programming language too.
Brendan Hansknecht (Oct 12 2024 at 17:50):by writing a generic JSON parser library
That could definitely be useful for when decode isn't enough. That said, Decode should be able to generate the tag you shared above.
Brendan Hansknecht (Oct 12 2024 at 17:55):Still not sure how one could do 2 without proper parsing but as I said
You have to be delaying some of the errors if you want max perf. So it is a fundamental trade off. That said, validating an entire json has correct syntax can still be faster than actual parsing the json into a nested tag
Niklas Konstenius (Oct 12 2024 at 18:00):That said, Decode should be able to generate the tag you shared above.
Cool, I'll have a look. It it is not obvious for me where to start though.
Is it the DecoderFormatting interface that i should implement to make this work?
Brendan Hansknecht (Oct 12 2024 at 18:13):You would implement Decoding then using the existing roc json decoder in combination. It would provide ways to decode all of the primitives. You need to map them to the the generic tag structure.
Niklas Konstenius (Oct 12 2024 at 18:52):Ok, I think I understand how one could use the DecoderFormatter from roc -json to parse the primitives even though it feels a little bit backwards to use parsers named after Roc primitives to parse json elements. But thats just a consequence of Encode/Decode being an generalised abstraction of serialising Roc types I guess.
Brendan Hansknecht (Oct 12 2024 at 19:01):Yeah...
Brendan Hansknecht (Oct 12 2024 at 19:02):You also can just write a direct parser
Brendan Hansknecht (Oct 12 2024 at 19:02):Might be more reasonable with your goals
Niklas Konstenius (Oct 12 2024 at 19:09):Yes I think I'll aim for the simplest possible parser from text => ADT that can be used to build higher level features on as a proof of concept.
I read in another thread something about implementing simdjson as a very fast built in support for JSON in the language itself.
If that feature could be used to provide the ADT a parser built with Roc would be obsolete pretty fast :-)
Luke Boswell (Oct 12 2024 at 19:31):I think @Monica has made one of these for a talk she gave recently on Roc.
Luke Boswell (Oct 12 2024 at 19:32):I would like to include decoding into a generic ADT JsonValue into roc-json too, I think it will be helpful.
Luke Boswell (Oct 12 2024 at 19:35):https://github.com/roc-lang/book-of-examples/pull/68
Niklas Konstenius (Oct 13 2024 at 07:25):Cool, Monica seems to like elm style decoding/encoding as much as I do. :smile:
As far as I could see Monicas example left out the step of parsing json text to the the Json ADT (or tag union which I understand is the correct term in Roc), right?
I think that makes the case for a library (or a bultin way) of generic json parsing to ADT in Roc even stronger. Anyone that reads that example and thinks, cool I want to use that technique in may app, will otherwise inevitably be forced to first implement his own json parser. That would be really confusing for a beginner I think...
Niklas Konstenius (Oct 13 2024 at 07:35):Another case thats a confusing for a Roc beginner like me is the RestApi exercise on Exercism. You can't solve it using Roc's bultin Encode/Decode but instead have to resort to string concatenation hacks just to serialise a json object. This certainly is not something that makes Roc shine compared to other languages. I was really disappointed myself by this (and the reason why I started this thread in the first place).
Niklas Konstenius (Oct 13 2024 at 07:43):But I should stop whining now and instead start implementing my json parser :smile:
Brendan Hansknecht (Oct 13 2024 at 17:40):You can't solve it using Roc's bultin Encode/Decode
Is this only due to dictionaries not encoding/decoding or are there other issues for that exercise specifically?
Niklas Konstenius (Oct 13 2024 at 19:38):No, I think it's only the Dict encoding that's missing.
Anton (Oct 14 2024 at 09:03):Another case thats a confusing for a Roc beginner like me is the RestApi exercise on Exercism. You can't solve it using Roc's bultin Encode/Decode but instead have to resort to string concatenation hacks just to serialise a json object. This certainly is not something that makes Roc shine compared to other languages. I was really disappointed myself by this (and the reason why I started this thread in the first place).
Perhaps we should take out this exercise until dict encoding/decoding is fixed @Aurélien Geron?
Monica (Oct 14 2024 at 15:17):Niklas Konstenius said:
Cool, Monica seems to like elm style decoding/encoding as much as I do. :smile:
As far as I could see Monicas example left out the step of parsing json text to the the Json ADT (or tag union which I understand is the correct term in Roc), right?
I think that makes the case for a library (or a bultin way) of generic json parsing to ADT in Roc even stronger. Anyone that reads that example and thinks, cool I want to use that technique in may app, will otherwise inevitably be forced to first implement his own json parser. That would be really confusing for a beginner I think...
Sorry bit late to the party - yeh my example is strictly Json decoding from a tag union/ADT representation into a domain model, along with some helper functions to abstract away the error handling bits.
Aurélien Geron (Oct 14 2024 at 20:21):Anton said:
Perhaps we should take out this exercise until dict encoding/decoding is fixed Aurélien Geron?
The platform does not allow a track to delete an exercise, but we can mark it as deprecated: it will make no change to anyone who has already started the exercise, but it will hide it from everyone else.
I see two other options:
.docs/instructions.append.md, so users know what to expect and they can skip the exercise if they want to.RestApi.rocI think I would prefer alternative 1, but it's not a strong opinion, I'm happy to go any direction here.
Richard Feldman (Oct 14 2024 at 21:21):we could also prioritize fixing that :grinning_face_with_smiling_eyes:
Luke Boswell (Oct 14 2024 at 21:23):Yeah, it would be really nice to have Dict supported. I thought someone worked on it for a bit, though I've not been able to find the old PR or branch (or remember who). I thought it was @Isaac Van Doren??
Luke Boswell (Oct 14 2024 at 21:24):Is it a big job to do? I don't have much idea in that part of the compiler
Isaac Van Doren (Oct 14 2024 at 21:36):I took a stab at decoding tag unions not Dict, but I agree that fixing it sounds like a good approach :smiley:
Oskar Hahn (Oct 15 2024 at 07:58):Decoding Tag Unions is one of my most wanted issues. If you could solve it in the same run, I would be so thankful.:pleading_face::kissing_cat:
Niklas Konstenius (Oct 15 2024 at 08:38):I took a stab at decoding tag unions...
Cool. I'm curious in how the design of decoding tag unions would work from a user perspective?
What kind of structures would be possible to decode? I guess it has to be the "inverse" of the tag function in EncoderFormatting in some way?
Anton (Oct 15 2024 at 10:22):they can skip the exercise if they want to.
Can exercises be skipped? I know there is some kind of unlocking of exercises mechanism
Oskar Hahn (Oct 15 2024 at 10:30):Yes. For roc (at the Moment) no exercise is locked. You can start with any of them.
Isaac Van Doren (Oct 15 2024 at 12:26):Cool. I'm curious in how the design of decoding tag unions would work from a user perspective?
What kind of structures would be possible to decode? I guess it has to be the "inverse" of thetagfunction inEncoderFormattingin some way?
It would be up to the DecoderFormatting implementation (such as roc-json) to decide how to handle the decoding. I think the way that makes the most sense as a start would be to decode the following json into the following roc:
[{ "a": { "field1": "foo" } }, { "b": { "field2": "bar" } }]
[A { field1: "foo" }, B { field2: "bar" }]
But there could be other approaches. Special casing for null would probably be a good thing to include also.
Last updated: Jul 05 2025 at 12:14 UTC