Eli Dowling (Apr 19 2024 at 08:45): Eli Dowling (Apr 19 2024 at 08:45):I've hit what seems to be kind of a language level roadblock in my json decoding.
I need to be able to encode and decode from a type that represents "some json value". This is required to make a generic implementation of the lsp spec because a response is just {id:string, params:JsonValue} where we don't know the contents.
The rust equivalent of this would be serde_json::Value, in serde this is represented this by having the idea of an "encoder" and "decoder" be more generic than just value->bytes so Value implements Decoder so it can be decoded into any type we like, same with encoding.
In the dotnet and go worlds, this would be done with reflection. In rust, under the hood, it's done with deriving macros.
As far as I can tell, roc would need, reflection, macros, comptime, or a different decoding system to represent this quite fundamental json structure. I'd love to hear some thoughts on whether I've got this all wrong, or what solution we could explore.
Also, I do think this raises the question, are there any other languages that get away with not having any form of metaprogramming?
Eli Dowling (Apr 19 2024 at 08:48):Oh, I can technically do this in roc right now but it would involve parsing all the json, and just storing a copy of all the bytes for that section. Then doing the actual decoding when it's accessed. Technically it's a solution, but it's very odd and feels like a hack because it involves re-parsing the JSON multiple times
Richard Feldman (Apr 19 2024 at 11:16):Eli Dowling said:
Also, I do think this raises the question, are there any other languages that get away with not having any form of metaprogramming?
yeah, Elm does it by explicitly building up JSON decoders - which includes a generic "JSON Value" type
Richard Feldman (Apr 19 2024 at 11:16):so that API can definitely work for this use case without any re-parsing
Richard Feldman (Apr 19 2024 at 11:17):the downside is that you have to build up the Decoder value (a JSON parser combinator, essentially) by hand instead of using the Decoding ability to infer it from the Roc type
Richard Feldman (Apr 19 2024 at 11:17):I can't think of a way to do this using the Decoding ability, but maybe @Ayaz Hafiz could think of a way it could be possible?
witoldsz (Apr 19 2024 at 11:23):I can add to it that sometimes I have to decode a JSON message by:
witoldsz (Apr 19 2024 at 11:32):Also, sometimes the JSON looks like this:
{ "ts": "2024-04-19T13:24:34.331Z"
, "EUR/USD": 1.0649
, "EUR/PLN": 4.3273
, "USD/CZK": "data unavailable"
, ...other pairs, not known in advance... }
I had to decode many super weird formats like this with Thoth.Json and it was a challenge at the beginning (notice: Thoth is a decoder/encoder lib for F# inspired by Elm decoders).
Richard Feldman (Apr 19 2024 at 12:29):one thing we could potentially do: have an opaque Undecoded type which you can decode into, which just wraps List U8 and which can be converted to List U8 later
Richard Feldman (Apr 19 2024 at 12:29):that would only involving parsing those bytes twice if we actually needed to look at their contents
Richard Feldman (Apr 19 2024 at 12:30):whereas if we're just passing them through to something else unaltered, they'd only be parsed once
Eli Dowling (Apr 19 2024 at 12:32):Eli Dowling said:
Oh, I can technically do this in roc right now but it would involve parsing all the json, and just storing a copy of all the bytes for that section. Then doing the actual decoding when it's accessed. Technically it's a solution, but it's very odd and feels like a hack because it involves re-parsing the JSON multiple times
May I refer you to this ^ ;)
You always have to parse it twice because even if you don't decode the json you have to parse it to figure out where the bytes for the object you are not decoding end
Richard Feldman (Apr 19 2024 at 12:50):right yeah
Richard Feldman (Apr 19 2024 at 12:50):oh does that work already with Decoding somehow?
Richard Feldman (Apr 19 2024 at 12:51):I assumed it would need an opaque type but maybe not
Brendan Hansknecht (Apr 19 2024 at 14:45):Does it really need parsing twice or just a super quick depth with various brackets scan?
Eli Dowling (Apr 20 2024 at 01:21):Well pretty much, you have to find all the strings to find any brackets within strings. And all the escapes within those strings to find the end of the string, so it's a big part of the parsing.
Eli Dowling (Apr 20 2024 at 01:24):Richard Feldman said:
I assumed it would need an opaque type but maybe not
It is an opaque type. I basically re-implemented a subset of the Json parsing to find the bytes for that value.
Ayaz Hafiz (Apr 20 2024 at 02:50):you could also re-implement serde's Value and decode into that. Then you don't need to re-parse on demand, but just switch on the actual value.
Ayaz Hafiz (Apr 20 2024 at 02:51):There's no other way to get runtime polymorphism other than to do it at runtime.
Brendan Hansknecht (Apr 20 2024 at 02:55):Well pretty much, you have to find all the strings to find any brackets within strings. And all the escapes within those strings to find the end of the string, so it's a big part of the parsing.
This is definitely where the learnings from simd-json will likely be quite useful.
Last updated: Jun 16 2026 at 16:19 UTC