Eli Dowling (Mar 14 2024 at 13:06): Eli Dowling (Mar 14 2024 at 13:06):So I was thinking about how to get null decoding to work inside roc.
I've got a prototype but I'd like some feedback
Now I know currently we don't have an Optional type, and so I tried to find a solution that would allow user defined types to be decodable from no data.
So I modified the records decoder in "decoder/records.rs" (my god... that certainly hurt my brain :sweat_smile: ), after a few iterations I settled on a replacement for the original finalizer function that is generated.
The generated code will look like this now:
finalizer2 = \rec, fmt ->
when
when rec.f0 is
Err _ ->
when Decode.decodeWith [] Decode.decoder fmt is
rec2 -> rec2.result
Ok a -> Ok a
is
Ok f0 ->
when
when rec.f1 is
Err NoField ->
when Decode.decodeWith [] Decode.decoder fmt is
rec2 -> rec2.result
Ok a -> Ok a
is
Ok f1 -> Ok { f1, f0 }
Err _ -> Err TooShort
Err _ -> Err TooShort
Basically if we find a record field that doesn't exist we first try to run the decoder with empty input to see if it has some valid state even if there was no data in the field.
The Option type decoder looks like this:
optionDecode = Decode.custom \bytes, fmt ->
if bytes |> List.len == 0 then
{ result: Ok (@Option (None)), rest: [] }
else
when bytes |> Decode.decodeWith (Decode.decoder) fmt is
{ result: Ok res, rest } -> { result: Ok (@Option (Some res)), rest }
{ result: Err a, rest } -> { result: Err a, rest }
I was wondering what people thought about this solution?
I'd like to get this working so i can start on my chapter for the book. Without optional types there is no way to decode json with optional fields which the LSP spec is totally full of
Eli Dowling (Mar 17 2024 at 04:50):I also have working union types using a similar principal:
Union2 u1 u2 := [U1 u1, U2 u2]
implements [
Eq {
isEq: union2Eq,
},
Decoding {
decoder: decodeUnionTwo,
},
]
union2Eq = \@Union2 a, @Union2 b -> a == b
union2a = \item -> @Union2 (U1 item)
union2b = \item -> @Union2 (U2 item)
union2Get = \@Union2 union -> union
decodeUnionTwo = Decode.custom \bytes, fmt ->
when bytes |> Decode.decodeWith (Decode.decoder) fmt is
{ result: Ok res, rest } -> { result: Ok (union2a res), rest }
_ ->
when bytes |> Decode.decodeWith (Decode.decoder) fmt is
{ result: Ok res, rest } -> { result: Ok (union2b res), rest }
{ result: Err res, rest } -> { result: Err res, rest }
I think this is a clever solution to support optional values. :heart_eyes:
My understanding of this proposal is that we modify the way the finalizer for records is derived (in crates/compiler/derive/src/decoding/record.rs).
The current finalizer derives an implementation that immediately returns an error if any of the fields are missing. This proposal is that when a field is missing, we first attempt to decode using an empty list (of bytes) and if that succeeds, we use that value for the field.
This means we can support optional fields in user-defined types by adding an implementation using Decode.custom, and then when e.g. roc-json finalises a record value that has this type it will use the default value if the field was missing from the provided json input.
I think if you have a working implementaiton then it would be good to make a PR so we can test it out. 
Eli Dowling (Mar 17 2024 at 04:58):I now have a draft PR: https://github.com/roc-lang/roc/pull/6587
Eli Dowling (Mar 17 2024 at 05:04):The full code example with tests is:
Option val := [None, Some val]
implements [
Eq {
isEq: optionEq,
},
Decoding {
decoder: optionDecode,
},
]
none = \{} -> @Option None
some = \a -> @Option (Some a)
isNone = \@Option opt ->
when opt is
None -> Bool.true
_ -> Bool.false
optionEq = \@Option a, @Option b ->
when (a, b) is
(Some a1, Some b1) -> a1 == b1
(None, None) -> Bool.true
_ -> Bool.false
optionDecode = Decode.custom \bytes, fmt ->
if bytes |> List.len == 0 then
{ result: Ok (@Option (None)), rest: [] }
else
when bytes |> Decode.decodeWith (Decode.decoder) fmt is
{ result: Ok res, rest } -> { result: Ok (@Option (Some res)), rest }
{ result: Err a, rest } -> { result: Err a, rest }
# Now I can try to modify the json decoding to try decoding every type with a zero byte buffer and see if that will decode my field
OptionTest : { y : U8, maybe : Option U8 }
expect
decoded : Result OptionTest _
decoded = "{\"y\":1}" |> Str.toUtf8 |> Decode.fromBytes TotallyNotJson.json
dbg "hil"
expected = Ok ({ y: 1u8, maybe: none {} })
isGood =
when (decoded, expected) is
(Ok a, Ok b) ->
a == b
_ -> Bool.false
isGood == Bool.true
OptionTest2 : { maybe : Option U8 }
expect
decoded : Result OptionTest2 _
decoded =
"""
{"maybe":1}
"""
|> Str.toUtf8
|> Decode.fromBytes TotallyNotJson.json
dbg "hil"
expected = Ok ({ maybe: some 1u8 })
expected == decoded
This looks good to me, what do you think @Richard Feldman?
Richard Feldman (Mar 18 2024 at 15:46):yeah, I like it! I've been thinking about this design since you posted it, and I think it makes a lot of sense :thumbs_up:
Richard Feldman (Mar 18 2024 at 15:48):we could potentially take it a step further in the future and auto-derive this behavior for decoding tag unions that look like Maybe - as in:
Just in Maybe's case, or Some in Option's case)None or Nothing, or payloads that hold no information, e.g. Err [FieldMissing]) Richard Feldman (Mar 18 2024 at 15:49):this would mean you could optional record fields into whatever tag names make sense for your use case
Richard Feldman (Mar 18 2024 at 15:50):e.g. email : [Specified Str, Unspecified]
Richard Feldman (Mar 18 2024 at 15:50):I'm not sure if that's the default behavior we want for tag unions (e.g. other formats may want a missing field to result in an error for tag unions instead), but it's interesting!
Eli Dowling (Mar 20 2024 at 11:30):I imagine you want tag unions to be encodable and decodable, that would loose the information along the way. Ie you couldn't encode email:[Specified Str, NotSpecified, NotAsked] and distinguish between NotAsked and NotSpecified
Last updated: Jul 05 2025 at 12:14 UTC