Stream: compiler development

Topic: roc-json compiler bug


view this post on Zulip Luke Boswell (Jan 11 2025 at 01:49):

Just upgrading roc json ... and I've got a heap of broken tests. I'm not sure what's change through.

The error message looks like they're are correct. Here's an example;

── EXPECT FAILED in package/Json.roc ───────────────────────────────────────────

This expectation failed:

1962│>  # Test decode of record with a string and number field
1963│>  expect
1964│>      input = Str.to_utf8("{\"first_segment\":\"ab\",\"second_segment\":10}")
1965│>      decoder = utf8_with({ field_name_mapping: SnakeCase })
1966│>      actual = Decode.from_bytes_partial(input, decoder)
1967│>      expected = Ok({ first_segment: "ab", second_segment: 10u8 })
1968│>
1969│>      actual.result == expected

When it failed, these variables had these values:

input : List U8
input = [123, 34, 102, 105, 114, 115, 116, 95, 115, 101, 103, 109, 101, 110, 116, 34, 58, 34, 97, 98, 34, 44, 34, 115, 101, 99, 111, 110, 100, 95, 115, 101, 103, 109, 101, 110, 116, 34, 58, 49, 48, 125]

decoder : Json
decoder = @Json { empty_encode_as_null: { list: Bool.false, record: Bool.true, tuple: Bool.true }, field_name_mapping: SnakeCase, null_decode_as_empty: Bool.true, skip_missing_properties: Bool.true }

actual : DecodeResult {
    first_segment : Str,
    second_segment : U8,
}
actual = { rest: [], result: Ok { first_segment: "ab", second_segment: 10 } }

expected : [
    Err DecodeError,
    Ok {
        first_segment : Str,
        second_segment : U8,
    },
]
expected = Ok { first_segment: "ab", second_segment: 10 }


22 failed and 86 passed in 1050 ms.

view this post on Zulip Luke Boswell (Jan 11 2025 at 01:50):

PR here https://github.com/lukewilliamboswell/roc-json/pull/42

view this post on Zulip Luke Boswell (Jan 11 2025 at 01:52):

actual.result   == expected
actual.result   == Ok { first_segment: "ab", second_segment: 10 }
expected        == Ok { first_segment: "ab", second_segment: 10 }

view this post on Zulip Anthony Bullard (Jan 11 2025 at 01:52):

Looks like the number type on one of the fields

view this post on Zulip Anthony Bullard (Jan 11 2025 at 01:53):

Wait

view this post on Zulip Ayaz Hafiz (Jan 11 2025 at 01:55):

expect
    input = Str.to_utf8("{\"first_segment\":\"ab\",\"second_segment\":10}")
    decoder = utf8_with({ field_name_mapping: SnakeCase })
    actual = Decode.from_bytes_partial(input, decoder)
    expected = Ok({ first_segment: "ab", second_segment: 10u8 })
    r = actual.result

    r == expected
expected : [
    Err DecodeError,
    Ok {
        first_segment : Str,
        second_segment : U8,
    },
]
expected = Ok { first_segment: "ab", second_segment: 10 }

r : Result {
    first_segment : Str,
    second_segment : U8,
} DecodeError
r = Err TooShort

view this post on Zulip Ayaz Hafiz (Jan 11 2025 at 01:55):

seems like a compiler bug

view this post on Zulip Ayaz Hafiz (Jan 11 2025 at 02:00):

this is really weird. definitely at least a compiler bug involved. if you add a dbg it changes the value of actual

[package/Json.roc:1967] actual = {rest: [], result: (Err TooShort)}
── EXPECT FAILED in package/Json.roc ───────────────────────────────────────────

This expectation failed:

1962│>  # Test decode of record with a string and number field
1963│>  expect
1964│>      input = Str.to_utf8("{\"first_segment\":\"ab\",\"second_segment\":10}")
1965│>      decoder = utf8_with({ field_name_mapping: SnakeCase })
1966│>      actual = Decode.from_bytes_partial(input, decoder)
1967│>      dbg actual

When it failed, these variables had these values:

input : List U8
input = [123, 34, 102, 105, 114, 115, 116, 95, 115, 101, 103, 109, 101, 110, 116, 34, 58, 34, 97, 98, 34, 44, 34, 115, 101, 99, 111, 110, 100, 95, 115, 101, 103, 109, 101
, 110, 116, 34, 58, 49, 48, 125]

decoder : Json
decoder = @Json { empty_encode_as_null: { list: Bool.false, record: Bool.true, tuple: Bool.true }, field_name_mapping: SnakeCase, null_decode_as_empty: Bool.true, skip_mi
ssing_properties: Bool.true }

actual : DecodeResult {
    first_segment : Str,
    second_segment : U8,
}
actual = { rest: [], result: Err TooShort }

view this post on Zulip Luke Boswell (Jan 11 2025 at 03:53):

Made a branch with just that file... https://github.com/lukewilliamboswell/roc-json/tree/bug

view this post on Zulip Luke Boswell (Jan 11 2025 at 03:54):

I'm going to try minimizing it

view this post on Zulip Luke Boswell (Jan 11 2025 at 03:59):

Could this be a clue?

I can't generate an implementation of the Eq ability for

    {
        rest : List U8,
        result : Result { first_segment : Str } DecodeError,
    }

In particular, an implementation for

    ?

cannot be generated.

Maybe it's related to snake_case idents?

view this post on Zulip Ayaz Hafiz (Jan 11 2025 at 04:01):

maybe

view this post on Zulip Luke Boswell (Jan 11 2025 at 04:02):

Maybe not... this works

module [Test]

Test := {
    lower_case : U8,
}
    implements [Eq]

expect @Test( {lower_case : 1}) == @Test( {lower_case : 1})
$ roc test Test.roc
0 failed and 1 passed in 468 ms.

view this post on Zulip Luke Boswell (Jan 11 2025 at 04:03):

So does this

module [Test]

Test := {
    result : Result { lower_case : U8 } Str,
}
    implements [Eq]

expect @Test({ result: Ok({ lower_case: 1 }) }) == @Test({ result: Ok({ lower_case: 1 }) })

view this post on Zulip Notification Bot (Jan 11 2025 at 04:03):

14 messages were moved here from #compiler development > casual conversation by Luke Boswell.

view this post on Zulip Luke Boswell (Jan 11 2025 at 04:32):

I've cut it down to something pretty minimal @Ayaz Hafiz

view this post on Zulip Luke Boswell (Jan 11 2025 at 04:36):

Ah maybe not... :sad:

view this post on Zulip Brendan Hansknecht (Jan 11 2025 at 04:38):

Feels like the most likely issue is that new idents some how broke ability deriving and some field checks? But that is a guess

view this post on Zulip Luke Boswell (Jan 11 2025 at 04:49):

Is this interesting?

https://gist.github.com/lukewilliamboswell/1e4b4bc2d4b6cb79dbc4fdc5c2125486

view this post on Zulip Luke Boswell (Jan 11 2025 at 04:51):

Probably not... we don't support Tag unions.

view this post on Zulip Artur Domurad (Jan 11 2025 at 08:23):

I think that this may be related to an old issue:
https://github.com/roc-lang/roc/issues/5727

expect
    actual = Err TestError
    expected = Ok { ownerName: "a" }

    actual == expected

roc test outputs:

actual : [
    Err [TestError],
    Ok { ownerName : Str },
]
actual = Ok { ownerName: "a" }

expected : [
    Err [TestError],
    Ok { ownerName : Str },
]
expected = Ok { ownerName: "a" }

view this post on Zulip Luke Boswell (Jan 11 2025 at 08:28):

Nice, that would certainly mask whatever the real issue is :sweat_smile:

view this post on Zulip Luke Boswell (Jan 11 2025 at 23:41):

I think I've found the roc-json issue here... and it's not some nasty compiler bug.

view this post on Zulip Luke Boswell (Jan 11 2025 at 23:42):

The above test display thing is definitely an issue, but I think my upgrade problem is just I need to update the default field formatting from camelCase to snake_case which requires some material changes to the implementation of roc-json

view this post on Zulip Brendan Hansknecht (Jan 11 2025 at 23:42):

I wonder if some of this related to the shared memory infrastructure we want to rip out?

view this post on Zulip Brendan Hansknecht (Jan 11 2025 at 23:42):

If you take that broken test case and run into in main (without any expect) just some printing does it work correctly? (Maybe use the conditional in a if to decide printing)

view this post on Zulip Brendan Hansknecht (Jan 11 2025 at 23:42):

Or does a dbg in the test print the correct thing?

view this post on Zulip Brendan Hansknecht (Jan 11 2025 at 23:43):

Those two both would be signs it is a bug with the shared mem infra

view this post on Zulip Luke Boswell (Jan 11 2025 at 23:43):

I'm just in the middle of rewriting the string conversions I can try that later

view this post on Zulip Luke Boswell (Jan 12 2025 at 01:11):

Turns out converting between all the cases (even just assuming ASCII) is a fairly deep rabbit hole :smiley:

Taking my time to implement these properly so we can have something reusable

view this post on Zulip Luke Boswell (Jan 12 2025 at 01:36):

My plan is to make a PR for @Hannes roc-ascii with some helpers to convert between different formats

view this post on Zulip Luke Boswell (Jan 12 2025 at 01:36):

It will be a dependency for roc-json then... but that should be ok I think. I might also just vendor the helpers

view this post on Zulip Anthony Bullard (Jan 12 2025 at 02:19):

Are you saying that roc-jsons implementation of camel->snake casing has a bug? Or the Roc compilers?

view this post on Zulip Ayaz Hafiz (Jan 12 2025 at 02:29):

both do

view this post on Zulip Luke Boswell (Jan 12 2025 at 04:35):

Announcing... CamelSplitting.roc -- sharing here in case anyone can see any issues with this implementation.

https://gist.github.com/lukewilliamboswell/326d451e24e5aba8c0c7f5b0775f2ef9

It took me a little while to flush out all the edge cases, but I'm reasonably happy with this I think. My primary goal was to be somewhat efficient and avoid allocating lots of strings.

view this post on Zulip Luke Boswell (Jan 12 2025 at 07:06):

Fixed it :smiley:

$ roc test package/Json.roc
0 failed and 157 passed in 1108 ms.

Now for the staged PR https://github.com/lukewilliamboswell/roc-json/pull/42

All CI checks are :check:

view this post on Zulip Anthony Bullard (Jan 12 2025 at 11:25):

Ayaz Hafiz said:

both do

I’d like to know what bugs there are in the compiler version. For ASCII idents that is

view this post on Zulip Brendan Hansknecht (Jan 12 2025 at 16:46):

Luke Boswell said:

Fixed it :smiley:

$ roc test package/Json.roc
0 failed and 157 passed in 1108 ms.

Now for the staged PR https://github.com/lukewilliamboswell/roc-json/pull/42

All CI checks are :check:

Is the compiler bug fixed too? I assume that bug is just with expect and shared memory?

view this post on Zulip Ayaz Hafiz (Jan 12 2025 at 16:50):

it's not fixed. yeah it's a bug with the shm


Last updated: Jul 06 2025 at 12:14 UTC