Stream: beginners

Topic: Parser hits out of memory error... why?

Asbjørn Olling (Dec 02 2023 at 20:48):

So I'm working on todays advent of code.
I started using @Luke Boswell's roc-parser because it looks like it has the comfy parser stuff I want.
It works pretty well, but it fails with an "out of memory" panic in some places, and I can't figure out why.
I don't think I'm doing any spooky infinite recusion stuff.

The error I get on the tests that panic is:
thread 'main' panicked at 'out of memory', /build/cargo-vendor-dir/bumpalo-3.14.0/src/li b.rs:1854:5

I've been going mad about this for over an hour now; I can't seem to work around it.
So now I'm asking for help in here again :sweat_smile:
I've whittled it down to some small-ish things that fail:

Color : [Red, Green, Blue]

parseColor : Parser RawStr Color
parseColor = oneOf [
  string "red" |> map (\_ -> Red),
  string "green" |> map (\_ -> Green),
  string "green" |> map (\_ -> Blue)
]
expect # this one works
  in = "red"
  result = parseStr parseColor in
  result == Ok Red

expect # this one works too
  in = "a;a;a"
  parser = sepBy1 (string "a") (string ";")
  out = parseStr parser in
  out == Ok (["a", "a", "a"])

expect # this panics... why???
  in = "red;green;blue"
  parser = sepBy1 parseColor (string ";")
  out = parseStr parser in
  out == Ok ([Red, Green, Blue])

expect # this works
  in = "a"
  result = parseStr (string "a") in
  result == Ok "a"

expect # ..but this panics
  in = "ab"
  result = parseStr (string "a") in
  result == Ok "a"

Asbjørn Olling (Dec 02 2023 at 20:50):

Using roc from commit 66fa633fde6cb7067de5303445a8402c15921b3d (current main), built using the nix flake.

Isaac Van Doren (Dec 02 2023 at 20:53):

You have string "green" |> map (\_ -> Blue), not sure if that is related to the panics though

Asbjørn Olling (Dec 02 2023 at 20:53):

hahah thanks! good catch. I don't think that should cause the memory error, though.

Asbjørn Olling (Dec 02 2023 at 20:53):

at least I hope not

Asbjørn Olling (Dec 02 2023 at 20:56):

it seems to happen whenever I

• use sepBy or sepBy1 on a more complicated parser (something with oneOf or many)
• run parseStr on any string that is longer than the expected input (e.g. parseStr (string "a") "ab")

Isaac Van Doren (Dec 02 2023 at 20:57):

For me changing it from green to blue did actually fix the error. But I don't think it should be causing one

Isaac Van Doren (Dec 02 2023 at 20:58):

I ran into the same issue when I was writing my solution

Asbjørn Olling (Dec 02 2023 at 20:58):

hmm yeah that does fix the "red;green;blue" test I posted

Asbjørn Olling (Dec 02 2023 at 20:59):

maybe that's enough that I can keep going
but parseStr (string "a") "ab" still panics

Asbjørn Olling (Dec 02 2023 at 21:00):

probably a compiler bug, right? I really feel there's no way that line should cause an "out of memory" error

Asbjørn Olling (Dec 02 2023 at 21:06):

hmm hold on; I'm not sure I can reproduce any more.

Brendan Hansknecht (Dec 02 2023 at 21:09):

My guess either:

1. we limit the size in the compiler of something and it really should be bigger
2. some sort of infinite recursion or memory bug that actually eats memory until we crash.

Haven't actually looked into what is going, might dig in a bit more if I have time.

Asbjørn Olling (Dec 02 2023 at 21:10):

I can reproduce it again. It seems to not happen if my expectation is actually correct. Maybe only happens on expectation fail?

expect # this succeeds. perfect
  out = parseStr (string "a") "ab"
  out == Err (ParsingIncomplete "b")

expect # this panics
  out = parseStr (string "a") "ab"
  out == Ok "a"

Asbjørn Olling (Dec 02 2023 at 21:12):

In any case I have 10+ gigs of unused memory, and it fails pretty quickly (doesn't seem like it's slowly eat up all my RAM first)

Asbjørn Olling (Dec 02 2023 at 21:13):

It also happens inside bumpalo, which seems to be an arena allocator? Sorry I don't know a whole lot about how that works :sweat_smile:

Brendan Hansknecht (Dec 02 2023 at 21:13):

Yeah, my guess is it is an internal compiler buffer that has limited size

Isaac Van Doren (Dec 02 2023 at 21:15):

In case it's useful, this does not panic

expect # this does not panic
    parseStr (string "a") "ab" == Ok "a"

Brendan Hansknecht (Dec 02 2023 at 21:15):

Do you have a full sample that I can just copy into an app.roc file that is a minimal-ish repro?

Asbjørn Olling (Dec 02 2023 at 21:19):

here:

app "hello"
    packages {
        pf: "https://github.com/roc-lang/basic-cli/releases/download/0.7.0/bkGby8jb0tmZYsy2hg1E_B2QrCgcSTxdUlHtETwm5m4.tar.br",
        parser: "https://github.com/lukewilliamboswell/roc-parser/releases/download/0.2.0/dJQSsSmorujhiPNIvJKlQoI92RFIG_JQwUfIxZsCSwE.tar.br"
    }
    imports [
        pf.Stdout,
        pf.Task.{ Task },
        parser.String.{ string, parseStr },
    ]
    provides [main] to pf

expect # this works
  out = parseStr (string "a") "ab"
  out == Err (ParsingIncomplete "b")

expect # this runs too (but fails)
  parseStr (string "a") "ab" == Ok "a"

expect # this panics
  out = parseStr (string "a") "ab"
  out == Ok "a"

main : Task.Task {} I32
main =
  Stdout.line "hi"

Brendan Hansknecht (Dec 02 2023 at 21:31):

What do you mean you can't allocate a string of length: 9223372036854775809

Brendan Hansknecht (Dec 02 2023 at 21:31):

haha

Brendan Hansknecht (Dec 02 2023 at 21:33):

Cool, I know exactly what the bug is actually. (maybe cause I indirectly caused it :shushing_face:)

Brendan Hansknecht (Dec 02 2023 at 23:59):

Should be fixed by #6158

Last updated: Jul 06 2025 at 12:14 UTC