Stream: beginners

Topic: big executables

Artur Swiderski (Sep 24 2023 at 22:09):

Hi I noticed executable produced on linux platform are huge like 100 MB which is crazy

Richard Feldman (Sep 24 2023 at 22:43):

which platform are you using? That will generally determine the size of the executable

Artur Swiderski (Sep 24 2023 at 22:55):

ubuntu 20.04

Artur Swiderski (Sep 24 2023 at 22:57):

both C and rust , roc render the same size

Richard Feldman (Sep 24 2023 at 23:04):

I mean what roc platform are you using

Richard Feldman (Sep 24 2023 at 23:04):

basic-cli?

Richard Feldman (Sep 24 2023 at 23:05):

also, are you running roc with the --optimize flag?

Artur Swiderski (Sep 25 2023 at 05:36):

I am compiling this with recent build (flag does not have impact whatsoever roc build --optimize file.roc )

app "peek"

packages { pf: "https://github.com/roc-lang/basic-cli/releases/download/0.5.0/Cufzl36_SnJ4QbOoEmiJ5dIpUxBvdB3NEySvuH82Wio.tar.br"
}

imports [

pf.Stdin,

pf.Stdout,

pf.Stderr,

pf.File,

pf.Path,

pf.Task.{ Task
}]

provides [main]
to pf





tokenize = \ str
->

Str.split str " "

|> List.dropIf Str.isEmpty





filterInternal = \ str, matches,
 inFlag ->

if List.isEmpty matches
== Bool.true
then

str

else

Str.split str "\n"

|> List.walk
""
( \ state
, elem
->

checkResult = List.walk matches Bool.false
( \flag, pat
->

if flag
== Bool.true
then

Bool.true

else

when
(Str.splitFirst elem pat
)
is

Ok _
-> Bool.true

Err _
-> Bool.false)

if
(checkResult
== inFlag)
&&
(Str.isEmpty
(Str.trim elem
)
== Bool.false)
then

Str.concat state elem

|> Str.concat
"\n"

else

state

)



filterOut = \ str, matches
->


workedOut = List.walk matches
{ whiteList
:
[], blackList
: []
}

( \ state
, word
->

if Str.startsWith word
"@"
== Bool.true
then

{ state & blackList: List.append state.blackList
(Str.replaceFirst word
"@"
"" )
}

else

{ state & whiteList: List.append state.whiteList word
}

)

filterInternal str workedOut.whiteList Bool.true

|> filterInternal workedOut.blackList Bool.false





main =

read =

path = Path.fromStr
"log.txt"

out = Path.fromStr
"out.txt"

hien <- File.readUtf8 path
|> Task.await


loopTask =

_
<- Stdout.line
"provide your filters: "
|> Task.await

text <- Task.await Stdin.line

result =
(filterOut hien
(tokenize text)
)

_
<- File.writeUtf8 out result
|> Task.await

_
<- Stdout.line result
|> Task.await

_
<- Stdout.line
"\n\nLast you used:
\(text)"
|> Task.await

Stdout.line "\nload data to out.txt"



Task.loop {} \_
-> Task.map loopTask Step




Task.attempt read \result ->

when result
is

Ok {}
-> Stdout.line
"Successfully wrote a string to out.txt"

Err err ->

msg =

when err
is

FileWriteErr _ PermissionDenied
->
"PermissionDenied"

FileWriteErr _ Unsupported
->
"Unsupported"

FileWriteErr _
(Unrecognized
_ other)
-> other

FileReadErr _
_
-> "Error reading file"

_
->
"Uh oh, there was an error!"



{}
<- Stderr.line msg
|> Task.await

Task.ok {}

Artur Swiderski (Sep 25 2023 at 05:38):

I am getting like 109 mega bytes executable which is outrageous

Brendan Hansknecht (Sep 25 2023 at 05:54):

Yeah, basic-cli is very bloated as a base. IIRC, we need to update some dead code elimination stuff on the rust side. This isn't exactly a roc issue. A much slimmer platform can be built, but especially with surgical linking, I think we have some changes that need to be done to get rust to slim down the platform binary.

Brendan Hansknecht (Sep 25 2023 at 05:55):

Almost none of that bloat is from actually roc code.

Brendan Hansknecht (Sep 25 2023 at 05:56):

Maybe in a bit, I double back to the dead code stuff and basic cli. See if I can get rust to stop including so much extra code and bloating the binary.

Artur Swiderski (Sep 25 2023 at 06:00):

it happens also when I am using C based platform

Brendan Hansknecht (Sep 25 2023 at 06:01):

That doesn't make sense. Maybe there is some other bug. Any info on the C platform?

Artur Swiderski (Sep 25 2023 at 06:02):

and it is not so insignificant yes hard drives are huge nowadays but 100 MB is crazy and unacceptable in long run for potential user, it just hurts eyes

Artur Swiderski (Sep 25 2023 at 06:04):

when I do ./roc examples/platform-switching/rocLovesC.roc and than compile resulting binary is more or less the same

Artur Swiderski (Sep 25 2023 at 06:05):

I am using ubuntu 20.04 with normal C libraries from repository just generic linux

Brendan Hansknecht (Sep 25 2023 at 06:17):

I'll take a look. See if I can repro

Artur Swiderski (Sep 25 2023 at 06:27):

size.png

Brendan Hansknecht (Sep 25 2023 at 06:30):

Hmmm. Yeah... not able to repro for the C platform. du -hs rocLovesC is reporting 32K

That said, I am seeing this for anything built on basic-cli. Hello world is 105MB because of basic cli.

Brendan Hansknecht (Sep 25 2023 at 06:30):

I assume peek is based on basic cli?

Artur Swiderski (Sep 25 2023 at 06:31):

you can se source code above

Brendan Hansknecht (Sep 25 2023 at 06:31):

Also, your screenshot does not show the size of rocLoveC (that or I am missing it)

Artur Swiderski (Sep 25 2023 at 06:32):

app "peek"

packages { pf: "https://github.com/roc-lang/basic-cli/releases/download/0.5.0/Cufzl36_SnJ4QbOoEmiJ5dIpUxBvdB3NEySvuH82Wio.tar.br"
}

imports [

pf.Stdin,

pf.Stdout,

pf.Stderr,

pf.File,

pf.Path,

pf.Task.{ Task
}]

provides [main]
to pf





tokenize = \ str
->

Str.split str " "

|> List.dropIf Str.isEmpty





filterInternal = \ str, matches,
 inFlag ->

if List.isEmpty matches
== Bool.true
then

str

else

Str.split str "\n"

|> List.walk
""
( \ state
, elem
->

checkResult = List.walk matches Bool.false
( \flag, pat
->

if flag
== Bool.true
then

Bool.true

else

when
(Str.splitFirst elem pat
)
is

Ok _
-> Bool.true

Err _
-> Bool.false)

if
(checkResult
== inFlag)
&&
(Str.isEmpty
(Str.trim elem
)
== Bool.false)
then

Str.concat state elem

|> Str.concat
"\n"

else

state

)



filterOut = \ str, matches
->


workedOut = List.walk matches
{ whiteList
:
[], blackList
: []
}

( \ state
, word
->

if Str.startsWith word
"@"
== Bool.true
then

{ state & blackList: List.append state.blackList
(Str.replaceFirst word
"@"
"" )
}

else

{ state & whiteList: List.append state.whiteList word
}

)

filterInternal str workedOut.whiteList Bool.true

|> filterInternal workedOut.blackList Bool.false





main =

read =

path = Path.fromStr
"log.txt"

out = Path.fromStr
"out.txt"

hien <- File.readUtf8 path
|> Task.await


loopTask =

_
<- Stdout.line
"provide your filters: "
|> Task.await

text <- Task.await Stdin.line

result =
(filterOut hien
(tokenize text)
)

_
<- File.writeUtf8 out result
|> Task.await

_
<- Stdout.line result
|> Task.await

_
<- Stdout.line
"\n\nLast you used:
\(text)"
|> Task.await

Stdout.line "\nload data to out.txt"



Task.loop {} \_
-> Task.map loopTask Step




Task.attempt read \result ->

when result
is

Ok {}
-> Stdout.line
"Successfully wrote a string to out.txt"

Err err ->

msg =

when err
is

FileWriteErr _ PermissionDenied
->
"PermissionDenied"

FileWriteErr _ Unsupported
->
"Unsupported"

FileWriteErr _
(Unrecognized
_ other)
-> other

FileReadErr _
_
-> "Error reading file"

_
->
"Uh oh, there was an error!"



{}
<- Stderr.line msg
|> Task.await

Brendan Hansknecht (Sep 25 2023 at 06:33):

Yeah, peek is built on basic CLI which is why it is giant. I'll take a look at that. See if I can fix the platform to properly do dead code elimination and remove bloat.

Artur Swiderski (Sep 25 2023 at 06:35):

I checked size of rocLoveC is 32k bytes but is sits in different location

Brendan Hansknecht (Sep 25 2023 at 06:36):

Ok. Good. So a base c platform has a fine size

Brendan Hansknecht (Sep 25 2023 at 06:37):

Yeah, so definitely a basic CLI issue leading to the giant binary size.

Brendan Hansknecht (Sep 25 2023 at 06:37):

Thanks for the report

Artur Swiderski (Sep 25 2023 at 09:58):

just a thought, I know that from perspective of language creators, things like this do not matter much , but for me as an user they are vital. I wanted to share this simple app with my collage, to make process of analyzing logs a bit more pleasant (I could do the same with grep but still I like this approach more) and right away he pointed out horrendous size of the file with suspicion. He would be no potential user or Roc under any circumstances anyway (because he has different interests) but still .. It is not good advertisement

Anton (Sep 25 2023 at 10:46):

We care deeply about the Roc user experience, including the produced file size. It's important to realize that as an open source project we have very little financial resources which unfortunately results in a user experience that currently falls short of our goals. We easily have enough work to employ 30 engineers full-time but we try to make it work with what we have.

Artur Swiderski (Sep 25 2023 at 11:56):

ok I got that

Anton (Sep 25 2023 at 16:40):

I almost forgot; the --opt-size flag can reduce the size of the produced executable a lot:

roc build your-program.roc --opt-size

Still larger than what we want it to be, but a nice improvement nonetheless.

Brendan Hansknecht (Sep 25 2023 at 16:46):

That won't affect the platform, which is the problem here.

Anton (Sep 25 2023 at 16:48):

It worked for me

basic-cli on  main [!?] via ❄  impure (nix-shell-env)
❯ ls -lh examples/command
-rwxr-xr-x 1 username users 104M sep 25 18:37 examples/command

basic-cli on  main [!?] via ❄  impure (nix-shell-env)
❯ ./roc_nightly/roc build examples/command.roc --opt-size
🔨 Rebuilding platform...
0 errors and 0 warnings found in 9239 ms while successfully building:

    examples/command

basic-cli on  main [!?] via ❄  impure (nix-shell-env) took 9s
❯ ls -lh examples/command
-rwxr-xr-x 1 username users 21M sep 25 18:38 examples/command

Brendan Hansknecht (Sep 25 2023 at 16:52):

Oh, you are rebuilding the platform and running the version of command.roc in the basic-cli repo? This would have no affect on the downloaded basic-cli platform.

Anton (Sep 25 2023 at 17:07):

Oh riiight, makes sense :+1:

Brendan Hansknecht (Sep 25 2023 at 17:41):

I think I have a solution that should drop it to about 15MB. (that said, I am not sure if it will run into some linker issues depending on platform)

Anton (Sep 25 2023 at 18:17):

Are the required changes isolated or spread out? I'm asking that to determine if it would be easy to revert if we run into trouble say 3 months from now.

Brendan Hansknecht (Sep 25 2023 at 18:33):

Very isolated: https://github.com/roc-lang/roc/pull/5855

Brendan Hansknecht (Sep 25 2023 at 18:33):

Then just need to rebuild basic-cli and other rust platforms and publish a new version.

Anton (Sep 25 2023 at 18:36):

Nice, then it's good for me! I will wait to merge to give others time to object

Anton (Sep 26 2023 at 09:08):

merged :tada:

Artur Swiderski (Sep 30 2023 at 19:13):

I took recent build but after compilation I am still getting 109 MB executable, basically the same code as mentioned earlier

Brendan Hansknecht (Sep 30 2023 at 20:28):

The basic cli platform hash not been updated yet. So it still has a cached 100MB exe that is being built off of.

Brendan Hansknecht (Sep 30 2023 at 20:28):

Sorry for the delay. basic-cli just releases separately from the compiler.

Brendan Hansknecht (Oct 05 2023 at 14:55):

Looping back to this @Anton, can you cut a new release of basic CLI?

Anton (Oct 06 2023 at 08:55):

Yes, I'll try to get it done today

Anton (Oct 06 2023 at 18:02):

I'm hitting this issue only with the musl target:

❯ export CARGO_BUILD_TARGET=x86_64-unknown-linux-musl

username in 🌐 ubu-22-04 in Downloads/temp2/roc_nightly
❯ ./roc build ../basic-cli/examples/countdown.roc
🔨 Rebuilding platform...
An internal compiler expectation was broken.
This is definitely a compiler bug.
Please file an issue here: https://github.com/roc-lang/roc/issues/new/choose
thread 'main' panicked at 'Undefined Symbol in relocation, (+a29, Relocation { kind: PltRelative, encoding: Generic, size: +20, target: Symbol(SymbolIndex(+70)), addend: +fffffffffffffffc, implicit_addend: false }): Ok(Symbol { name: "roc_alloc", address: +0, size: +0, kind: Label, section: Undefined, scope: Unknown, weak: false, flags: Elf { st_info: +10, st_other: +0 } })', crates/linker/src/elf.rs:1486:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I do test basic-cli with the latest nightly once a day, but not with a musl target, so I did not catch it before.
I'll bisect it tomorrow.

Brendan Hansknecht (Oct 06 2023 at 20:12):

That could be cause by this, but would be surprising if musl refuses to expose symbols, but others dont

Brendan Hansknecht (Oct 06 2023 at 20:12):

Oh.....wait....musl is fully static? If so, I wonder if rust is ignoring that it should be exposing dynamic symbols......hmm

Anton (Oct 07 2023 at 10:52):

Changing -C link-args=-rdynamic back to -C link-dead-code indeed fixes the issue

Brendan Hansknecht (Oct 07 2023 at 14:27):

I'll do some testing

Brendan Hansknecht (Oct 09 2023 at 15:57):

@Anton, any tips or setup for building with musl? I am getting weird issues like: cannot find function, tuple struct or tuple variant Some in this scope when building dependencies.

Anton (Oct 09 2023 at 17:13):

Hmm, perhaps you need to install musl-tools?
You can find the exact build steps I use on ubuntu-20.04 github CI here and here.

Brendan Hansknecht (Oct 21 2023 at 18:36):

I know this is super delayed, but I think this pr should enable you to cut a release of basic cli: https://github.com/roc-lang/basic-cli/pull/113

Brendan Hansknecht (Oct 23 2023 at 02:26):

An extra bonus once the new basic cli is released. Mac linking for basic cli based apps gets about 8x faster

Last updated: Jul 05 2025 at 12:14 UTC