basic-cli reqwest lib size · contributing

Are platform (in this case rust) features something that's on the table? For example if someone is making a cli that does not use networking they can save lots of binary size.

Agus Zubiaga (Nov 16 2023 at 16:47):

Maybe linking with —gc-sections is enough?

Agus Zubiaga (Nov 16 2023 at 16:48):

It’s still nice to have a small http lib for when you do use it, though

Brendan Hansknecht (Nov 17 2023 at 00:10):

So we totally could do something akin to rust features, but how Roc plans to distribute platforms, this is not really possible. In the Roc model, the entire platform is precompiled and distributed. So having some form of feature selection is not possible unless we either:

Stop precompiling platforms
Build a special system to enable opting into compiling a platform locally with certain features disabled
Distribute 2^n binaries for each of the n features a platform has

Brendan Hansknecht (Nov 17 2023 at 00:16):

Also, I'm pretty sure gc-sections won't change anything. We have a reference to the http function and it references another 5MB of code. So nothing for GC sections to do.

Elias Mulhall (Nov 17 2023 at 00:23):

That is a pickle

Where does the effect interpreter live, is it implemented by hand per-platform?

Brendan Hansknecht (Nov 17 2023 at 00:25):

Yep. Or it will once we switch to that. Currently, it is technically all generated on the roc side and more adhoc, but that is changing

Notification Bot (Nov 17 2023 at 01:04):

23 messages were moved here from #contributing > contributor coordination meeting by Brendan Hansknecht.

Brendan Hansknecht (Nov 17 2023 at 01:09):

Just as a validation test, I made a basically empty rust binary (no roc involved) that uses reqwest to do a get request. That binary is 4.8MB.

Brendan Hansknecht (Nov 17 2023 at 01:10):

Oh, with lto, it drops to 3.0MB. That's a nice gain.

Luke Boswell (Nov 17 2023 at 01:19):

Are you still thinking about using hyper? Is that much smaller?

Brendan Hansknecht (Nov 17 2023 at 01:22):

about to test

Brendan Hansknecht (Nov 17 2023 at 01:23):

Though I don't expect it to be much smaller, reqwest is just a wrapper for hyper

Agus Zubiaga (Nov 17 2023 at 01:25):

Brendan Hansknecht said:

We have a reference to the http function and it references another 5MB of code.

Even when no Roc code calls the effect function?

Agus Zubiaga (Nov 17 2023 at 01:30):

I may have a bad understanding of how the platform and app are linked. I thought the platform just exposed C-like functions and roc would generate code that calls them directly. But I guess there’s some sort of switch on effect type that dispatches to the right function?

Brendan Hansknecht (Nov 17 2023 at 01:43):

But I guess there’s some sort of switch on effect type that dispatches to the right function?

Technically it isn't this yet, but it will be in the future. So I would just work under that assumption. Even without this all of the notes on precompilation apply here (plus surgical linking) and clean up is not a simple task.

Brendan Hansknecht (Nov 17 2023 at 01:44):

Are you still thinking about using hyper?

Initial tests with hyper seem promising (but I haven't wired up most of the features), I will try to update the platform as a whole.

Brendan Hansknecht (Nov 17 2023 at 22:58):

So removing reqwest and directly using hyper saved about 700KB.

Our hello world with basic-cli is now 3.0MB on my linux machine. Of that, I think 2.1MB comes from hyper, tokio, rusttls, etc to support http requests. The other 0.9MB is other libraries and rust code.

Brendan Hansknecht (Nov 17 2023 at 23:07):

general question, does basic cli need to support http2 for web requests? Given it is just directly requesting webpages and not doing anything with keep alive, I assume that 1 vs 2 shouldn't matter much?

But honestly I don't mess with web that much anymore, so not sure if I am missing something. Removing http2 save another 400KB.

Brendan Hansknecht (Nov 17 2023 at 23:49):

I am just gonna remove http2 for now. Can be added back in later if someone has an explicit need PR now up: https://github.com/roc-lang/basic-cli/pull/130

Luke Boswell (Nov 18 2023 at 00:08):

Makes sense, this is basic-cli right. That sounds like a good candidate for more-advanced-cli

Richard Feldman (Nov 23 2023 at 02:52):

hm, this made it into 0.6.0, right? I just saw https://www.reddit.com/r/ProgrammingLanguages/comments/17zp4m7/comment/ka5akck/ and tried it out locally on macOS; when I build helloWorld.roc (which is using 0.6.0 for its platform) I'm seeing:

$ du -h helloWorld
 23M    helloWorld

is it a lot smaller on Linux? :thinking:

Brendan Hansknecht (Nov 23 2023 at 04:19):

Would make sense for Linux to be smaller, when building a binary, rust and friends build something a lot slimmer than for an object file

Brendan Hansknecht (Nov 23 2023 at 04:28):

Also, have the size can be stripped away, probably debug info. Is basic cli release in debug instead of release for some reason?

Brendan Hansknecht (Nov 23 2023 at 04:32):

Yeah, pretty sure our basic cli package is a debug build and that is the issue.

Brendan Hansknecht (Nov 23 2023 at 04:32):

I just built from source with --optimize and it is 6.5MB on macos

Brendan Hansknecht (Nov 23 2023 at 04:32):

yep and debug is 23MB

Brendan Hansknecht (Nov 23 2023 at 04:33):

so we need to cut a new package release that is actually an optimized build. That is the root issue.

Brendan Hansknecht (Nov 23 2023 at 04:37):

As a note, this probably affects basic-webserver as well, so they probably both need to cut a new release that is an optimized build.

Luke Boswell (Nov 23 2023 at 04:38):

Nice pick up

Anton (Nov 24 2023 at 11:16):

Fixed in PR#6078

Richard Feldman (Nov 25 2023 at 00:53):

Brendan Hansknecht said:

I just built from source with --optimize and it is 6.5MB on macos

do you mean using cargo build --release on the basic-cli Rust host?

I tried adding --optimize to roc build on macOS and it didn't make a difference for me :thinking:

Brendan Hansknecht (Nov 25 2023 at 01:57):

roc build --optimize but on an example that will build basic-cli from source which will use --release. this hasn't made it to the url package yet

Brendan Hansknecht (Nov 25 2023 at 01:57):

Will require a new release with Anton's pr

Anton (Nov 25 2023 at 09:41):

I'll start building 0.6.2 after today's nightlies are up

Anton (Nov 25 2023 at 13:17):

Here it is.

Richard Feldman (Nov 26 2023 at 19:26):

I can't believe I didn't think of this until just now, but an obvious way to shrink the binary size here is for someone to implement http 1.1 in Roc

Richard Feldman (Nov 26 2023 at 19:26):

then basic-cli doesn't need any host libraries at all to do http, just tcp (which comes from the OS anyway)

Richard Feldman (Nov 26 2023 at 19:27):

and any http logic that isn't used by a given roc application will be dead code eliminated anyway!

Richard Feldman (Nov 26 2023 at 19:28):

http 1.1 is also ridiculously simple (http2 not so much)

Richard Feldman (Nov 26 2023 at 19:29):

I guess tls would also probably need to be in the host for now

Richard Feldman (Nov 26 2023 at 19:29):

but I'm not sure how big that is relative to all of hyper

Brian Carroll (Nov 26 2023 at 19:52):

I wrote a HTTP 1.1 parser in Roc and it's in the repo already. Could be something to build on.

https://github.com/roc-lang/roc/blob/main/examples/parser/Parser/Http.roc

Pearce Keesling (Nov 26 2023 at 20:12):

Is dynamic linking an option for something like tls? Or do platforms need to be totally static?

Richard Feldman (Nov 26 2023 at 20:28):

they don't need to be, but I think static is the best user experience for end users of the compiled program! :big_smile:

Brendan Hansknecht (Nov 26 2023 at 21:04):

I think about 1-2MB of the 5MB of extra size is from rustls. Most of the rest is from hyper and Tokio. If we build own on http on top of the current tcp I'm not sure exactly how much we will gain. Also, if we later switch to async tcp, that might readd a lot of the costs from Tokio.

Richard Feldman (Nov 26 2023 at 22:03):

interesting, good to know!

Luke Boswell (Nov 26 2023 at 22:54):

a little off topic, but I was reading about some of the developments with WASM runtimes, and it seems a lot of these primitives are being developed into these runtimes pretty quickly, so I don't think it will be long and I can imagine a Roc platform that is almost zero overhead and provides a lot of functionality. See interface proposals like wasm-http and wasi-filesystem. I'm keen to work on something like this, but maybe that is a Post-AoC thing.

Luke Boswell (Nov 26 2023 at 22:59):

I mean on the order of the next few years*

Last updated: Jul 06 2025 at 12:14 UTC