Brendan Hansknecht (Dec 28 2024 at 20:02): Brendan Hansknecht (Dec 28 2024 at 20:02):So I was looking at the sqlite3 issues that basic-webserver hit with sqlite3-sys: https://github.com/roc-lang/basic-webserver/pull/82
Brendan Hansknecht (Dec 28 2024 at 20:03):I think the root issue is just that the sqlite crate does not bundle sqlite, but when switching to sqlite-sys, we switched to bundling sqlite
Brendan Hansknecht (Dec 28 2024 at 20:03):So everything probably would work if we just reloaded everything but avoided bundling sqlite
Brendan Hansknecht (Dec 28 2024 at 20:03):That said, I still want to look into if we can bundle sqlite somehow cause that is nicer overall.
Brendan Hansknecht (Dec 28 2024 at 20:05):Also, when we tried just disabling default features, that failed cause that leads to neither bundling or linking sqlite3. We needed to switch to the linkage feature if we wanted to link instead of bundle.
Brendan Hansknecht (Dec 28 2024 at 20:07):As a note, if we can't get sqlite3-sys to work bundled, we can also try libsqlite3-sys (that is what rusqlite uses and maintains)
Brendan Hansknecht (Dec 28 2024 at 20:07):Just collecting info on all of this now
Brendan Hansknecht (Dec 28 2024 at 20:38):@Luke Boswell is the current path for basic cli to just manually implement glue and have the roc_* crates implement what they use. So roc_file implements all file related types. roc_sqlite will implement all sqlite related types.
Luke Boswell (Dec 28 2024 at 21:25):Yeah, that's what I've been trying... no idea if it's the best way, just the first thing I thought of.
Luke Boswell (Dec 28 2024 at 21:25):I tried to group them into crates that basic-webserver and basic-ssg would need. Sqlite would definitely be its own I think.
Brendan Hansknecht (Dec 28 2024 at 22:06):SG
Brendan Hansknecht (Dec 28 2024 at 22:06):Hopefully will have a PR soon
Brendan Hansknecht (Dec 29 2024 at 00:03):@Anton What do I need to do to make sure my new PR for sqlite doesn't break releasing?
Do I just need to attempt to build with musl?
Luke Boswell (Dec 29 2024 at 00:20):You could make a new tag on the webserver repo, and then make a new PR just like that release one, and just build a new release. It wont upload any files, that is done manually by Anton
Luke Boswell (Dec 29 2024 at 00:20):Or the basic-cli repo... whichever you are more interested in
Luke Boswell (Dec 29 2024 at 00:21):If it passes CI in roc-lang/roc then you know it's all good
Brendan Hansknecht (Dec 29 2024 at 00:24):To tag, I'll need to merge into main, right? So I guess first I need to land in basic cli. Then will make a test release to make sure everything is working in basic-cli. If not, will iterate.
Luke Boswell (Dec 29 2024 at 00:24):No I don't think you need it on main
Luke Boswell (Dec 29 2024 at 00:24):Just a tag -- in the remote (roc-lang/basic-cli) repo
Brendan Hansknecht (Dec 29 2024 at 00:24):Oh, ok
Luke Boswell (Dec 29 2024 at 00:25):We can delete the tag afterwards, it's just so the CI runners checkout the correct branch/commit
Brendan Hansknecht (Dec 29 2024 at 00:30):Makes sense
Brendan Hansknecht (Dec 29 2024 at 00:34):kicked off here: https://github.com/roc-lang/roc/actions/runs/12530963666/job/34948060619?pr=7428
I think it should build. Though won't be fully functional until #7427 lands and is pulled in
Brendan Hansknecht (Dec 29 2024 at 03:09):I feel really close here for the release. Only issue is on linux arm64 with musl. I think the issue is that cc is compiling for glibc instead of musl for some reason. And an old version of glibc at that. As such, it is generating a call to fcntl64 which no longer exists. It is just fcntl now.
Brendan Hansknecht (Dec 29 2024 at 03:10):Maybe I can just update glibc on the system. Maybe there is some config I can set to get the c properly building targeting musl libc.
Luke Boswell (Dec 29 2024 at 03:38):Only issue is on linux arm64 with musl
It's not building against musl though is it?
Luke Boswell (Dec 29 2024 at 03:38):It's just running + cargo build --release on the native linux arm64
Luke Boswell (Dec 29 2024 at 03:39):(deleted)
Luke Boswell (Dec 29 2024 at 03:39):Maybe we could try building against the musl target
Luke Boswell (Dec 29 2024 at 03:41):So instead we do cargo build --release --target aarch64-unknown-linux-musl?
Luke Boswell (Dec 29 2024 at 03:46):Ignore my last, looking at the actual workflow in .github/workflows/basic_cli_build_release.yml we can see
- name: build basic-cli
env:
CARGO_BUILD_TARGET: aarch64-unknown-linux-musl
CC_aarch64_unknown_linux_musl: clang-18
AR_aarch64_unknown_linux_musl: llvm-ar-18
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS: "-Clink-self-contained=yes -Clinker=rust-lld"
run: ./ci/build_basic_cli.sh linux_arm64
Yeah, I can test that. Though it should be building against musl due to the environment variable CARGO_BUILD_TARGET: aarch64-unknown-linux-musl
Luke Boswell (Dec 29 2024 at 03:53):I wonder how far we are from fully statically linked... and if that effort would also help us here?
Brendan Hansknecht (Dec 29 2024 at 04:05):Also, in the final link command, I see a bunch of aarch64-unknown-linux-gnu. So I guess that CARGO_BUILD_TARGET is not enough
Luke Boswell (Dec 29 2024 at 04:10):I think I see the issue... in build_basic_cli.sh ... hmm, actually I'm not sure
Brendan Hansknecht (Dec 29 2024 at 04:12):I just pushed something to set --target as well. We'll see if it works.
Luke Boswell (Dec 29 2024 at 04:14):Might be helpful to echo $CARGO_BUILD_TARGET in there too somewhere so we can see what's getting to that point
Luke Boswell (Dec 29 2024 at 04:15):Also in build.roc we might need to do something... I think that gets called as well
Luke Boswell (Dec 29 2024 at 04:16):So I wonder if the call to roc build.roc in ci/build_basic_cli.sh is passing the CARGO_BUILD_TARGET env var down
Luke Boswell (Dec 29 2024 at 04:16):There's two layers of shells there
Brendan Hansknecht (Dec 29 2024 at 04:25):I'm not sure, but currently we are failing in the cargo build --release of jumpstart.sh from basic-cli
Brendan Hansknecht (Dec 29 2024 at 04:56):can now see: + cargo build --release --target aarch64-unknown-linux-musl
Yet still failing while linking to stuff in 1.82.0-aarch64-unknown-linux-gnu
Brendan Hansknecht (Dec 29 2024 at 04:57):Feels like I am missing some sort of linker config to update to musl
Brendan Hansknecht (Dec 29 2024 at 05:00):Testing now with rust-lld disabled, maybe that is the issue
Brendan Hansknecht (Dec 29 2024 at 05:11):That didn't help. Though it did give a slightly different form of the error. Some minor context from searching it:
No worries. I found out it was eventually a problem with incompatible versions of glib for cross-compilation. I would assume you cross-compiled on <= Ubuntu 18 with GLIB 2.27 which is strictly not-forward compatible with GLIB2.28 on the Raspbian (Debian Buster). Faintly recalling, they have changed the header for fcntl64 to a macro in the latest version of GLIB.
Brendan Hansknecht (Dec 29 2024 at 05:12):given we are building with musl, idk why we are hitting glibc errors, but I wonder if we can just update glibc on the pi
Brendan Hansknecht (Dec 29 2024 at 05:13):cc: @Anton in case you have ideas
Brendan Hansknecht (Dec 29 2024 at 05:13):If this doesn't pan out, eventually I'll give in and switch to linking sqlite instead of bundling it, but it would be nicer to bundle it if possible.
Ayaz Hafiz (Dec 29 2024 at 05:33):Does the build target set CC? https://github.com/stainless-steel/sqlite3-src/blob/7704dc7f0842ba2708dfcc8baa2a526dc0730020/build.rs#L5
Luke Boswell (Dec 29 2024 at 05:58): Brendan Hansknecht (Dec 29 2024 at 06:00):The CC environment variable is set to clang-18
Luke Boswell (Dec 29 2024 at 06:01):We could try setting CC_ENABLE_DEBUG_OUTPUT
Brendan Hansknecht (Dec 29 2024 at 06:01):I think clang should support musl. Nothing fails until the final link step
Brendan Hansknecht (Dec 29 2024 at 06:01):Luke Boswell said:
We could try setting
CC_ENABLE_DEBUG_OUTPUT
I tried, didn't see any extra output in CI.
Luke Boswell (Dec 29 2024 at 06:03):https://github.com/rust-lang/cc-rs/issues/1251 maybe?
Luke Boswell (Dec 29 2024 at 06:10):@Brendan Hansknecht
Mind if I try adding this to .cargo/config.toml?
[target.aarch64-unknown-linux-musl]
linker = "aarch64-linux-musl-gcc"
rustflags = ["-C", "target-feature=+crt-static"]
[profile.release]
lto = true
codegen-units = 1
panic = "abort"
Oh, interesting, disabled clang as the default cc and it is looking for aarch64-linux-musl-gcc. This feels relevant. Maybe clang doesn't correctly support musl and thus is leading to still linking glibc. Sadly, even though musl-tools should be installed, aarch64-linux-musl-gcc is not found.
Brendan Hansknecht (Dec 29 2024 at 06:13):Luke Boswell said:
Mind if I try adding this to
.cargo/config.toml?
go for it
Luke Boswell (Dec 29 2024 at 06:14):Do you think we need these flags? "-Clink-self-contained=yes -Clinker=rust-lld"
Luke Boswell (Dec 29 2024 at 06:14):Maybe using rust-lld is better if we know it's available
Brendan Hansknecht (Dec 29 2024 at 06:14):To my understanding link-self-contained=yes should be default with musl and linker=rust-lld should lead to faster linking
Brendan Hansknecht (Dec 29 2024 at 06:15):So neither should be needed.
Brendan Hansknecht (Dec 29 2024 at 06:15):Feel free to push to either branch and update the tag if needed. I am gonna log off for the night
Brendan Hansknecht (Dec 29 2024 at 18:06):From more reading, it sounds like rust just tends to hit toolchain issues when compiling for aarch64-unknown-linux-musl (looks to even happen from time to time with x86_64 musl). My best guess is that rust does not actually correctly enable the toolchain. Scouring the web, there are a mix of various linking and compilation issues that are regularly hit with this musl.
I am currently testing a workaround that someone claimed work for sqlite: https://github.com/briansmith/ring/issues/1414#issuecomment-1596300080
That said, it sounds like the only sure-fire way to get rust to compile for aarch64-unknown-linux-musl is to use cross. I guess musl is different enough that it is best to fully isolate the toolchain to make it work correctly. So we may want to start using cross for all of our musl builds.
Brendan Hansknecht (Dec 29 2024 at 18:21):Workaround is functional. This means the fundamental issue was from mixed toolchains (pulling in some stuff from gnu and some from musl). I think that we should eventually change all of our musl builds to use cross to completely rule out this class of issues.
Brendan Hansknecht (Dec 29 2024 at 18:23):Anyway, this should mean we are good to land https://github.com/roc-lang/basic-cli/pull/299 with bundled sqlite.
Brendan Hansknecht (Dec 29 2024 at 18:23):We can figure out moving to cross in a separate PR. The workaround is functional and can be used for whenever the next release is.
Brendan Hansknecht (Dec 29 2024 at 18:28):And just to post this here to perserve it, this is the fundamental part of the workaround. Ensure clang does not pull in the default glibc and instead pulls in the musl version:
CFLAGS_aarch64_unknown_linux_musl: "-nostdinc -nostdlib -isystem/usr/include/aarch64-linux-musl/"
Last updated: Jul 05 2025 at 12:14 UTC