Aurélien Geron (Aug 12 2026 at 21:18): Aurélien Geron (Aug 12 2026 at 21:18):The Str type has a few new functions I just noticed:
str.count_utf8_bytes() ⇔ str.to_utf8().len()str.drop_first_bytes(42) ⇔ str.to_utf8().drop_first(42).from_utf8()str.drop_last_bytes(42) ⇔ str.to_utf8().drop_last(42).from_utf8()str.with_ascii_uppercased() ⇔ str.to_utf8().map(|c| if c >= 'a' and c <= 'z' { c - 'a' + 'A' } else { c }) |> Str.from_utf8() ?? { crash "Unreachable" }str.with_ascii_lowercased() ⇔ str.to_utf8().map(|c| if c >= 'A' and c <= 'Z' { c - 'A' + 'a' } else { c }) |> Str.from_utf8() ?? { crash "Unreachable" }I'm not sure I see the benefits of these functions: the first one is longer to write, and the other four encourage poor text processing practices that lead to ASCII-only programs. Also, if we're going to have str.drop_first_bytes(42), why not also str.keep_first_bytes(42) and str.substr_bytes(12, 34)? I would prefer having an Ascii type with a nice UI (like https://github.com/Hasnep/roc-ascii/) and not touch Str.
Also, now that the Unicode 3.0.0 library is much more complete and fast, I'm wondering whether we might consider including it in the standard library, and updating Str to give it full unicode capability. After all, it won't affect compile time, binary size, or runtime performance if the user doesn't use unicode features.
If the language makes it easier to use Unicode than Ascii, people are going to write apps that handle text properly.
Richard Feldman (Aug 12 2026 at 21:38):so the ASCII stuff is specifically for things like textual programming formats like case-sensitive HTTP 1.1 headers
Richard Feldman (Aug 12 2026 at 21:39):those don't support Unicode and it would be inefficient to use Unicode parsing for that use case
Richard Feldman (Aug 12 2026 at 21:40):I think these days people are aware enough of ASCII vs unicode that just having ASCII available in the stdlib isn't going to be a footgun
Richard Feldman (Aug 12 2026 at 21:40):since they all have ASCII right in the name
Richard Feldman (Aug 12 2026 at 21:42):I think the footgun is when there are string operations that look like they'll handle Unicode correctly but actually don't, e.g. most APIs that work on code points
Richard Feldman (Aug 12 2026 at 21:42):"count utf8 bytes" might be obsolete
Richard Feldman (Aug 12 2026 at 21:43):in the old compiler it was critical for performance, because if you converted a small string to a List just to check its length, you got a heap allocation for no reason
Richard Feldman (Aug 12 2026 at 21:44):we debated a bunch whether List(U8) should have the small string representation and I think we decided that it should in the new compiler, but I don't remember whether we actually implemented it :sweat_smile:
Richard Feldman (Aug 12 2026 at 21:45):if we're doing that, then count_utf8_bytes should be removed - not just because it's unnecessary, but because I think it's actually more obvious what is being counted if you have to convert to a list of bytes (which is free if lists of bytes get the small string optimization) and then ask for its length
Aurélien Geron (Aug 12 2026 at 21:48):Should the ASCII stuff be in Str or should there be a separate Ascii type (like in the roc-ascii lib)? The advantage would be that you only need to do the UTF-8 check at the boundaries. Any operation on an Ascii would just return an Ascii, no need to return a Try.
Aurélien Geron (Aug 12 2026 at 21:49):You would also get a more complete API, without polluting the Str API.
Richard Feldman (Aug 12 2026 at 21:52):worth considering, but I'd say start an #ideas thread about it
Luke Boswell (Aug 12 2026 at 21:54):str.to_utf8().drop_first(42).from_utf8()
This feels like a smell .. and I think that is a good thing because I would be concerned if I saw that. I like that our API is not so subtly suggesting you are probably doing the wrong thing here.
Luke Boswell (Aug 12 2026 at 21:55):Wait.. are you pointing out the new functions which are equivalent. I may have missed the point
Luke Boswell (Aug 12 2026 at 21:55):the other four encourage poor text processing practices that lead to ASCII-only programs
ok I got it, ignore me
Luke Boswell (Aug 12 2026 at 21:56):I'm two sips through my morning coffee... the brain cells are still warming up
Aurélien Geron (Aug 12 2026 at 21:57):Dropping 42 bytes will probably be intended as "drop the first 42 ASCII characters" but it actually means "drop the first 42 bytes in the UTF-8 representation, and pray that it yields a proper UTF-8 string". It could remove 42 ASCII characters, or it could remove 10 emojis.
Luke Boswell (Aug 12 2026 at 22:03):I'd be open to a Str.Advanced... or something and put these scary functions under that maybe
Luke Boswell (Aug 12 2026 at 22:04):This isn't really workable though because we lose static dispatch, but maybe thats ok?
Luke Boswell (Aug 12 2026 at 22:04):We would be making this raw byte thing even less ergnomic and pushing people towards the better ASCII and Unicode package experience
Luke Boswell (Aug 12 2026 at 22:06):The other thing I've ran into and haven't made an issue for... I don't think we have a way to walk the bytes in a string without allocating a list. Like I think the only option is to convert it to a List(U8)
Aurélien Geron (Aug 12 2026 at 22:08):Richard Feldman said:
worth considering, but I'd say start an #ideas thread about it
I've started this ideas topic.
Eric Rogstad (Aug 13 2026 at 05:23):Richard Feldman said:
in the old compiler it was critical for performance, because if you converted a small string to a
Listjust to check its length, you got a heap allocation for no reason
Is this not true anymore? I thought creating a List always allocated.
Eric Rogstad (Aug 13 2026 at 05:31):A bunch of the work in this PR from a few weeks ago was based on my understanding that converting a Str that had been inlined into a List(U8) would cause a heap allocation. Is that not accurate?
Eric Rogstad (Aug 13 2026 at 05:34):Richard Feldman said:
we debated a bunch whether
List(U8)should have the small string representation and I think we decided that it should in the new compiler, but I don't remember whether we actually implemented it :sweat_smile:
Oh, I should have read this part. I think you're saying we could cause small List(U8)s to be inlined, just like small (<=23 byte) Strs are. I don't believe that's been implemented (based on my memory of all the benchmarking and allocation counting my Claude did when we were developing the above PR).
Eric Rogstad (Aug 13 2026 at 05:42):Luke Boswell said:
The other thing I've ran into and haven't made an issue for... I don't think we have a way to walk the bytes in a string without allocating a list. Like I think the only option is to convert it to a List(U8)
Should be pretty simple to add since you already added support for this via an internal-only function in this PR (sharing just in case you forgot you did that). So would just need to add a public wrapper for str_get_utf8_byte_unsafe.
Luke Boswell (Aug 13 2026 at 05:59):yeah I forgot about that... :sweat_smile:
Richard Feldman (Aug 13 2026 at 12:23):Eric Rogstad said:
A bunch of the work in this PR from a few weeks ago was based on my understanding that converting a
Strthat had been inlined into aList(U8)would cause a heap allocation. Is that not accurate?
it's accurate - I couldn't remember whether we had changed it or not in the new compiler because we'd gone back and forth on the design question, but currently lists indeed always allocate :smile:
Last updated: Sep 03 2026 at 15:16 UTC