Aurélien Geron (Aug 12 2026 at 22:06): Aurélien Geron (Aug 12 2026 at 22:06):The Str type currently contains a couple functions that are clearly meant for ASCII strings: with_ascii_uppercased and with_ascii_lowercased. It also contains a few more functions that programmers will likely use for ASCII text: drop_first_bytes, drop_last_bytes, and count_utf8_bytes.
I propose to create a new Ascii type (similar to the one in the roc-ascii library) and add it to the standard library. Here are some benefits:
Ascii, you can manipulate it safely, there's no risk of UTF-8 conversion error, most functions can return Ascii instead of Try.str to be an ASCII string might write str.drop_first_bytes(42)? thinking that it will remove the first 42 ASCII characters, but in fact it removes the first 42 bytes, so it could have removed just 10 emojis, for example. This is unsafe. With Ascii, the problem would be caught when calling Ascii.from_str. Moving all ASCII-only functions to Ascii would avoid the risk of people using ASCII-only functions on non-ASCII text.Ascii could have a richer API, without polluting Str. Luke Boswell (Aug 12 2026 at 22:08):Why a builtin and not use roc-ascii if that has the type and helpers already?
Aurélien Geron (Aug 12 2026 at 22:11):Great question. I feel like having some really basic ASCII features in the language would be nice, but it's not a strong opinion. However, if we don't include it in the standard lib, I think we should still remove drop_first_bytes, drop_last_bytes, and count_utf8_bytes: I fear that they will do more bad than good.
Luke Boswell (Aug 12 2026 at 22:12):Theres a few interrelated things here I guess. One advantage of not having these types in the builtins is that packages can innovate without being coupled to Roc releases.
Aurélien Geron (Aug 12 2026 at 22:14):True. I suppose it's the batteries-included debate: how far do we go? Regarding Ascii text, I would argue that it feels (to me) like a fairly basic thing that a language should have, and I'm not sure there would be a huge amount of innovation on basic ASCII text processing.
Luke Boswell (Aug 12 2026 at 22:16):I totally agree regarding the benefits of an Ascii type, and even a Unicode type.
Historically when we discussed where the line should be for builtins it was a pretty hard minimalistic approach (to avoid coupling Roc to external standards that may change I think was the main driver), but that appears to have softened over time. We even have Json in the builtins now...
Aurélien Geron (Aug 12 2026 at 22:18):Exactly! Having Json.parse but not Ascii.substr feels a bit odd.
Luke Boswell (Aug 12 2026 at 22:20):I would be open to reviving roc-json and taking Json out of the builtins...
Over time we may find more and more things that could be in the builtins, or maybe we wont because the domain of programming language primitives has been thoroughly explored and matured over the years so you can just look across the pond at every other mature ecosystem and cherry pick the best parts.
Aurélien Geron (Aug 12 2026 at 22:21):Yeah, I'm fine either way. My main point is: if Str.drop_first_bytes is available easily but not Ascii, people will abuse it.
Luke Boswell (Aug 12 2026 at 22:40):I mentioned this in the other thread so adding here to this discussion
Basically we could approach the problem the other direction and think about ways tomake the Str.drop_first_bytes less ergonomic so there is more incentive to reach for a proper library and type like roc-ascii or unicode... ideas in this direction could be move these to another type like Str.Advanced then they don't participate in static dispatch and you have to really reach for them if you need this functionality. If you are a package author then you will probably be thinking more deeply about the trade-offs and these are likely rare functions to reach for.
Jasper Woudenberg (Aug 12 2026 at 22:48):Are there many downsides to using List(U8) as a light-weight ASCII representation? I've been doing that for parsing work and personally think it's fine: all the functionality you might need is available on the List type.
Jasper Woudenberg (Aug 12 2026 at 22:56):I guess to_lower and to_upper are the ones you don't get on List :thinking:. Tricky if it's just those two.
Luke Boswell (Aug 12 2026 at 23:04):This might be crazy talk ... but what if we shipped a standard library separate from the builtins?
Richard Feldman (Aug 12 2026 at 23:05):I don't want to do that :smile:
Luke Boswell (Aug 12 2026 at 23:05):The criteria for what lives in the std lib could then be relaxed a little to accomodate things like Json.parse and Ascii etc
Richard Feldman (Aug 12 2026 at 23:07):Jasper Woudenberg said:
Are there many downsides to using
List(U8)as a light-weight ASCII representation? I've been doing that for parsing work and personally think it's fine: all the functionality you might need is available on theListtype.
currently the downside is that we don't (I just checked) use the small string representation for anything other than Str
Luke Boswell (Aug 12 2026 at 23:07):I think Re Json specifically... it's in the builtins not because of ergonomics but because in future it will use SIMD under the hood to make it very fast. But now we have Buitlin SIMD primitives maybe Json doesn't need to be in the builtins anymore
Richard Feldman (Aug 12 2026 at 23:08):so if you convert from "abc" (small string) to List(U8), you get a heap allocation when it goes from Str to List
Richard Feldman (Aug 12 2026 at 23:08):I think ideally we would make that not happen, although that has the downside that the more typical use of List(U8) gets less efficient because it has to deal with the small-string conditional
Luke Boswell (Aug 12 2026 at 23:09):I had an LLM scan through all our previous builtin related discussions and then reduce that down to some enduring principles here https://gist.github.com/lukewilliamboswell/96c61cc71de813f6ea53f04ead81c35f
I'm not sure we completely endorse this, but it's food for thought.
Richard Feldman (Aug 12 2026 at 23:10):one idea I had was to make a Str.AsciiByte type, and then we let you convert from Str to List(AsciiByte), and have that use the small string optimization
Richard Feldman (Aug 12 2026 at 23:11):and AsciiByte is a custom number type (so number literals etc work) that goes from 0 to 127
Luke Boswell (Aug 12 2026 at 23:13):Will we consider a small List optimization?
Richard Feldman (Aug 12 2026 at 23:14):we can, but:
Richard Feldman said:
that has the downside that the more typical use of
List(U8)gets less efficient because it has to deal with the small-string conditional
Richard Feldman (Aug 12 2026 at 23:15):other than strings specifically, basically all other use cases of List(U8) will not be that small
Richard Feldman (Aug 12 2026 at 23:16):so if we do it always, it speeds up strings at the cost of making all other byte operations slower
Richard Feldman (Aug 12 2026 at 23:20):so an advantage of the List(AsciiByte) idea is that we can do the optimization only for that exact List type
Luke Boswell (Aug 12 2026 at 23:21):It feels like there are two related problems here we are talking about...
Str.drop_first_bytes being too easy to misuse Richard Feldman (Aug 12 2026 at 23:23):so it's an absolute requirement that we do one of the following:
Str to List without a heap allocation (e.g. the List(AsciiByte) ideaStr" etc. without requiring a heap allocation Richard Feldman (Aug 12 2026 at 23:23):we can't have it be the case that getting the byte length of a Str requires a heap allocation; that's just not an acceptable API
Richard Feldman (Aug 12 2026 at 23:24):so either we need the methods or we need the conversion to List to be free
Luke Boswell (Aug 12 2026 at 23:31):It sounds like in principle you are in favour of adding an Ascii "something" to the builtins -- based on the grounds that we can optimise that under the hood to make it efficient.
Luke Boswell (Aug 12 2026 at 23:32):Does Str.AsciiByte have any advantages over an Ascii type which doesn't use List(U8) but can be free to use builtin or efficient zig implementations?
Luke Boswell (Aug 12 2026 at 23:35):Unlike Unicode, Ascii is very mature and not something that will ever change in the future
Luke Boswell (Aug 12 2026 at 23:39):I remember talking about wanting a pit of success to to encourage people to reach for the right tool which is a mature Unicode package, and not build software assuming the world is Ascii just because that is easier.
Richard Feldman (Aug 12 2026 at 23:57):Luke Boswell said:
Does
Str.AsciiBytehave any advantages over anAsciitype which doesn't useList(U8)but can be free to use builtin or efficient zig implementations?
assuming Ascii means something like AsciiStr, the difference is that validating that a string contains only valid ASCII requires converting the entire string - so you can't cheaply do things like (for example) "verify that the first byte is an ASCII uppercase letter, even if the rest of the string might contain non-ASCII content" and you also can't convert from Str to AsciiStr without walking the entire string, whereas converting from Str to List(AsciiByte) can be a no-op
Richard Feldman (Aug 12 2026 at 23:58):Luke Boswell said:
I remember talking about wanting a pit of success to to encourage people to reach for the right tool which is a mature Unicode package, and not build software assuming the world is Ascii just because that is easier.
I think it's important to not have bad Unicode primitives, and I also don't think Unicode belongs in builtins, but I also don't think it's optimal to (for example) not be able to support ASCII camelCase <-> snake_case conversions in the builtin Json module because builtins don't have a concept of ASCII uppercase/lowercsae
Richard Feldman (Aug 12 2026 at 23:58):same with ASCII-only case-insensitive HTTP headers
Richard Feldman (Aug 13 2026 at 00:00):like I said in #beginners > ASCII and Unicode, I'm not super worried about people writing user-facing applications reaching for something with ASCII in the name to handle user-supplied input.
Richard Feldman (Aug 13 2026 at 03:40):I just realized - it would definitely have to be List(Utf8Byte) if it's going to be free to convert from Str, because of course those are not necessarily all ASCII bytes in there :joy:
Richard Feldman (Aug 13 2026 at 03:40):so List(AsciiByte) wouldn't make sense after all
Richard Feldman (Aug 13 2026 at 12:32):the main thing that's interesting to me about something like List(Utf8Byte) is that whenever I'm working on low-level string stuff and I want to be operating in terms of bytes, the natural way to do that is with a List of bytes - but that costs an unnecessary heap allocation which will likely dwarf the cost of everything else I'm doing with the string put together.
If I know about the heap allocation concern, I can carefully avoid the performance footgun by using all these special-purpose Str methods instead...but then not only are the ergonomics worse, anyone who doesn't know about the (nonobvious, unfortunately) footgun is going to end up with a bunch of List-based code that needs to be rewritten
Richard Feldman (Aug 13 2026 at 12:39):so that feels like the "pit of success" API for when you actually do want to be working on UTF-8 bytes
Richard Feldman (Aug 13 2026 at 12:41):if we wanted to take it further, we could also have like:
Utf8Byte.to_ascii_char : Utf8Byte -> Try(AsciiChar, [NonAscii])
but since Utf8Byte and AsciiChar would have from_numeral, and single-quote literals are just number literals that call from_numeral at compile time, you could do this:
my_utf8_byte == 'a'
my_ascii_char == 'a'
so it's not like you'd need to convert to AsciiChar to to ASCII comparisons
Eric Rogstad (Aug 13 2026 at 16:57):Richard Feldman said:
If I know about the heap allocation concern, I can carefully avoid the performance footgun by using all these special-purpose
Strmethods instead...but then not only are the ergonomics worse, anyone who doesn't know about the (nonobvious, unfortunately) footgun is going to end up with a bunch ofList-based code that needs to be rewritten
Don't most languages let you index into strings without converting to a list? I wouldn't expect people to convertStr into a List(U8) by default.
If we provide safe methods for working with the bytes of a string attached to Str, I would just expect folks to use those. And the only reason we'd end up with a bunch of list-based code would be if we don't provide those methods on Str.
Does that not seem right?
Richard Feldman (Aug 13 2026 at 17:01):Eric Rogstad said:
Richard Feldman said:
If I know about the heap allocation concern, I can carefully avoid the performance footgun by using all these special-purpose
Strmethods instead...but then not only are the ergonomics worse, anyone who doesn't know about the (nonobvious, unfortunately) footgun is going to end up with a bunch ofList-based code that needs to be rewrittenDon't most languages let you index into strings without converting to a list? I wouldn't expect people to convert
Strinto aList(U8)by default.
yes, and personally I think they have all been mistaken to offer that because people assume it means "index by character" and then end up with Unicode bugs when it turns out to be a different definition of "character" than the one they had in mind
Richard Feldman (Aug 13 2026 at 17:02):e.g. it indexes into bytes or code points, when they assumed it was extended grapheme clusters, and their code works right up until it encounters emojis
Richard Feldman (Aug 13 2026 at 17:03):or it indexes on extended grapheme clusters (very rarely done), in which case it runs much more slowly than they might assume because it requires a linear scan of the entire string up to that point
Richard Feldman (Aug 13 2026 at 17:06):and then that latter approach can also run into new versions of Unicode coming out, which can have breaking changes, so it's not great if the language's stdlib has to be pinned to one specific Unicode version and upgrading otherwise minor version bumps of the language is potentially coupled to bringing in breaking changes to your Unicode handling
Richard Feldman (Aug 13 2026 at 17:08):we can add methods on Str for operating directly on the underlying bytes, but I think it's very important to put in the names of those methods what they're operating on - which is why we have Str.count_utf8_bytes and emphatically not Str.len - because I think naming it len would be the same footgun that it has been in practice in other languages
Eric Rogstad (Aug 13 2026 at 17:15):Richard Feldman said:
we can add methods on
Strfor operating directly on the underlying bytes, but I think it's very important to put in the names of those methods what they're operating on - which is why we haveStr.count_utf8_bytesand emphatically notStr.len- because I think naming it len would be the same footgun that it has been in practice in other languages
Yeah I think this seems like the right approach. Users will reach for methods on Str to do whatever they're trying to do, and then they'll learn from the method names (and secondarily from their doc strings) whether they're operating on bytes or characters.
So I think what we're already doing makes sense. I'm just trying to (lightly) push against the idea of taking those Str methods away and making you operate on List(U8) instead.
Richard Feldman (Aug 13 2026 at 17:17):yeah, both approaches are reasonable imo
Richard Feldman (Aug 13 2026 at 17:18):the problem with the current approach is that it's more ergonomic in many cases to work on a List, but it's not obvious that converting from Str to List is sometimes free and other times expensive
Eric Rogstad (Aug 13 2026 at 17:18):I guess I'm glossing over the fact that there's a Unicode library that presumably is the thing that lets you get a character-based length of a string. I haven't touched it yet, but I guess people are mostly supposed to be using that.
Richard Feldman (Aug 13 2026 at 17:18):and we could make a pit of success by making it so that converting to the list was just always free, and actually becomes the only way to do it
Richard Feldman (Aug 13 2026 at 17:19):well it depends on what you're doing - like for example if you're trying to validate password length to see if it will fit in a database column, the database is probably storing it in bytes, not characters. Same with SMS messages having a "character" limit - it's actually a byte limit.
Richard Feldman (Aug 13 2026 at 17:20):so I think the best we can do is to never use vague terms like "character" or "index" and always be very explicit about what units we're dealing with
Richard Feldman (Aug 13 2026 at 17:21):because you legitimately sometimes want one unit and other times want the other, so there's no escaping the requirement that people know which one to use when
Eric Rogstad (Aug 13 2026 at 17:21):Hmm, what's a case where you don't want the individual byte-based view? Just when you're trying to render the text?
Richard Feldman (Aug 13 2026 at 17:21):but we can help by trying to prevent the situation where they think they've got one type of unit but actually they have another, and they won't realize it in common test scenarios because the difference only causes bugs in edge cases
Richard Feldman (Aug 13 2026 at 17:22):rendering is one case, and another is when you're trying to do like a "character count" in a word processor or something like that
Richard Feldman (Aug 13 2026 at 17:23):or if you have a cap for conciseness and not storage, e.g. "Description: (500 chars max)"
Eric Rogstad (Aug 13 2026 at 17:27):Hmm, so in that kind of case, I (as a hypothetical novice Roc'er) will reach for Str.len. It won't exist, so then maybe I'll stumble upon Str.count_utf8_bytes, and wonder whether it's what I want. But what would make me realize that what I actually need is a function from a separate Unicode library?
(Or actually, in the conciseness cap case, I could probably live with just capping at 500 bytes. I mean do I actually care that I'm capping at fewer than 500 unicode characters in some cases? Maybe I've already backed with this with a DB field that's limited to 500 bytes. So maybe users will just reach for Str.count_utf8_bytes and then be done.)
Eric Rogstad (Aug 13 2026 at 17:28):I guess rendering is the clearer case, where capping at X bytes (rather than X characters) is truly a bug.
Eric Rogstad (Aug 13 2026 at 17:30):Maybe there should be a Str.len entry in the docs, but it just says:
Str.count_utf8_bytesUnicode.XXX Richard Feldman (Aug 13 2026 at 18:58):yeah I've thought about things like that in the past
Richard Feldman (Aug 13 2026 at 18:59):like "fake docs entries" that maybe have a strikethrough in them
Aurélien Geron (Aug 13 2026 at 19:25):"Unicode characters" could mean codepoint or grapheme. I think we should avoid the word "character".
Eric Rogstad (Aug 13 2026 at 20:24):Would it be crazy to have three families of functions on Str?
They would operate on:
And if we're worried about the Unicode spec evolving, you could have to pass a Unicode object to #2 or #3.
Then if I think I want a character count, I see Str.count_utf8_bytes, Str.count_unicode_codepoints, and Str.count_unicode_graphemes, and then I have to decide which one I want. Having to make that decision is some up-front cost, but it sounds like we're saying we want to force that decision onto the user to prevent bugs.
To me, this seems like it would be much more discoverable than just having the byte-based versions on Str, and hoping that users find the Unicode stuff separately.
Eric Rogstad (Aug 13 2026 at 20:26):Or alternatively, you could have just a single function like Str.len, but have it take an enum describing the different kinds of "length" that you might mean. So you'd pass in either Utf8Byte or UnicodeCodepoint or UnicodeGrapheme.
Edit: retracting this idea, since my Claude points out that they'd have different runtimes (O(1) for bytes, and O(n) for others), which would be weird for one function with different modes.
Anton (Aug 14 2026 at 14:17):Maybe there should be a
Str.lenentry in the docs, but it just says:
We used to have this in Roc alpha4: Str.len : Str -> [LearnAboutStringsInRoc Str]
Anton (Aug 14 2026 at 14:18):And the docs for that function told you what to do in which situation.
Anton (Aug 14 2026 at 14:18):I don't know why it was dropped when porting builtins to the zig compiler.
Luke Boswell (Aug 14 2026 at 19:14):We havent finished porting everything across, we can add this and the docs I guess I wasnt tracking it in a git issue
Kasper Møller Andersen (Aug 15 2026 at 06:48):Tangential to this, when we were discussing sorting, we reached the conclusion that having something like a Bytes/ByteString/list of bytes type would be best for the sorting API. @Jonathan summed it up here:
Anton (Aug 17 2026 at 14:34):Luke Boswell said:
We havent finished porting everything across, we can add this and the docs I guess I wasnt tracking it in a git issue
I will add it once GitHub works again
Last updated: Sep 03 2026 at 15:16 UTC