souf (Sep 27 2025 at 10:48): souf (Sep 27 2025 at 10:48):In many mainstream languages (python, rust, js...), HTTP functionality is entirely handled in userland libraries. Developers often build different libraries around HTTP: wrappers around libcurl, full HTTP implementations, etc... Each with their own types and APIs. This leads to fragmentation: libraries rarely share a common interface.
A common problem arises when a library depends on a specific HTTP client. Consumers of the library must also include that client, and if multiple libraries use different HTTP clients, projects can end up with many HTTP clients when only one is actually required, and even desired.
If Roc provided a common HTTP interface in its standard library, libraries could target that interface instead of a specific client. This would allow users to choose their preferred HTTP client while keeping library APIs consistent.
This pattern has been successfully applied in PHP via psr-7 and psr-18. I heard that Gleam also abstracts HTTP in a way that supports multiple async paradigms, allowing libraries to run on both the BEAM and JS runtimes without changing the API.
HTTP is common enough that a standard interface could significantly reduce friction in library authoring. Roc could benefit from adopting such a pattern early, avoiding fragmentation and providing a smoother ecosystem for library authors and users alike.
I'm curious what others think about standardizing HTTP in Roc?
Richard Feldman (Sep 27 2025 at 13:12):I've thought about this! I think Request, Response, Path, and Url are all contenders for things that maybe should be in the standard library
Richard Feldman (Sep 27 2025 at 13:13):since they can all be implemented in pure Roc (and don't need special compiler integration like e.g. number types or Str) I think it makes sense for them to start out as separate, because that way we can make releases of changes to them without having to do a whole compiler release
Richard Feldman (Sep 27 2025 at 13:14):right now that distinction doesn't matter much, but it will matter more once we do an 0.1.0 release and not everyone will be on Nightly like they are today
Richard Feldman (Sep 27 2025 at 13:14):so I figure once we get to an API that seems reasonable, we can discuss putting it in the standard library
souf (Sep 27 2025 at 13:36):Maybe also a Message that can be a string, bytes (list of u8), or a stream.
Brendan Hansknecht (Sep 27 2025 at 15:15):Richard Feldman said:
I've thought about this! I think
Request,Response,Path, andUrlare all contenders for things that maybe should be in the standard library
What is your thoughts of not just types, but also effectful interfaces for common primitives? I feel like those could easily be shared and built on top of. Plus platforms could make adaptors to those interfaces if they don't directly support them
souf (May 23 2026 at 17:36):@Richard Feldman I'm back to it after a while. Has there been any work done in that direction or something alternative already? If not I'd like to start working on something here
Richard Feldman (May 23 2026 at 17:39):I don't think so!
Richard Feldman (May 23 2026 at 17:40):I'd recommend starting with Url, Request, and Response - nominal types and methods on them, and sharing here so we can discuss designs
Richard Feldman (May 23 2026 at 17:45):some things I think are important:
Richard Feldman (May 23 2026 at 17:54):a good example of tricky design questions: it probably makes sense to have a Http.Header type (and possibly also Http.Headers), because header names are supposed to be case-insensitive, they're not necessarily unique (so a Dict could lose information unless you turned the values into collections of multiple values per key), pseudo-headers are a thing, etc...
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers
souf (May 23 2026 at 17:55):@Richard Feldman I imagined to have it in a way the messaging layer is an interface we can swap for another implementation.
I'll take time to explore what other languages do, but as I mentioned above I think that the PHP community has come up with a good model for that kind of things
Richard Feldman (May 23 2026 at 17:57):I'd strongly recommend looking at Rust for inspiration
Richard Feldman (May 23 2026 at 18:00):I don't have any personal experience with PHP, but Rust has pretty much the best reputation when it comes to type-safe high performance APIs that handle edge cases :smile:
Richard Feldman (May 23 2026 at 18:03):and yeah I think the most important place to start is with those Request, Response, and Url data structures
Luke Boswell (May 31 2026 at 00:24):Related to the API design ideas... I'd love it if we found a pattern that was using Record Builders -- I feel like that is fairly unique to Roc and applicative builders are really nice to work with
Luke Boswell (May 31 2026 at 00:27):@Ian McLerran migrated the roc-lang/http package to the new compiler, and I've made a PR to add CI workflows. It will be good to roll our design ideas into that when we are ready.
I've found some compiler bugs from this which I can make issues for also which is helpful.
souf (May 31 2026 at 07:28):@Luke Boswell thank you for this! FYI, I'll send some proposal next week to start some discussions.
Luke Boswell (Jun 10 2026 at 03:02):@souf how is it going? anything I can help with?
souf (Jun 10 2026 at 06:11):@Luke Boswell thanks for the heads up! I just came back from vacation this week, I'll start looking at it in the next few days. :octopus:
souf (Jun 11 2026 at 11:05):@Luke Boswell what's the current state and plans for handling (or not) concurrency in Roc (promise/future/actor..)? Is there anything existing? Is it planed to add something?
Because http is fundamentally async it's useful to know what's the long term plan for the language.
Luke Boswell (Jun 11 2026 at 11:13):I think for the short term at least there wont be anything like that in Roc -- concurrency will be something the platforms are responsible for
souf (Jun 11 2026 at 11:19):How do you foresee cases where people start requesting the ability to do things like Future.all or Future.race - in other terms the ability to start multiple async tasks and wait for one or for all to continue?
Is it likely that it's something that comes in a few years? In which case designing the http layer in a way that is easy to migrate to an async model would help with the migration.
Bryce Miller (Jun 11 2026 at 12:58):Are the all or race functions things the platform could expose if all effects are async?
Richard Feldman (Jun 11 2026 at 13:03):I have some designs around this, but I don't think they affect Http :smile:
Richard Feldman (Jun 11 2026 at 13:03):or at least, not the "standardized HTTP interface"
souf (Jun 11 2026 at 13:21):@Richard Feldman thank you. For the http thing I was thinking at multiple layers, it might affect how we write HTTP API clients offered as a library.
souf (Jun 11 2026 at 13:22):I'm currently reviewing the state of the http things in a few relevant languages and I will post a bit later today as a starting point for this work.
Richard Feldman (Jun 11 2026 at 13:26):HTTP API clients offered as a library
I could be wrong, but I suspect this will not end up being a thing :smile:
I think http clients will end up platform-specific, and libraries which want a HTTP client will just ask for "how do I send a request?" as part of their initialization
Richard Feldman (Jun 11 2026 at 13:26):see #ideas > effects in packages
souf (Jun 11 2026 at 13:36):@Richard Feldman I think this will be a very important part of the design of how to use Roc to talk to API. For example when you want to connect to openai/anthropic API, or to AWS SDK API, or to any API that exposes a OpenAPI schema, etc...
Richard Feldman (Jun 11 2026 at 13:38):yep! and #ideas > effects in packages has more about that design :smile:
souf (Jun 11 2026 at 16:20):I'm starting this work on http for roc with some researches from other languages.
As a foreword. As a developer I have spent a lot of time trying to pick between multiple community packages that are doing the same thing. I think that a user of Roc lang should never have to ask "Which http package should I use?".
If any of you has been working for js in the early days, you would know how much of a pain it has been. You're looking for all the http packages that exist, you have to pick the best one. Two years later it is discontinued in favor of something brand new.
Javascript in the browser has suffered of having a very unconvenient http history and slowly evolving standard for async, (from callbacks, then library backed promises, then native promises), and also the platform differences from node vs the browser. Ultimately almost no http package has survived overtime and now the standard is to use the new js's fetch native api.
In the meanwhile that caused a lot of development time wasted by many library maintainers and also a lot of app developers who potentially had to migrate multiple times as the standards have evolved.
HTTP is critical. Most applications depend on it and getting it wrong has a long tail of consequences. Other languages have shown us what happens when you don't think about it early enough. Roc is starting fresh, so we have a rare opportunity to get it right from the start.
Good types alone don't prevent fragmentation though. We also need a canonical pattern for library authors, good documentation and clear guidelines. So that library authors don't reinvent the wheel and library users don't have to wonder which package to trust.
This is what I'm trying to outline with some short case studies below.
souf (Jun 11 2026 at 16:21):I took a look at the state of http of some languages that I considered worth looking at. I picked:
Rust has a layered stack:
http crate for types, hyper for low-level I/O, reqwest for ergonomic clients, tower for middleware abstraction. Each layer is solid individually. But a developer writing an API client library faces a question rust does not answer: do I hardcode reqwest? do I accept a tower::Service? do I accept a closure? do I have a feature flag for blocking vs async? The rust forum has many threads without a canonical answer. Some API libraries just hardcode reqwest internally. They are not mockable, not injectable, and they lock callers to tokio. Some others define their own http interface, some take a closure.
The base HTTP layer is solid. The http crate is a zero-dependency, pure-types package, no I/O, just Request, Response, Method, StatusCode, Uri, HeaderMap. Every major framework (axum, tonic, hyper, etc.) uses it. That part works and it works well.
Also one issue I found: Hyper normalizes all header names to lowercase causing issues in some case. There was a github issue from a user who needed to send Content-Type capitalized. As Richard noted before it should be considered.
Rust does not have any documentation of a good practice to follow to implement a HTTP API client library which causes fragmentation in the community.
One library that seems to work well (aws sdk crate) implemented its own HttpClient independently. See an example from their documentation:
let config = aws_config::defaults(BehaviorVersion::latest())
.load()
.await;
let s3 = aws_sdk_s3::Client::new(&config);
let result = s3.create_bucket()
// Set some of the inputs for the operation.
.bucket("my-bucket")
.create_bucket_configuration(
CreateBucketConfiguration::builder()
.location_constraint(aws_sdk_s3::types::BucketLocationConstraint::UsWest2)
.build()
)
// send() returns a Future that does nothing until awaited.
.send()
.await;
They use a config that they inject to a client, and the client is hiding the http layer and owns the effects.
The AWS SDK library implemented a custom interface for their http client so that users can use any of rust's http client (see .http_client(http_client)):
let http_client = Builder::new()
.tls_provider(tls::Provider::Rustls(CryptoMode::AwsLc))
.build_with_resolver(StaticResolver);
let sdk_config = aws_config::defaults(
aws_config::BehaviorVersion::latest()
)
.http_client(http_client)
.load()
.await;
The lesson: they designed a full http interface before being able to focus on the core of the library because rust does not have a good standard for it.
Additionally some thread with developers asking how to build a library for an HTTP API client in rust and the answers are generally "it depends":
As a conclusion: even a well-designed types layer doesn't prevent fragmentation at the library-authoring layer.
souf (Jun 11 2026 at 16:21):Haskell has some http types, but I mostly picked because I found an interesting post about heavy haskell company (Mercury) who stated in a recent blog post (march 30 2026) that HTTP effects need to be instrumentable to be useful.
Here is a quote from the article:
Libraries that do not do this [(records of functions pattern)] are the ones that cost us the most operational pain. At Mercury, we very rarely use web API client bindings published on Hackage. This is not because they are necessarily poorly written (some are quite good). The problem is that we cannot trust code we cannot instrument. If a third-party binding makes HTTP calls through concrete functions, we have no way to add tracing, no way to inject timeouts tuned to our SLOs, no way to simulate partner outages in testing, and no way to explain the 400ms gap in a trace except by squinting at it and developing theories. So we write our own. More work upfront, but the clients we write are observable by construction, because we built them that way from the start.
[...]
If you are writing a Haskell library, leave escape hatches. Provide records of functions, or effect types, or callbacks, or something that lets the consumer of your code inject behavior without modifying it. Haskell's type system is wonderful for enforcing constraints. But it can also, if you are not careful, seal a system so tightly that the people who have to operate it cannot see inside. The perfect abstraction, if it is operationally opaque, simply cannot be used in production.
The full article is available here: https://blog.haskell.org/a-couple-million-lines-of-haskell/ and the quote above is from the section "Designing for Introspection".
I think the most important lesson to take from this story is that http (and any costly effects) should be instrumentable in some ways.
souf (Jun 11 2026 at 16:21):Gleam offers http types without an implementation (Response, Request, Method, etc.).
That makes it easy to write code that is easy to test and reuse and also makes it possible to use the exact same types for http clients and http servers.
It's also not bound to any concurrency model, the library author just writes the types and the library users adds the concurrency model on top of it, effectively making it plugable to any adapter, the library author does not have to choose.
This is important for gleam because it has multiple targets it compiles/transpiles to (JS and Erlang VM) which have different async/concurrency models.
They have different adapters depending of the target: gleam_httpc for Erlang, gleam_fetch for JS, they use the same types but they offer different implementations for the send function. Below is an example from the gleam_httpc homepage of how to send a request:
import gleam/http/request
import gleam/http/response
import gleam/httpc
import gleam/result
pub fn send_request() {
// Prepare a HTTP request record
let assert Ok(base_req) =
request.to("https://test-api.service.hmrc.gov.uk/hello/world")
let req =
request.prepend_header(base_req, "accept", "application/vnd.hmrc.1.0+json")
// Send the HTTP request to the server
use resp <- result.try(httpc.send(req))
// We get a response record back
assert resp.status == 200
let content_type = response.get_header(resp, "content-type")
assert content_type == Ok("application/json")
assert resp.body == "{\"message\":\"Hello World\"}"
Ok(resp)
}
How they handle an http client as a library with an example from the gleam library "sturnidae" which is an http client to connect to a bank api and recommended pattern for gleam community:
import sturnidae
import gleam/httpc
pub fn main() {
// Build an API request
let request =
sturnidae.get_feed_items_request(
personal_access_token,
account_uid,
category_uid,
"2015-01-01T01:01:00.000Z",
)
// Send it with a HTTP client such as gleam_httpc
let assert Ok(response) = |> httpc.send
// Decode the response into Gleam data
let assert Ok(items) = sturnidae.get_feed_items_response(response)
}
The library has zero effects. It builds requests and decodes responses, both pure operations. The application uses they custom adapter's send. This makes the library testable: construct a Request directly and assert on its fields, or pass a custom Response to the handler, no network or mock needed.
The tradeoff is that the HTTP seam is always visible to the application author. There is no client function that sends the request directly and returns the response, such as feed_items = sturnidae.get_feed_items(token, ...). The httpc.send call lives in app code, not library code. For simple use cases this is fine. For a library that wants to hide the HTTP details entirely behind a clean domain API, this pattern does not allow it.
The lesson: They've designed something that genuinely works well with their language constraints and is easy to mock and test. It's a new language with a small community so there is not enough data to draw long term conclusions, but I think they will be able to keep community fragmentation low because they've clearly designed how HTTP API client libraries should be written.
Gleam is also somehow close to Roc in terms of effect boundaries (pure functions and multiple targets is analogous to platforms), so this pattern is worth taking a look at. The send call lives in app code, not library code, which means a library can never fully hide HTTP behind a clean domain API.
souf (Jun 11 2026 at 16:21):For elixir I'm just sharing this post: https://elixirforum.com/t/mint-vs-finch-vs-gun-vs-tesla-vs-httpoison-etc/38588
They say:
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the tools to get behind, which results in very little fragmentation.
Surely someone here has experience with the majority of these and can save newcomers multiple hours of testing/fiddling around
They couldn't escape the fragmentation despite being good at avoiding it usually.
souf (Jun 11 2026 at 16:22):I think that not everything here is relevant to Roc, because in Roc the actual HTTP implementation will be done in the platform,
but still there are good lessons to consider.
I tried to keep this post as a short research one but leaked some of my opinions in it.
It is open to discussion and reactions, notably regarding Roc's specifics around the API clients.
For the next steps I'd like to offer the following course of action:
Firstly design the http types themselves - which I think will be quite straightforward and already partially done (roc-lang/http).
- I think we can take a good look at Rust, or Haskell ones
Secondly how to write http clients that fit Roc design and constraints. (see ideas > effects in packages )
Keeping Richard's three priorities in mind: no edge cases discarded, no performance cliff that forces anyone to opt out, and ergonomics.
Also keep in mind whether record builders, as Luke raised, can help with ergonomics.
Does that ordering make sense to you, and is there anything you'd like to mention given Roc's constraints? Looking forward to your reactions and what to focus on first!
Richard Feldman (Jun 11 2026 at 16:47):yeah I'd definitely just start with the http types themselves, specifically Request, Url, and Response
souf (Jun 11 2026 at 18:00):@Richard Feldman thank you, you've given a very short reply to it, does it mean it was unnecessary to put this together?
Richard Feldman (Jun 11 2026 at 18:06):I think it's really really important to start with the data structures and figure those out, and I think the other parts will be like 1% of the work haha
Richard Feldman (Jun 11 2026 at 18:08):(at least the "standardized" part of it - each platform will have a varying amount of work to do depending on what http client design makes the most sense for that platform, e.g. browser wasm app vs web server vs CLI all have different tradeoffs in what they want)
Richard Feldman (Jun 11 2026 at 18:10):so the reason I keep emphasizing Request/Url/Response over and over is that I think that's what will take ~all of the design work, and I want to make sure it's super clear that that these are the most important things to work on!
Richard Feldman (Jun 11 2026 at 18:16):as an example, I suspect there are multiple hours of design work involved in figuring out what the right representation is for the collection of headers.
Richard Feldman (Jun 11 2026 at 18:18):those are just off the top of my head - there's a lot of sloppy API design out there (e.g. Node assumes UNIX APIs and a ton of its file I/O data structures make no sense on Windows, they assume file paths are always valid UTF-8 even though in practice that's absolutely not a safe assumption, etc.)
Richard Feldman (Jun 11 2026 at 18:19):we need to have really high standards for anything that's going to be standardized, and that means a ton of data structure design work! :smiley:
souf (Jun 11 2026 at 18:23):@Richard Feldman I think that this is quite straightforward and has been solved by many libraries in many languages.
For the headers specifically. All headers should be stored. The HTTP spec allows multiple headers with the same name, and they should be case sensitive. A good approach I have seen multiple times is a getting to get headers matching a naming that is case insensitive and one that is case sensitive to get them exactly the way it was passed to the requests
I could be wrong but I think that the base types would represent closer to the 1% work and the challenge will be designing around Roc's specific such as the platform layer and having a good standard that people will want to adopt and stick to on the long term without breaking changes
souf (Jun 11 2026 at 18:25):notably things like: does Roc handle the types only and we implement everything in the platform (reading headers etc...) or do we implement things in Roc and the platform is responsible only for the I/O?
Richard Feldman (Jun 11 2026 at 18:36):All headers should be stored. The HTTP spec allows multiple headers with the same name, and they should be case sensitive.
from https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers
In HTTP/1.X, a header is a case-insensitive name
this is the kind of thing I mean :smile:
Richard Feldman (Jun 11 2026 at 18:37):so given that the HTTP 1.x spec explicitly says headers are case-insensitive, what should we do about that?
Richard Feldman (Jun 11 2026 at 18:38):while keeping in mind that:
Richard Feldman (Jun 11 2026 at 18:40):A good approach I have seen multiple times is a getting to get headers matching a naming that is case insensitive and one that is case sensitive to get them exactly the way it was passed to the requests
this is an option, but is it a footgun? As a user of this data structure, how would I decide when to use one and when to use the other? Maybe the answer is "if I'm sending requests between two devices where I wrote the code on both ends, I know I don't need case-insensitive" but also if that's the easy option to reach for, do we silently incorrectly handle the case-insensitive case, leading to bugs in practice that could have been avoided?
Richard Feldman (Jun 11 2026 at 18:41):like my general standard for something like this is "every decision has been researched slightly past the point of diminishing returns"
Richard Feldman (Jun 11 2026 at 18:42):"here are the edge cases, here are the performance considerations, here's how others have done it, here are some problems people have encountered in practice with those approaches, here are some other options that seem plausible but don't seem to have been tried, here are the tradeoffs that seem most important" etc. etc.
souf (Jun 11 2026 at 18:49):Sorry, my assumption is that the case sensitive thing was in the specs, but practically that's right that http is flawed due to various servers going with their implementation not following the standards and we have to support them. But I believe that overall we will find our answer quite easily for each situation as it's been done many times before.
Richard Feldman (Jun 11 2026 at 18:50):well the problem is that HTTP 1.1 (still in wide use) says they're case-insensitive, but the HTTP 2 spec says when making a request or sending a response, you have to lowercase everything:
Just as in HTTP/1.x, header field names are strings of ASCII characters that are compared in a case-insensitive fashion. However, header field names MUST be converted to lowercase prior to their encoding in HTTP/2. A request or response containing uppercase header field names MUST be treated as malformed
souf (Jun 11 2026 at 18:51):Another question that comes in mind is about http methods and http status codes. Which Roc type will be a good match for them, knowing that in practice any string can be used for a http method (edge cases).
Richard Feldman (Jun 11 2026 at 18:56):so based on that case-sensitivity observation, here's one possible design direction around case-sensitivity for Request (setting aside Response for now):
some questions about that design direction:
Http2.Header type (or something) where you can just write a string literal for it like "cache-control" and it'll be converted to a Http2.Header at compile-time, but if you write the literal "Cache-Control" then you get a compile-time error. could that be a good way to enforce it? souf (Jun 11 2026 at 19:08):so let's focus on the headers first. I'll do some research around it in the next few days and try to list the possible edge cases and the solutions that different languages/libraries use to handle them.
souf (Jun 11 2026 at 19:09):Just a side question. This will be implemented in Roc directly and the platform is used only for the actual http connection and effects, right?
Richard Feldman (Jun 11 2026 at 19:34):correct!
Richard Feldman (Jun 11 2026 at 19:34):a nice source of finding out what people have hit problems with in the past is to look through (or ask an agent to) the issue trackers for popular web frameworks, both clients and servers
Richard Feldman (Jun 11 2026 at 19:35):and see what people have been bitten by in practice (perf, hash flooding attacks, bugs, footguns, edge cases, etc.) and make a list of them and see if we can find a design that learns from the mistakes of the past and avoids them :smile:
souf (Jun 13 2026 at 14:54):@Richard Feldman I was thinking about the performance point around headers. It made less sense to me at first since I was mostly thinking about HTTP clients, but I think I understand now the performance concern. It is mainly relevant for servers, which need to handle high volumes of incoming requests and may not need to inspect every header on every request.
For a client the situation is different. You're constructing the outgoing request yourself, so there's no to little parsing cost. And for the response you receive back, volume is rarely a concern. Typically you don't need to check the case precisely when reading the response. So rare edge cases shouldn't penalize common usage.
That being said, while I was looking at Rust's http and hyper crates earlier I noticed that they have a flag to preserve headers case. By default it's normalized to lower case.
The doc says:
when creating a
HeaderNamewith a string, even if upper case characters are included, when getting a string representation of theHeaderName, it will be all lower case. This allows for fasterHeaderMapcomparison operations.
As for the header case flag, it's http1_preserve_header_case and the doc says:
Set whether to support preserving original header cases.
Currently, this will record the original cases received, and store them in a private extension on theRequest. It will also look for and use such an extension in any providedResponse.
Since the relevant extension is still private, there is no way to interact with the original cases. The only effect this can have now is to forward the cases in a proxy-like fashion.
Default isfalse.
But as you can see the original cased headers are private.
It's stored in additional metadata named "extension", i.e. response.extensions().get... and the doc says:
Extensions allow Hyper to associate extra metadata or behaviors with HTTP messages, beyond the standard headers and body.
[...]
Header Case Tracking: Internal types for tracking the original casing and order of headers as received.
Any thoughts on their approach to headers case and the fact they managed to not penalize performance by making it opt-in?
souf (Jun 13 2026 at 14:58):An additional note on header case in request we send. Hyper also have a flag to force title case on headers sent in a request: https://docs.rs/hyper/latest/hyper/server/conn/http1/struct.Builder.html#method.title_case_headers
But I couldn't find anything that let them use any case of their choice.
I also found a thread on reddit in which they're looking for a library that lets them use any case for the header name in http/2: https://www.reddit.com/r/rust/comments/qz4w6s/http_libary_that_doesnt_enforce_lowercase_headers/
souf (Jun 13 2026 at 14:58):Regarding the data structure they use for storing data, it's called HeaderMap which is a multi map and the implementation differs from a HashMap and as per their documentation, the HeaderMap is using Robin Hood Hashing.
Richard Feldman (Jun 14 2026 at 03:26):very interesting!
Richard Feldman (Jun 14 2026 at 03:27):thanks for finding these!
Richard Feldman (Jun 14 2026 at 03:27):something else I'm curious about is the idea of doing it "decoder-style" where basically you treat the entire header string like you would a JSON string, and just decode into the headers you care about
Richard Feldman (Jun 14 2026 at 03:30):some tradeoffs I can think of:
Request would need to store headers as a plain Str and then you'd need to run something like .parse_headers() to get back a HeaderMap (or similar) if that's the representation you wanted Richard Feldman (Jun 14 2026 at 03:34):that idea sounds interesting in part because "I don't care about the headers and don't want to pay for turning them into a data structure" is also a valid thing to want
Richard Feldman (Jun 14 2026 at 03:36):so basically don't eagerly choose something for the end user, let Request store all the headers as a single Str internally, and then on demand you can access them in one of several ways, and then you pay the cost of turning them into that format at the time you request that format
souf (Jun 14 2026 at 07:36):@Richard Feldman Headers reader could be an ability (or to confirm what's the right approach in Roc) and produces a HeaderMap. Then the implementation can optimize for the need.
For example if you have a server that needs absolute performances, the server can define its own reader that will only extract the relevant headers it needs to? It would require to define ahead of time what headers the server will use. Not sure if it's a good idea or not, but ergonomics is not great.
What took me here is that I think the decoder/Str idea is interesting, but if you wanted absolute performances, the fact you've already stored the string from the HTTP request could still be optimized, notably you could read the headers immediately from the incoming HTTP stream and store only relevant headers rather than the full string.
So I think that locking to a Str could prevent possible future optimizations.
souf (Jun 14 2026 at 08:01):If we mimic rust's approach we have a set of opaque low level http types with no implementation, and on top we have something like hyper that will handle the actual implementation and optimizations.
If we get the base http types right, we can start implementing and optimize easily.
souf (Jun 14 2026 at 08:59):As a reference, I asked ClaudeAI to search for known optimized servers and what is their approach with dealing with headers, here is what came back as the most relevant points for our discussion:
picohttpparser / h2o (C) — the parser allocates nothing and returns only pointers/lengths into the original input buffer. No header list, map, or string is built until the caller explicitly asks for one. Used in production by h2o and several Perl PSGI servers (Plack, Starman, Starlet) — a real-world proof that a parser can do its job while building zero data structures up front.
Rust's http::HeaderMap — two details beyond "Robin Hood hashing": (a) it uses adaptive/randomized hashing specifically to resist hash-flooding/collision attacks — relevant for any hash-based header map, since an attacker can send thousands of headers crafted to collide into one bucket; (b) short header names are stored inline with no allocation, and only longer names spill to the heap — so custom/non-standard header names aren't penalized relative to common ones with a dedicated fast-path representation.
Go's net/http vs fasthttp — net/http eagerly builds a map[string][]string and converts every header from bytes to string on every request, regardless of whether the handler reads them — a real, allocation-per-header, map-insert-per-header cost paid upfront. fasthttp instead keeps the raw header bytes in a reusable buffer and exposes Peek("Header-Name"), which does a case-insensitive linear scan over those raw bytes and returns a []byte slice — no allocation, no map, at all, for headers the handler doesn't ask for. The handful of headers fasthttp's own connection-handling logic always needs (Host, Content-Length, Content-Type, Connection) are pulled into dedicated struct fields during that same single parse pass. Result is roughly a 10x throughput difference on header-heavy workloads — a useful quantified data point for why lazy vs. eager materialization is worth real design effort, though it comes with real ergonomics tradeoffs (linear scan instead of O(1) map lookup, and reused/pooled objects whose lifetime doesn't extend past the handler — a documented source of bugs when people hold onto header byte slices too long).
Node.js http module — the underlying parser (llhttp) is already zero-copy/zero-allocation, but the IncomingMessage layer eagerly builds both rawHeaders (flat array, original casing) and headers (lowercased object) for every request regardless of use. A proposal to make headers lazy (getter-based) was never merged, because by then the eager object was already part of the public API and changing it would break existing code. Cautionary example: once a convenience representation ships as public API, you can't make it lazy retroactively — worth getting that shape right before it's load-bearing.
Convergent finding — nginx's header list, Envoy's fallback map, Rust's HeaderMap, and Node's rawHeaders are four unrelated implementations across different eras/languages that all independently preserve the complete raw header set (duplicates, original order, everything), even while also providing fast paths for a known subset of headers. Decent evidence that "keep all headers, not just a curated subset" is the right baseline regardless of what fast-path optimizations sit on top.
souf (Jun 14 2026 at 10:13):I asked the agent to perform two research regarding header things we should be aware of, a first one on various issues trackers and forums, and a second one on top of it on any other resources. It's attached verbatim for review and later triage.
Richard Feldman (Jun 15 2026 at 04:10):yeah I have a proof of concept I'm working on where the ergonomics would look something like this:
# roc application specifies this for how
# the server starts up - spin up the database, etc.
init! = |env, _args| {
{ log_level, db_auth } = env.parse()?
db = init_db!(db_auth)?
Ok({ db, log_level })
}
# roc application specifies this for how to handle
# an individual request - first arg is what init!() returned,
# then path/headers/body come from the request
handle_request! = |{ db, log_level }, path, headers, body| {
auth_token = headers.x_auth_token
user_agent = headers.user_agent
# match `path` for routing, use `body` and `auth_token` etc...
}
so here, the fact that you used headers.x_auth_token and headers.user_agent would cause the headers parser to be auto-derived to look for (lowercased and translating dashes to underscores automatically) "x-auth-token" and "user-agent"
Richard Feldman (Jun 15 2026 at 04:11):and whatever other fields on headers you used would just expand the parser to look for those
Richard Feldman (Jun 15 2026 at 04:15):so in the proof of concept I did, this design has these characteristics:
headers record, and of course the headers keys are just memory offsets at runtime (since that's how Roc records always work)content-length so it knows how big the body is, and then also to find the blank line that ends the headers and validate that it was all valid utf-8; then it gives the entire headers Str to Roc, which in turn can do the header decoding in 1 pass (so, 2 passes total) now knowing that it's all valid utf-8 and where it begins and ends Richard Feldman (Jun 15 2026 at 04:17):zero allocations per request and 1-2 passes is what (from my limited research) it seems like the fastest C/C++/Rust/etc. implementations in the TechEmpower benchmarks do, like the ones that are really trying to win at the benchmark by squeezing every last drop of perf out...but this doesn't require any of that, just "use headers in a fairly natural way and the platform takes care of everything"
Richard Feldman (Jun 15 2026 at 04:18):so this seems like nice ergonomics and perf to me! Obviously people might want a more "put it in a map and I'll look them up later" kind of experience, depending on their use case, but one option is to make it so that it's easy to provide a decoder which decodes into one of those instead of into a headers record like this.
Karl (Jun 15 2026 at 04:37):the fact that you used
headers.x_auth_tokenandheaders.user_agentwould cause the headers parser to be auto-derived
I would like this feature in general for selecting fields on a query object (SQL or GraphQL) and automatically building the dependencies array in React/Solid2 reactivity primitives. I assumed it was off the table due to the general opposition to clever code.
souf (Jun 15 2026 at 06:08):@Richard Feldman unrelated question - but what's the mechanism that allows Roc to derive headers.user_agent? Is that something that exists already or something you'd like to add?
Richard Feldman (Jun 15 2026 at 10:57):we've had it since the old compiler - previously it was almost exclusively used for JSON decoding, but it can be used for other things too!
Richard Feldman (Jun 15 2026 at 10:57):I made some revisions to it for the proof of concept, but they're about performance rather than capability
Richard Feldman (Jun 15 2026 at 11:00):Karl said:
the fact that you used
headers.x_auth_tokenandheaders.user_agentwould cause the headers parser to be auto-derivedI would like this feature in general for selecting fields on a query object (SQL or GraphQL) and automatically building the dependencies array in React/Solid2 reactivity primitives. I assumed it was off the table due to the general opposition to clever code.
I think it's a good use of the language feature if it's reliable and runs fast.
I haven't thought about it for those specific use cases!
souf (Jun 15 2026 at 12:57):@Richard Feldman getting back to the implementation of http types. Would you agree that something like what Rust does would be a good fit for Roc as well? Base http types without implementation, on top of it a concrete implementation, for server/client?
Richard Feldman (Jun 15 2026 at 14:18):do you mean the HeaderMap specifically? or a different aspect of how they're doing things?
souf (Jun 15 2026 at 14:42):I mean globally, for header, request, response, uri types..
Rust has a standard http crate that is only types, with no implementation. More precisely from the http crate doc:
This crate is a general purpose library for common types found when working with the HTTP protocol. You’ll find
RequestandResponsetypes for working as either a client or a server as well as all of their components
[...]
You will notably not find an implementation of sending requests or spinning up a server in this crate. It’s intended that this crate is the “standard library” for HTTP clients and servers without dictating any particular implementation.
souf (Jun 15 2026 at 15:06):And hyper crate is the commonly used implementation of http types, but not the only one.
It helps with letting people ship their own implementation if for some reasons they have to, and also for versioning, you can version the types and the implementation separately, upgrading the implementation wont break the types.
souf (Jun 15 2026 at 15:07):(deleted)
Richard Feldman (Jun 15 2026 at 15:20):I'm not sure what the right design is yet, but I will say that this "HTTP header Decoder" proof-of-concept is making me rethink some fundamental assumptions :smile:
Richard Feldman (Jun 15 2026 at 15:22):like for example, if http verb is just an anonymous tag union like [GET, PUT, POST] etc., path is just a string, and we can ship a Decoder.http_headers as part of builtins (with settings for how you want to handle things like uppercase/lowercase, duplicates, etc.), I'm not sure how necessary it is to have a standardized Request concept
souf (Jun 15 2026 at 15:22):In terms of performances, does Roc receive performances penalty for using non concrete types?
Richard Feldman (Jun 15 2026 at 15:22):not at runtime, no
souf (Jun 15 2026 at 15:25):@Richard Feldman would you see a case where we would have a standard implementation, but some users would like to have their own more (or differently) optimized implementation, with let's say something that hits the platform instead of being implemented in Roc. For example when you read a header value?
Richard Feldman (Jun 15 2026 at 15:34):the way I think about this is:
Request (for example), it could make sense to have a standardized Request type that packages can useall that said, so far this exploration has made me less convinced that a standardized Request type is actually necessary (or a good goal) :smile:
Richard Feldman (Jun 15 2026 at 15:34):Url and Response might still be though
souf (Jun 15 2026 at 15:41):do you mean that, for a server request, you'd just have a few standalone arguments rather than a request type holding them?
souf (Jun 15 2026 at 15:54):if an application author only ever writes code against their platform and has no other dependencies, standardization doesn't really matter
A case where standardization could help - if you reuse http related tools. Such as observability, http test mocking, etc... if you comply with the standard types you gain access to all the community tools
Richard Feldman (Jun 15 2026 at 16:23):oh that doesn't require standardization - we have a design (not implemented yet) for simulating effects in the general case
souf (Jun 15 2026 at 17:11):I'm not sure to follow you.
If you have a library that takes an opaque request type. How would it be possible to use a different implementation if the types diverge for any reasons?
Also I'm not fully getting the point on simulating effects. For example if I use a middleware to transform a request, it's a pure function. Maybe we're not talking of the same thing?
souf (Jun 15 2026 at 17:26):I'm getting a bit lost in the thread, lot of ideas going around and I'm trying to figure out the concrete next step. There is already roc-lang/http with working Request/Response types, what would you say is the next concrete thing to figure out? I was thinking I could go ahead and do some concret work on the headers, but it feels like you have a fairly clear idea of where this is going and I'm not quite seeing it yet, hence my questions.
Richard Feldman (Jun 15 2026 at 17:55):souf said:
If you have a library that takes an opaque request type. How would it be possible to use a different implementation if the types diverge for any reasons?
based on this exploration I'm leaning in the direction of "libraries should not actually do this" :smile:
Richard Feldman (Jun 15 2026 at 17:56):I think maybe a good next step would be to make an actual example package that would want to do HTTP across platforms
Richard Feldman (Jun 15 2026 at 17:56):for example, something wrapping the api for https://sentry.io or https://www.bugsnag.com/ or similar
Richard Feldman (Jun 15 2026 at 18:13):the specific strategy I'd try is something like:
Bugsnag :: {
app_version : Str,
user : [Guest, SignedIn(User)],
get! : Str => { code: u16, body : Str },
post! : { path : Str, body : Str } => { code: u16, body : Str },
}.{
init : {
app_version : Str,
user : [Guest, SignedIn(User)],
get! : Str => { code: u16, body : Str },
post! : { path : Str, body : Str } => { code: u16, body : Str },
}
-> Bugsnag
init = |config| config
warn! : Bugsnag => {}
warn! = |bugsnag, warning_info| {
body = ... # turn warning_info into some json or whatever
(bugsnag.post!)({ path: "events/warn", body })
}
error! : Bugsnag => {}
error! = |bugsnag, error_info| {
body = ... # turn error_info into some json or whatever
(bugsnag.post!)({ path: "events/error", body })
}
}
and then the application using the package would initialize it like this:
bugsnag = Bugsnag.init({
app_version: "1.2.3",
user: Guest,
get!: |path| Http.send!(
GET,
"https://api.bugsnag.com/${path}",
[("X-API-Key", my_api_key)],
NoBody,
),
post!: |path, body| Http.send!(
POST,
"https://api.bugsnag.com/${path}",
[("X-API-Key", my_api_key)],
Body(body),
),
}
so this is unusual in that:
Bugsnag package is only capable of sending to api.bugsnag.com, so if an attacker compromises the package, it's impossible for them to exfiltrate data to their own serversBugsnag package never even sees my API key (I append that header myself)Http.send! coming from my platform but obviously if my platform had any other API, I could just use that instead Richard Feldman (Jun 15 2026 at 18:23):and then after that one-time setup cost, you now can use it like you'd use it in most languages, bugsnag.error!(...), bugsnag.warn!(...) etc.
souf (Jun 15 2026 at 18:27):Richard Feldman said:
post!: |path, body| Http.send!( POST, "https://api.bugsnag.com/${path}", [("X-API-Key", my_api_key)], Body(body), ),
My immediate thoughts about this approach: I get the security point and fully agree to it. However it involves the user to know what bugsnag header expects for authentication, and also involves that all post requests to bugsnag will take the header x-api-key and all have the same form. Could make upgrades of the library / api harder for the user.
As you mentioned it will require more upfront documentation for the library authors.
Their might be opportunities to improve the ergonomics while keeping the security aspect of things.
souf (Jun 15 2026 at 18:30):I think we're holding something with the security side and we should hold it firmly, but my instinct tells me that passing get and post that way is not the right final move.
souf (Jun 15 2026 at 18:31):I'll leave it to you to explore the server side of things and will look at this client thing specifically.
souf (Jun 15 2026 at 18:39):Also if I understand it correctly - the library itself will not have access to the env variable as it's an effect.
I'm aware of some api clients I have worked with in the past such as AWS or GCP that often have authentication helpers that will automatically detect the right authentication method from the env so that it can run on many places on their cloud seamlessly.
Sometimes they have complex secret management involving network round trips with multiple services mixing authenticated and unauthenticated calls.
Just an example, but for these the complexity of the configuration could become meaningful if we applied the same pattern where the user has to copy a full configuration that can drift overtime.
souf (Jun 15 2026 at 18:53):Would it be safe to pass the api key to a pure function from a library? In which case the library could simply expose pure helpers to make the authenticated request inside of the effect:
post!: |path, body| Http.send!(
POST,
"https://api.bugsnag.com/${path}",
BugSnag.pure_and_secure_helper_to_build_auth_headers('my api key'),
Body(body),
),
If that's possible, it makes the whole thing viable with possibly good ergonomics and maintainability.
souf (Jun 15 2026 at 18:59):On a side note, given the security layer you want to create by, for example, giving the ability to force hostname using a hardcoded string at the app level, I can see now that in this situation a request type would not be a good fit.
Richard Feldman (Jun 15 2026 at 19:23):souf said:
My immediate thoughts about this approach: I get the security point and fully agree to it. However it involves the user to know what bugsnag header expects for authentication, and also involves that all post requests to bugsnag will take the header
x-api-keyand all have the same form. Could make upgrades of the library / api harder for the user.
I thought about this, but I think this is actually optimal. The only way to avoid this is to have the library expose in code what URL and header to use, and that could itself be a vulnerability. :smile:
for example:
"https://api.bugsnag.com/ I do like Bugsnag.base_url then an attacker who compromises the package can just change that Bugsnag.base_url valueso my overall thinking is that docs are going to say "here is how to set this thing up" and part of that can just be "here are the strings to put in these places" - it's unusual, but honestly it doesn't seem like a serious maintenance burden to get the security benefits. Those strings are very unlikely to ever change!
Richard Feldman (Jun 15 2026 at 19:25):souf said:
Would it be safe to pass the api key to a pure function from a library? In which case the library could simply expose pure helpers to make the authenticated request inside of the effect:
post!: |path, body| Http.send!( POST, "https://api.bugsnag.com/${path}", BugSnag.pure_and_secure_helper_to_build_auth_headers('my api key'), Body(body), ),If that's possible, it makes the whole thing viable with possibly good ergonomics and maintainability.
yep, that is an option! pure functions are fine to pass sensitive data to, because all they can do is return things :smile:
Aurélien Geron (Jun 15 2026 at 19:55):Naive question: how could an attacker compromise a package without the user knowing since there's a hash in the import? It would have to be a new version, and surely someone would notice.
Also, perhaps the library author could mark some things as immutable (such as the server's API URL) and the client would be asked to confirm if anything marked as immutable ever changes? So a compromised package would only ever affect new users, and old users would see immediately that something is fishy.
souf (Jun 15 2026 at 20:04):Aurélien Geron said:
Naive question: how could an attacker compromise a package without the user knowing since there's a hash in the import? It would have to be a new version, and surely someone would notice.
it's mostly related to the more and more frequent supply chain attacks that are occurring in multiple ecosystems recently (js's npm, python's pip, etc...). The compromised packages are usually caught quickly but just a few minutes live and you already have it downloaded and executed in many environments (CI, developer workstations,etc...). Many companies have terrible (non-)security practice and don't anticipate it. Also helps when you pull a fresh new version of a package, or for some reason it was not caught by anyone for a while.
It does not prevent the attack but doing that would massively reduce that blast radius of an attack, and as a maintainer of an app I also feel more at ease upgrading my packages or installing new dependencies.
I personally run most of my local development in sandboxed environments for this reason. With this proposal implemented I would feel way safer whenever I pull dependencies from a package manager!
Richard Feldman (Jun 15 2026 at 20:16):Aurélien Geron said:
Naive question: how could an attacker compromise a package without the user knowing since there's a hash in the import? It would have to be a new version, and surely someone would notice.
it would have to be if they published a new version, yeah.
Richard Feldman (Jun 15 2026 at 20:17):but you can get new versions of something because a direct dependency of yours picked up an indirect dependency and maybe didn't do as good a job as you'd prefer of vetting its new versions :sweat_smile:
Richard Feldman (Jun 15 2026 at 20:17):one way to think of the security benefit is that you don't have to worry as much about upgrading to new versions of things if the amount of damage an attacker could possibly do is limited
Richard Feldman (Jun 15 2026 at 20:18):the other benefit is that if attackers can't benefit from compromising packages, it dramatically reduces their incentive to bother doing it in the first place
souf (Jun 15 2026 at 20:19):Attacker's mind; :"Roc is rock solid - let's not bother with it and focus on JS!"
Aurélien Geron (Jun 15 2026 at 20:26):After more thought, my "immutable" idea is stupid. Please ignore. :sweat_smile:
Richard Feldman (Jun 15 2026 at 20:34):it's a good point to bring up though! In some ecosystems that's a legit attack vector too, sadly :sweat_smile:
souf (Jun 15 2026 at 20:35):@Richard Feldman regarding the package initialization thing. I think the ergonomic is not the best, but security is something that wins the trade. Also, I suppose that overtime patterns will emerge to make it convenient, probably some directory structure conventions, and tools to help maintaining it such as templates or cli to upgrade it or whatever solution it is I'm pretty sure we can figure out to make it convenient and not something we fight. It's beyond the scope of the HTTP thing and it's for any package that owns effects, so let's not worry too much about it right now and focus on getting HTTP basics right, we can discuss this in another thread if necessary.
Richard Feldman (Jun 15 2026 at 20:36):cool, in that case I think the next step is maybe to figure out what the API should look like for sending a request
Richard Feldman (Jun 15 2026 at 20:36):like what would Http.send! look like?
Richard Feldman (Jun 15 2026 at 20:37):it doesn't need to be standardized, but it's good to think through how to cover the edge cases, how to efficiently represent headers when you're providing them as opposed to parsing them, etc.
Richard Feldman (Jun 15 2026 at 20:38):one interesting consideration here is what's available in the browser vs outside it
souf (Jun 15 2026 at 20:38):I can try with various APIs that have their specifics and see how everything fits together!
Richard Feldman (Jun 15 2026 at 20:39):e.g. imagine 3 different platforms:
Http.send! that works in both places even if that means it's a "lowest common denominator" of both Richard Feldman (Jun 15 2026 at 20:40):yeah like this game is written in roc and compiled to wasm: https://bren077s.itch.io/rocci-bird
souf (Jun 15 2026 at 20:44):@Richard Feldman is there any doc for integrating a platform? I'd like to try with an actual implementation of it
Richard Feldman (Jun 15 2026 at 21:09):not yet haha - @Luke Boswell has some examples working in the new compiler though!
Luke Boswell (Jun 15 2026 at 22:40):@souf you could take https://github.com/lukewilliamboswell/roc-platform-template-zig or https://github.com/lukewilliamboswell/roc-platform-template-rust and patch that to provide HTTP. If you need help with that I could make a branch.
Last updated: Jun 16 2026 at 16:19 UTC