Richard Feldman (Jan 16 2025 at 05:59): Richard Feldman (Jan 16 2025 at 05:59):I just made a repo for a platform-agnostic roc-lang/http package that basic-cli and basic-webserver can depend on, such that they no longer need publicly exposed Request and Response modules, and can instead just offer effectful functions like Http.send! (and similar) and then have some internal functions which handle translating these Request and Response types to/from host types
Richard Feldman (Jan 16 2025 at 06:00):right now I mainly wanted to share the design and get thoughts/feedback/questions on it at this point; I figure it can be productionalized later :big_smile:
Richard Feldman (Jan 16 2025 at 06:00):(productionalized in the sense of setting up the publishing workflow, docs, CI, actually using it in basic-cli and basic-webserver, etc.)
Richard Feldman (Jan 16 2025 at 06:01):unlike roc-lang/path, as far as I know this is actually ready to use right now
Richard Feldman (Jan 16 2025 at 06:02):there are zero tests, but also there's almost nothing to test :laughing:
Richard Feldman (Jan 16 2025 at 06:02):in terms of logic that is
Luke Boswell (Jan 16 2025 at 06:02):Is it just lifted from basic-cli?
Richard Feldman (Jan 16 2025 at 06:02):no, I made some API changes
Richard Feldman (Jan 16 2025 at 06:03):I explained the overall design philosophy in the readme
Richard Feldman (Jan 16 2025 at 06:03):I think backwards compatibility is important in a package like this that's likely to be depended on by a good chunk of the ecosystem
Richard Feldman (Jan 16 2025 at 06:03):and this is more focused on backwards compatibility than the current basic-cli one is
Jasper Woudenberg (Jan 16 2025 at 07:38):Is the idea that app authors would use these API's directly, or are these intended to be exclusively platform-author-facing?
If the app author is going to be using these directly, I've some concerns about how nice the documentation can be. The app author will either be making http requests and need an http client, or handling http requests and need an http server. Depending on that they need request getters and response setters or vice-versa. In this design both modules offer part of the API you need for both http clients and servers.
| | Setters | Getters |
| Request | client | server |
| Response | server | client |
I think that will make it hard to tell the story of, say, making an http request as a client, in a single place and without distraction.
Jasper Woudenberg (Jan 16 2025 at 07:50):Separately, still assuming (part of) this API is intended to be app-author-facing, I've been thinking about whether as a platform author I'd want core functionality from the platform to be provided by external libraries. For instance, the static site generator platform I'm working on has a module for constructing Html, and though we have a nice library for that available, I still decided to do a platform-custom implementation of the same, for these reasons:
By doing a custom implementation I'm able to optimize the underlying opaque Html type for how the platform consumes it.
This argument I can imagine existing around http requests/responses as well. Maybe a platform author would like the Request type internally to be a bytestring builder for instance and not allocate an intermediate data structure. Or the platform separately offering a web sockets API pushes into the design of simple http requests as well, to make both options part of a single cohesive design.
I really dislike the idea that an application author would need to go look up sepate documentation for core platform functionality. It seems at odds with the 'batteries included' approach I want to take with the platform.
elm-csslike styling functionality for instance. Kilian Vounckx (Jan 16 2025 at 08:21):Do we want a custom URI type as well? I find it really handy to be able to do something like
uri.from_str("https://example.com")?
.addPath("foo")
.addPath("bar")
.addFragment("frag")
.addQuery("name", "Frodo")
.addQuery("age", "50")
And having it be resolved to "https://example.com/foo/bar?name=Frodo&age=50#frag". I'm not even sure if the fragment should be before or after the queries, but that is exactly why I want such an api. It would also take care of url encoding. Especially nice when some parts are user input.
If we do want something like this, should it be another package?
Sam Mohr (Jan 16 2025 at 08:25):I already have some work for a URL package
Sam Mohr (Jan 16 2025 at 08:25):I can try to clean it up soon and get that published
Richard Feldman (Jan 16 2025 at 13:22):Jasper Woudenberg said:
Is the idea that app authors would use these API's directly, or are these intended to be exclusively platform-author-facing?
If the app author is going to be using these directly, I've some concerns about how nice the documentation can be. The app author will either be making http requests and need an http client, or handling http requests and need an http server. Depending on that they need request getters and response setters or vice-versa. In this design both modules offer part of the API you need for both http clients and servers.
| | Setters | Getters | | Request | client | server | | Response | server | client |I think that will make it hard to tell the story of, say, making an http request as a client, in a single place and without distraction.
I think everybody needs both for simulating HTTP roundtrips in testing. Even if I'm only using a client, I still want to be able to write tests which simulate a server making a certain response (e.g. a 500 error or a 403 vs 401), which in turn means I need access to both. Similarly, if I'm writing a server, I want to be able to simulate certain shapes of requests coming in! :big_smile:
Richard Feldman (Jan 16 2025 at 13:23):Jasper Woudenberg said:
Separately, still assuming (part of) this API is intended to be app-author-facing, I've been thinking about whether as a platform author I'd want core functionality from the platform to be provided by external libraries. For instance, the static site generator platform I'm working on has a module for constructing
Html, and though we have a nice library for that available, I still decided to do a platform-custom implementation of the same [...]
this is an interesting tradeoff! I think the main question here is about code sharing across the ecosystem
Richard Feldman (Jan 16 2025 at 13:59):so for example, let's say there's a platform-agnostic html library
Richard Feldman (Jan 16 2025 at 14:00):and all it does is to basically define a Node type where each node is either text or a list of child nodes and attributes
Richard Feldman (Jan 16 2025 at 14:00):in other words, essentially the "builder" part of things
Richard Feldman (Jan 16 2025 at 14:01):I could be wrong, but I don't think any platform would benefit significantly from using a different data structure to represent HTML
Richard Feldman (Jan 16 2025 at 14:02):so if that package exists, then anyone can build reusable platform-agnostic html packages with it
Richard Feldman (Jan 16 2025 at 14:04):but at the same time, any platform can also offer its own batteries-included API for actually building and using these things, including exposing a type alias to Node so users of the API don't necessarily even need to realize that type is defined in another package
Richard Feldman (Jan 16 2025 at 14:08):so at that point, the delta between doing absolutely everything in the platform and using the platform-agnostic html package becomes:
Richard Feldman (Jan 16 2025 at 14:10):to me this seems like a pretty minor cost to get the ecosystem benefit, but I think it's also reasonable for a platform to go the other way and prioritize having fewer dependencies over ecosystem code sharing
Richard Feldman (Jan 16 2025 at 14:14):the pattern between this example and http is that the platform-agnostic package is basically just "here's the common data we need to share if we're going to interoperate"
Richard Feldman (Jan 16 2025 at 14:15):so headers and urls and whatnot in the case of http, and node tree structure in the case of html
Richard Feldman (Jan 16 2025 at 14:16):but everything else, including the APIs for how to assemble and manipulate that data, can be customized as much as desired without sacrificing interoperability!
Anthony Bullard (Jan 16 2025 at 15:19):I think Platforms as a concept is unworkable at scale without packages like these coming from roc-lang and developed by the community. And the community coalescing around them
Anthony Bullard (Jan 16 2025 at 15:20):Or I’d want platforms to expose way more low level abstractions
Jasper Woudenberg (Jan 16 2025 at 16:46):Richard Feldman said:
I think everybody needs both for simulating HTTP roundtrips in testing. Even if I'm only using a client, I still want to be able to write tests which simulate a server making a certain response (e.g. a 500 error or a 403 vs 401), which in turn means I need access to both. Similarly, if I'm writing a server, I want to be able to simulate certain shapes of requests coming in!
Fair point, I hadn't considered the testing angle!
I'm still wondering if for my ideal Roc Http client I wouldn't want the docs to be written, not in terms of generic setters and getters, but more like "here are funcctions for making an http request, and here's some helpers testing that logic".
Separately, but this might be more of a static dispatch thing, if I'm writing a platform with support for sending http requests, I can't really see myself being happy with an API that requires multiple import statements to use, and/or of which the documentation is spread across >=3 separate modules. Maybe that's a matter of getting used to SD more, but I really think I'd want to tweak that a bit more to try make it nicer.
Jasper Woudenberg (Jan 16 2025 at 16:49):What I have an easier time imagining is writing a platform that provides custom Request/Response APIs, but offers support for external middlewares using the shared Request/Response types, a bit like the role wai plays in the Haskell ecosystem. I might do something like that if that hooked into an existing ecosystem of middlewares. That said, such middlewares almost certainly need to perform effects, so it might end up being integrated in the platform after all :shrug:.
Jasper Woudenberg (Jan 16 2025 at 16:49):Richard Feldman said:
but at the same time, any platform can also offer its own batteries-included API for actually building and using these things, including exposing a type alias to
Nodeso users of the API don't necessarily even need to realize that type is defined in another package
If as a platform owner you'd like your own API for a type to be the default one, I don't think you have a choice but to define your own opaque type. A type alias is going to get it's methods from the module that defines it, and any API you layer on top is going to be second-class.
Jasper Woudenberg (Jan 16 2025 at 16:53):Richard Feldman said:
I could be wrong, but I don't think any platform would benefit significantly from using a different data structure to represent HTML
I have a custom Html type like that for my static site generator platform currently. It looks like this:
Xml : List
[
FromSource SourceLoc,
RocGenerated (List U8),
]
SourceLoc : {
start : U32,
end : U32,
}
This being static site generation, the output HTML might be a mix of Html generated by a Roc Html.* module, mixed with slices from .html source files, mixed with Html output from libcmark. The above type allows me to mix raw Html from other sources into snippets generated from Roc without having to parse it first. Less importantly, the relatively flat representation makes it a relatively easy type to pass between host and platform.
Jasper Woudenberg (Jan 16 2025 at 16:55):Anthony Bullard said:
I think Platforms as a concept is unworkable at scale without packages like these coming from roc-lang and developed by the community. And the community coalescing around them
My current perception of this type of code sharing is that it makes the platform authoring experience a bit nicer at the expense of making the app authoring experience a bit less nice. I think it's a tricky pitch, because I imagine the motivation for many platform authors is the exact opposite: to do the hard bits in the platform and create a super nice application writing experience. It is for me!
A type of code-sharing I can see working better for me, is one where I depend on some combination of a C-dependency with a Roc wrapper, and maybe I vendor that into the platform project.
All that said, the existence doesn't block a platform author from taking a different approach and I'm not against it existing. But given the goal of a shared http package like this might in part be to try for consistency across platforms I wanted to raise some obstacles I see to using a package like this in my own platform work.
Richard Feldman (Jan 16 2025 at 17:09):good points! Another possible option is to have the platform do its own thing in terms of types, but then offer a conversion function to let you convert to/from another representation.
that would make code sharing still possible, just less convenient for the application author due to needing to do the conversion explicitly instead of having the shared code just drop right in
Richard Feldman (Jan 16 2025 at 17:09):but maybe for some platforms that's the right tradeoff!
Richard Feldman (Jan 16 2025 at 17:13):btw I also think regarding the ecosystem as a whole, it's both valuable to have a high-quality obvious default choice and yet also have it be possible for people to use alternatives with different tradeoffs
Richard Feldman (Jan 16 2025 at 17:14):without the high-quality obvious default choice, you get the kind of fragmentation you see in the npm ecosystem where there are all these competing standards for how to do basic things
Richard Feldman (Jan 16 2025 at 17:14):but without the ability for people to choose alternatives, whole categories of platform use cases (which are designed to be specializable!) would be ruled out
Richard Feldman (Jan 16 2025 at 17:16):so I think the ideal distribution of how people use things is that there's this massive concentration of usage around the default, and then a ton of tiny niche bespoke things, none of which are anywhere remotely near the usage of the default thing on their own - so they're not competing standards, but rather intentionally special-case alternatives
Richard Feldman (Jan 16 2025 at 17:17):as opposed to a concentration of several large competing alternatives, and then debates rage on endlessly about which http package is slightly better than the others, package authors have to decide which one they want to depend on, which makes the fragmentation spread further, etc. etc.
Jasper Woudenberg (Jan 16 2025 at 17:59):Yeah, that's fair!
I think this might also come down to some platforms being super-opinionated and intentionally restricted to serve some narrow use cases really well, and more general less opinionated platforms that leave more open to the application author. Both are worth supporting, and they'll have different needs from the wider Roc ecosystem.
Last updated: Jul 26 2025 at 12:14 UTC