Stream: ideas

Topic: roc patch

Kiryl Dziamura (Jun 25 2025 at 09:03):

I want to propose a package patching/vendoring functionality built-in to the compiler.

what

package overloading functiobality that allows managing local changes of dependencies, including full vendoring

why

1. a bug somewhere deep in dependencies. sometimes you need to just make it work not waiting for a new release (and a release of depended package)
2. ability to modify internal packages for debugging

how

Introduce new commands

• roc patch <package url> - fetches package contents to ./patches/<package>
• roc patch collapse [package] - transforms the modifications to a diff file ./patches/<package>.diff

during compilation, roc either resolves imports from the local patched folder, or creates a new in a temp dir with applied diff and resolves from there

patches then possible to add to the index

uncertainties

• nested package overloads: how to manage them?
• should the patching be allowed only in applications?
• patching introduces implicit indirection

alternatives

• modify cached dependencies in the local roc registry: while it's good enough for debugging, it doesn't help with vendoring and also it invalidates cache for other applications

similar tools

https://pnpm.io/cli/patch

Richard Feldman (Jun 25 2025 at 11:34):

I think it would be good to make vendoring easy, which would cover this use case too

Richard Feldman (Jun 25 2025 at 11:36):

vendoring would be useful not just for this use case, but also for having a quick way to vendor all of your deps if that's your preferred way to manage deps (which I think is a reasonable way to choose to do it, and which I'd like to be nicely supported)

Richard Feldman (Jun 25 2025 at 11:37):

because version resolution will be done based on version number in a URL, I think we'd need something like an as to make it so that a local dep can be treated as a URL in package resolution

Richard Feldman (Jun 25 2025 at 11:38):

for example:

json: "vendor/json/main.roc" as "https://..."

Kiryl Dziamura (Jun 25 2025 at 11:39):

roc patch collapse is an optional step in my proposal. when I want to fully vendor a dependency - add the patch folder to the index. if you want to keep on track with the amount of changes you introduced to the external dep - use roc patch collapse

but I agree, roc vendor seems more reasonable name for it

Kiryl Dziamura (Jun 25 2025 at 11:40):

because version resolution will be done based on version number in a URL, I think we'd need something like an as to make it so that a local dep can be treated as a URL in package resolution

how to overload deeply nested dependency this way?

Richard Feldman (Jun 25 2025 at 13:16):

so this is tricky because of security

Richard Feldman (Jun 25 2025 at 13:19):

the best answer I've thought of is that if I do this:

json: "vendor/json/main.roc" as "https://.../roc-json/1.2.3/..."

...it means that I am saying "I am going to use this as my dependency, and also I am requiring that all of my dependencies use version 1.2.3, and if they need a 1.x.y version that's higher than 1.2.3, fail the build"

Richard Feldman (Jun 25 2025 at 13:20):

if I do this in a package, then that package is not participating in the roc-json version 1.x.y version resolution at all; it's as if the entire package has vendored that dependency, and all of its transitive deps are using the vendored one

Richard Feldman (Jun 25 2025 at 13:21):

this is important for security; I don't want to depend on package foo, and then foo "vendors" roc-json to do something malicious, and now that affects my application

Anthony Bullard (Jun 25 2025 at 13:21):

Can't you do vendoring pretty easy today by using local paths?

Richard Feldman (Jun 25 2025 at 13:21):

if I'm depending on foo, it's absolutely critical that the only surface area of that I need to audit are its entrypoints

Richard Feldman (Jun 25 2025 at 13:21):

Anthony Bullard said:

Can't you do vendoring pretty easy today by using local paths?

today yes, but that's only because we don't support version resolution today

Richard Feldman (Jun 25 2025 at 13:22):

like for example if I have dependencies which depend on different versions of roc-json (e.g. one of my deps is using version 1.2.3, and another one needs 1.4.5, and another one needs 1.6.7, which means in practice they'll all end up using 1.6.7), and I want to vendor roc-json to override it for all of my deps...

Richard Feldman (Jun 25 2025 at 13:23):

that needs more special care than just adding a local path, because by default my dependencies don't know that local path exists; they just know about the URLs of their dependencies etc.

Richard Feldman (Jun 25 2025 at 13:23):

that's the problem the as "https://..." solves

Anthony Bullard (Jun 25 2025 at 13:24):

so that basically is a remap of the url for a package of that name?

Anthony Bullard (Jun 25 2025 at 13:24):

and affects all packages recursively?

Richard Feldman (Jun 25 2025 at 13:24):

pretty much yeah

Richard Feldman (Jun 25 2025 at 13:25):

again there's the important distinction of "recursively for me and my dependencies, but not anyone who is depending on me"

Richard Feldman (Jun 25 2025 at 13:25):

for security etc

Anthony Bullard (Jun 25 2025 at 13:25):

yeah the root of a tree can determine the url for any package in its subtree

Richard Feldman (Jun 25 2025 at 13:26):

right

Anthony Bullard (Jun 25 2025 at 13:26):

and therefore if it becomes a subtree in another package or app, that consumer gets control

Richard Feldman (Jun 25 2025 at 13:26):

right

Anthony Bullard (Jun 25 2025 at 13:26):

Consumer's Rights

Richard Feldman (Jun 25 2025 at 13:26):

one way to think of it is that as basically says "I'm vendoring it, and that means everything I depend on is also using my vendored version"

Richard Feldman (Jun 25 2025 at 13:27):

it's a relevant distinction though, because if I'm not using as, then my dependencies can affect things that depend on me

Richard Feldman (Jun 25 2025 at 13:27):

for example, let's say I depend on roc-bugsnag which depends on roc-json version 1.2.3

Richard Feldman (Jun 25 2025 at 13:27):

and then I have another package which depends on roc-bugsnag and also roc-json version 1.0.0

Anthony Bullard (Jun 25 2025 at 13:28):

And it's now on you to resolve any compiler issues they may arise from your vendored version

Richard Feldman (Jun 25 2025 at 13:28):

normally, that other package will end up with roc-json 1.2.3, even though it only said it needs 1.0.0, because it depends on roc-bugsnag which depends on 1.2.3

Richard Feldman (Jun 25 2025 at 13:28):

however, if roc-bugsnag vendors its json dep and uses as to override the url

Richard Feldman (Jun 25 2025 at 13:28):

then the package that depends on it will end up with 1.0.0 after all, instead of 1.2.3

Richard Feldman (Jun 25 2025 at 13:29):

because it no longer sees roc-bugsnag as depending on roc-json at all; rather, it just sees "here's roc-bugsnag, which has no deps (or any deps it has are vendored, which I can't tell)"

Richard Feldman (Jun 25 2025 at 13:30):

and that prevents roc-bugsnag from sneaking in a malicious fake roc-json 1.2.3 dependency that would affect other code paths than the ones where I'm calling roc-bugsnag functions directly

Anthony Bullard (Jun 25 2025 at 13:30):

i think the snag (no pun intended) is if roc-bugsnag vendored roc-json and exposed a different API you would have to either vendor bugsnag or conform your vendored version of roc-json

Richard Feldman (Jun 25 2025 at 13:30):

you mean if roc-bugsnag exposes functions that include opaque types from its roc-json dependency?

Anthony Bullard (Jun 25 2025 at 13:31):

i mean if it changes the signature of a roc json function (or type) that is used by another consuming package

Anthony Bullard (Jun 25 2025 at 13:32):

the point of vendoring for bugsnag was to "fix" one function

Richard Feldman (Jun 25 2025 at 13:33):

Anthony Bullard said:

i mean if it changes the signature of a roc json function (or type) that is used by another consuming package

yeah, so for security reasons this would only affect consuming packages that roc-bugsnag directly depends on

Richard Feldman (Jun 25 2025 at 13:35):

so if I depend on roc-bugsnag and roc-json, and roc-bugsnag is vendoring roc-json, there's no chance my roc-json dependency will resolve to the vendored one that roc-bugsnag is claiming to be roc-json

Richard Feldman (Jun 25 2025 at 13:36):

but within the roc-bugsnag Cinematic Universe (that is, roc-bugsnag and all of its dependencies), roc-json version 1.x.y will resolve to that vendored one

Kiryl Dziamura (Jun 25 2025 at 13:36):

Richard Feldman said:

so this is tricky because of security

In "uncertainties" I mentioned that patching should probably be limited to applications only. What's the security concern there?

Richard Feldman (Jun 25 2025 at 13:36):

maybe another way to explain it is that as is like if I not only vendored the dep myself, but also I automatically went and vendored it for all my dependencies, and all their dependencies, recursively

Richard Feldman (Jun 25 2025 at 13:37):

Kiryl Dziamura said:

Richard Feldman said:

so this is tricky because of security

In "uncertainties" I mentioned that patching should probably be limited to applications only. What's the security concern there?

I think if the design is centered around vendoring (which I think is the better way to think about this) rather than patching, then it shouldn't be limited to applications!

Anthony Bullard (Jun 25 2025 at 13:43):

This makes me want to require all dependencies to be PEER and version solved to a single version per package

Anthony Bullard (Jun 25 2025 at 13:44):

An upside of that is that deep dependency hierarchies become unworkable and encourages less use of dependencies (or at least those dependencies that themselves have many dependencies).

Anthony Bullard (Jun 25 2025 at 13:45):

I can easily have 10000 deps in say npm because it bends over backward and does tons of magic to make that somehow work out for me. That is bad for software quality

Richard Feldman (Jun 25 2025 at 13:48):

it's been so long that I worked with npm that I forget what peer dependencies do, although the emotion that's coming back to me is frustration

Kiryl Dziamura (Jun 25 2025 at 13:50):

Tbh, I can't see any difference between patching and vendoring then. Are there any limitations vendoring provides that patching doesn't? What I'd like to have for security is exactly patch because it shows what exactly was changed. Even better would to hoist all overloads to the app level. Like, if there's overloading happens in nested dependency - you should first copy these changes to your roc project. But it's messy

Anthony Bullard (Jun 25 2025 at 13:51):

Peer just means "I'm telling you that you should bring this in because I need it, but I don't own the dependency itself"

Richard Feldman (Jun 25 2025 at 13:52):

yep I'm already frustrated that I don't understand the implications of "I don't own the dependency itself" :laughing:

Anthony Bullard (Jun 25 2025 at 13:52):

And it's hell in Javascript, a dynamic interpreted language with lot's of duck-typing and runtime effects

Anthony Bullard (Jun 25 2025 at 13:53):

It means that the eventual application that consumes me (after however many levels of nesting) gets to decide what version to provide to fulfill a dependency

Anthony Bullard (Jun 25 2025 at 13:53):

Basically think of it as your application is a monorepo

Anthony Bullard (Jun 25 2025 at 13:55):

WIth your app in one directory, and all the packages (deduped by name) in their own directories. And when one package dep looks for a dep of their own, their only option is the one provided

Richard Feldman (Jun 25 2025 at 13:55):

the way I think about it is:

• today, we have no formal concept of versioning. everyone just gets the exact version of whatever dependency they say they want.
• the only code sharing that happens is if multiple packages happen to specify that they want exactly the same version of a package, and it's obviously fine to code share in that circumstance because we verified via hash that they're all the same
• in terms of correctness, this is all totally fine! There is a slight edge case where multiple packages expose functions that use opaque types from a dependency that they share, but since they're using different versions of that dependency, the opaque types are treated as incompatible types (since they were defined in different modules) - but from what I've seen in the Elm and Rust ecosystems, this happens so rarely in practice that I'm not really considering it a serious downside

Anthony Bullard (Jun 25 2025 at 13:55):

This is how Google's monorepo works.

Anthony Bullard (Jun 25 2025 at 13:55):

You have a third_party directory that contains _one, single version_ for every third party dep, and all internal code has a single version used by all other packages

Richard Feldman (Jun 25 2025 at 13:56):

however, there are downsides to only "exact match" code sharing like this:

• potentially lots of unnecessary code duplication, which is a downside for both compile times and for final compiled binary size
• if performance improvements, bug fixes, etc. are released, I don't have a way to get those updates to my indirect dependencies unless my direct dependencies release a new version which upgrades to them, and then I upgrade my direct dependencies

Anthony Bullard (Jun 25 2025 at 13:57):

Yep

Richard Feldman (Jun 25 2025 at 13:58):

so to me, the whole point of the versioning system is for packages to choose to participate in dependency resolution - in other words, they are saying "hey I just need at least this version of this depdendency, and I'm fine with being given another one that's backwards-compatible with what I've been tested against"

but it's also fine for them to go the other way and say "I'm choosing not to participate in that, and I want to just have all of my dependencies exactly self-contained"

Richard Feldman (Jun 25 2025 at 13:58):

a simple way to choose not to participate in version resolution is just to vendor all your deps

Richard Feldman (Jun 25 2025 at 13:58):

that doesn't require any language features, it's just a thing you can obviously do

Anthony Bullard (Jun 25 2025 at 13:59):

There is some pain with the approach i'm outlining, but I think it's worth it. One of my jobs at Google was bringing in new Flutter releases from Github into the monorepo. That created a changelist that updated the files in the third_party/flutter directory, and then all packages that used flutter had their tests ran and builds checked overnight . It might cause a number of failures, but you could see them ALL AT ONCE, and then everyone kept rowing along

Richard Feldman (Jun 25 2025 at 13:59):

yeah that's an example of why I think it's good to have first-class support for vendoring

Richard Feldman (Jun 25 2025 at 14:00):

I think it's a totally reasonable approach for some projects to choose, and we should have nice support for it

Richard Feldman (Jun 25 2025 at 14:00):

but the problem with the "vendoring is just copy everything into local source folders" approach is that it doesn't deal with version ranges

Anthony Bullard (Jun 25 2025 at 14:01):

Yep, and then once vendored at the application level, you can make any "patches" you need. As we indeed did have to do sometimes to third party packages at Google

Richard Feldman (Jun 25 2025 at 14:01):

e.g. if I copy/paste my bugsnag dependency, by default it's saying "I want to go get this version of this dependency from this URL" and then I need to go hand-edit that to tell it to get the version from my local directory instead

Richard Feldman (Jun 25 2025 at 14:01):

but the as design I mentioned earlier fixes that

Anthony Bullard (Jun 25 2025 at 14:01):

Ah, I see

Richard Feldman (Jun 25 2025 at 14:02):

it means everyone can use URLs for specifying (dependency name + version required) even though the actual code will end up coming from a local directory

Richard Feldman (Jun 25 2025 at 14:02):

and URLs are a great way to specify (dependency name + version required) for all the usual reasons

Anthony Bullard (Jun 25 2025 at 14:02):

I think we are on the same page mostly. I just think it should affect ALL levels of the dependency graph. I should have ONE version of each package, and then be able to concretize that with either local path or a url I provide - possibly using something like your as syntax

Richard Feldman (Jun 25 2025 at 14:03):

and then you still get use cases like single-file scripts, which don't want to vendor deps because that requires vendoring them all into the same file ( :scream:), are still able to just use URLs like normal

Anthony Bullard (Jun 25 2025 at 14:03):

Yeah, just using URLs is still the base case

Richard Feldman (Jun 25 2025 at 14:03):

Anthony Bullard said:

I just think it should affect ALL levels of the dependency graph.

if you do it at the application root, it would do that

Anthony Bullard (Jun 25 2025 at 14:04):

Apps should always get the last say

Richard Feldman (Jun 25 2025 at 14:04):

an important caveat here is that packages with different major version numbers are treated as different packages

Richard Feldman (Jun 25 2025 at 14:04):

e.g. roc-json version 2.x.y is treated as a totally different package from version 1.x.y

Anthony Bullard (Jun 25 2025 at 14:05):

Sure since they are incompatible to begin with

Richard Feldman (Jun 25 2025 at 14:05):

as in, they might as well have been called roc-json1 and roc-json2 because they just do not interact in any way from a solving persepctive

Anthony Bullard (Jun 25 2025 at 14:05):

So roc-json@1 versus roc-json@2

Richard Feldman (Jun 25 2025 at 14:05):

and the major version number is just a convenience to not have to put a number in the package name (plus being able to centralize the docs in one place)

Anthony Bullard (Jun 25 2025 at 14:06):

Yeah they are functionally to the compiler different packages

Richard Feldman (Jun 25 2025 at 14:06):

exactly, yeah

Anthony Bullard (Jun 25 2025 at 14:06):

I think that all aligns with my thoughts

Richard Feldman (Jun 25 2025 at 14:07):

anyway, I think this design covers all the important use cases, because if I just want to do some local testing of a bugfix of a transitive dependency, it's super easy: I just run roc vendor to get a local version of that dep, then override it from URL to local folder using as in my root module, and now I'm able to just try out my bugfix (and even commit it while I wait for it to land upstream if I like)

Richard Feldman (Jun 25 2025 at 14:08):

but also that same strategy can work for vendoring all my deps if I want to

Richard Feldman (Jun 25 2025 at 14:08):

and version solving still works just as well as it does if no vendoring is taking place, and in the same way, and it's also all still decoupled from any centralized package index

Anthony Bullard (Jun 25 2025 at 14:08):

I think it'd be nice if vendor could update your main.roc app header to the correct syntax and path :-)

Richard Feldman (Jun 25 2025 at 14:08):

and the security guarantees are all strong

Richard Feldman (Jun 25 2025 at 14:08):

yeah that would be cool!

Anthony Bullard (Jun 25 2025 at 14:09):

That makes thing feel more like using a package manager

Anthony Bullard (Jun 25 2025 at 14:10):

Do we have a design doc or RFC for the version management syntax and operation?

Anthony Bullard (Jun 25 2025 at 14:10):

I'm assuming we still want the hashes to live on for referential integrity

Richard Feldman (Jun 25 2025 at 14:10):

I've written several of them over the years but I forget which, if any, are shared here :joy:

Richard Feldman (Jun 25 2025 at 14:11):

yeah the hashes work great in this design as it turns out

Anthony Bullard (Jun 25 2025 at 14:11):

If you have one to share, I know I at least would like to check it out

Richard Feldman (Jun 25 2025 at 14:11):

because it means that if, for example, bugsnag depends on roc-json 1.2.3 and a different dep depends on 1.3.4, not only do we know that fact, we also know the hashes of 1.2.3 and 1.3.4 (from the URLs provided by both of them)

Richard Feldman (Jun 25 2025 at 14:12):

and if any two deps are using the same version but different hashes, we know there's a serious problem and can report it at compile time

Richard Feldman (Jun 25 2025 at 14:13):

I think the relevant docs and threads are #ideas > package versioning and #ideas > package naming and search

Richard Feldman (Jun 25 2025 at 14:15):

that said, I do think I had some stuff in there about CLI flags to temporarily override things, but I've since realized I think roc vendor would be a better design than what's in that doc

Kiryl Dziamura (Jun 25 2025 at 14:41):

You mentioned that vendoring is not a security concern but patching is. However I don't see how they are fundamentally different. Could you please elaborate?

Richard Feldman (Jun 25 2025 at 15:53):

oh I don't think there's a security difference there

Richard Feldman (Jun 25 2025 at 15:59):

the specific security thing that needs to be prevented is that I have 2 dependencies, both of them depend on roc-json, and one of them overrides the roc-json version that other one uses with a version it has specified locally

Richard Feldman (Jun 25 2025 at 16:00):

overriding a versioned package with code that's specified locally needs to only affect the package which does that, along with its relevant direct dependencies

Richard Feldman (Jun 25 2025 at 16:00):

and must not affect any ancestors or siblings

Kiryl Dziamura (Jun 25 2025 at 16:36):

Gotcha. Yes, I agree, package or app should be able to override only their own dependency trees not touching anything above themselves.

Also, I think it should be possible to override dependency in a specific place of the tree:

app
    package-a
        json
    package-b
        json

So at the level of app I should be able to override specifically package-a/json.

If package-a/json is already overwritten by package-a, the patch would be applied on top of it (or in terms of vendoring, app owner takes the overwritten version of package-a/json and applies their own changes to it)

My take on patches is that they are easier for auditioning than complete vendoring. But assuming the package version you override is always specified, you can generate the diff so vendoring is a superior functionality

Richard Feldman (Jun 25 2025 at 16:44):

Kiryl Dziamura said:

I think it should be possible to override dependency in a specific place of the tree:

:thinking: what would be the use case?

Kiryl Dziamura (Jun 25 2025 at 16:52):

Specifically for debugging

Richard Feldman (Jun 25 2025 at 16:58):

but why would I want to debug changing a dep for just one of my dependencies if, after I finish debugging and upstream the dependency, it would get picked up by the others?

Richard Feldman (Jun 25 2025 at 16:58):

wouldn't I be going out of my way to test an unrealistic situation and potentially create new bugs? :sweat_smile:

Kiryl Dziamura (Jun 25 2025 at 17:10):

E.g. I want to check if the serialization problem is in the package-a and not package-b so I put dbg statement to the package-a/json only.

But I can vendor package-a and inside of it vendor json, right? So the possibility is still there

Kiryl Dziamura (Jun 25 2025 at 20:03):

Also, as a developer of a package I able to do whatever I want with my part of the dependency tree. The only question is how to optimize the DX. Unrealistic situations are most painful with lack of tooling.

But it's already possible to implement whatever crazy tool for deps management even outside the roc compiler app. I proposed it as part of compiler because it can be smart about path resolutions.

What I like about "as" is that it becomes an explicit building block of overrides inside of code and not filesystem.

Fabian Schmalzried (Jun 26 2025 at 12:43):

Would vendoring a package vendor only the package itself, or all of its dependencies as well? Because if I want to patch something, the first option might be the right choice, but if I want to vendor everything, a recursive option would be more useful.

Kiryl Dziamura (Jun 27 2025 at 07:46):

Now I think it's enough for roc compiler to introduce only a package override mechanism in code (e.g. as).

There are multiple scenarios how you can use it:

• patching
• override a particular transitive dependency (either with local location or with url)
• plain vendoring (either partial or full copy paste)
• other scenarios, e.g. via submodules idk

All of that could be outsourced to community projects

Last updated: Jun 16 2026 at 16:19 UTC