Stream: ideas

Topic: package versioning

Richard Feldman (Dec 14 2023 at 11:55):

here is a proposal for how package versioning can work in Roc!

Kevin Gillette (Dec 14 2023 at 14:57):

In the doc, you have an example of a version: v1.9.0-rc2, and the doc highlights the 1.9.0 part. According to semver, the version includes -rc2 and is not the same as version 1.9.0.

What happens if one dependency specifies v1.9.0-rc2 while another specifies v1.9.0 ? More generally, what happens if the same apparent version is listed in two different urls (i.e. have different hashes)?

Kevin Gillette (Dec 14 2023 at 14:58):

Could we support full semver, rather than just the numbered portion?

Richard Feldman (Dec 14 2023 at 14:58):

I don't think we should, no

Richard Feldman (Dec 14 2023 at 14:58):

I think just the numbers is best

Richard Feldman (Dec 14 2023 at 14:59):

if a package author wants to do a prerelease, we already have a workflow for that: publish it to some other URL and then anyone who wants to try it out can override their local version using a flag

Kevin Gillette (Dec 14 2023 at 15:00):

That's fair. The ^^ explanation might be worth copying into the doc ;)

Kevin Gillette (Dec 14 2023 at 15:09):

For bundle, I wonder if it's worth making the default version 0.1.0. The 0.x releases of course promise nothing about compatibility, which is where many projects may wish to start. I fear if we start with 1.0.0, a package might get to 6.x (in a matter of days or weeks) before the author figures out what they really want the contract to look like.

Certainly we could discourage early publishing. Or we could embrace that the tooling will do the right thing.

That said, my general hope would be that most packages have a meaningful 1.0.0, and that each major release beyond that also represents a meaningful, considered departure from the previous major release...

Perhaps we could provide a warning, or a confirmation flag in the tooling before it stamps the new major version.

Richard Feldman (Dec 14 2023 at 15:20):

yeah 0.1.0 is a possibility!

Agus Zubiaga (Dec 14 2023 at 15:24):

The proposal looks great to me! Dealing with previous might be slightly awkward, but I get the motivation behind it and I couldn't think of a better option. I guess if you're publishing to a central index, you either won't need it or can automate the updating somehow.

Kevin Gillette (Dec 14 2023 at 15:27):

Regarding patch bumps, let's say just reorder the code with no other changes, or perhaps I change a local function-scoped variable name (one which is not used with dbg or any other way in which the change is observable at runtime).

In other words, my changes provably have no impact on behavior. Should bundle offer to patch bump that without a --force flag?

Agus Zubiaga (Dec 14 2023 at 15:28):

Oh, what if you just used the URL for previous? That way you don't have to have it in your cache, and you can copy-paste entirely without having to manually extract the version and hash from it.

Agus Zubiaga (Dec 14 2023 at 15:29):

Kevin Gillette said:

Regarding patch bumps, let's say just reorder the code with no other changes, or perhaps I change a local function-scoped variable name (one which is not used with dbg or any other way in which the change is observable at runtime).

I think the API would remain the same and therefore it would go with a patch bump.

Kevin Gillette (Dec 14 2023 at 15:47):

Right, it would be a patch bump if a bump was made. The question is that if we end up with good change detection tooling, should non-behavior-impacting changes even produce a patch bump? i.e. do they warrant a release, or should they be rolled into the next behavior-impacting release?

Kevin Gillette (Dec 14 2023 at 15:51):

Making it a release puts pressure on the source control/artifact system, build/test systems, central package index, the package cache of users and the users themselves ("there's a new patch release, so I better go get it and update all the things because it _probably_ fixes a bug"), all for what amounts to no behavioral change whatsoever.

Agus Zubiaga (Dec 14 2023 at 16:18):

Interesting. I think detecting that is an extension of the halting problem, but even if you did it only for simple cases, would someone even try to make a release in that situation?

Agus Zubiaga (Dec 14 2023 at 16:20):

I guess it could happen if they have a CI action set up to do it automatically on changes to main or something like that.

Brendan Hansknecht (Dec 14 2023 at 16:26):

When an app depends on a particular version of a package, it is saying "I require exactly this version."

I'm not a fan of this

Brendan Hansknecht (Dec 14 2023 at 16:26):

or this:

When a package depends on a particular version of another package, it is saying "I require at least this version."

Agus Zubiaga (Dec 14 2023 at 16:27):

Brendan Hansknecht said:

When an app depends on a particular version of a package, it is saying "I require exactly this version."

I'm not a fan of this

Isn't that what everyone ends up doing with lock files anyway?

Brendan Hansknecht (Dec 14 2023 at 16:28):

Two main reasons:

1. sharing a base application that users build off of, I want users to automatically get upgraded packages. Or if I am working on a project, maybe I want to pull in security patches automatically. So I much prefer saying similar controls to rust

Brendan Hansknecht (Dec 14 2023 at 16:29):

Lock files do require a specific version, but I have seen systems without lockfiles checked in that auto upgrade in all CI runs. I have seen workflows that will fail if the lock file isn't upgraded all the way or that automatically upgrade.

Brendan Hansknecht (Dec 14 2023 at 16:30):

I have seen build shell scripts that just update package minor versions

Brendan Hansknecht (Dec 14 2023 at 16:31):

So there are lots of workflows where people want to stay as updated as possible (sometimes even automatically testing minor and patch version) because they want to ensure that don't miss any sort of security patch or accidentally fall way behind such that upgrades become painful

Agus Zubiaga (Dec 14 2023 at 16:33):

With a central index, you could have a CLI command or flag for build which checks whether there are newer versions

Brendan Hansknecht (Dec 14 2023 at 16:33):

As for packages just specifying at least a version, I have seen cases where a specific minor version is locked to in a package because the update subtly changes things in a way that brakes the depending package. It also can be defense after a malicious attack like left pad. A package author can just lock to the old version instead of updating to the new patch update that claims to change nothing but leads to all code paths recursing forever or crashing.

Brendan Hansknecht (Dec 14 2023 at 16:34):

With a central index, you could have a CLI command or flag for build which checks whether there are newer versions

what if I need to pin 1 package but not another. Much nicer to write that out in a file than to deal with it in cli with flags.

Agus Zubiaga (Dec 14 2023 at 16:36):

Brendan Hansknecht said:

what if I need to pin 1 package but not another. Much nicer to write that out in a file than to deal with it in cli with flags.

Good point. I suppose that could be part of the packages header syntax, but I'd definitely have some way to lock dependencies since that's very common and useful.

Brendan Hansknecht (Dec 14 2023 at 16:39):

Yeah, in my mind there are 3 states, can update minor version and patch, can just update patch version, can not update at all. If we just have those 3, I think it would be enough support.

Brendan Hansknecht (Dec 14 2023 at 16:40):

Question, will 0.1.0 follow the modified version of semver for under 1.0.0? Not actually sure if semver specifically specifies this or not, but it is essentially that it becomes 0.major-version.minor-version in terms of what changes are allowed.

Brendan Hansknecht (Dec 14 2023 at 16:42):

I guess from the spec, semver means nothing essentially before 1.0.0:

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

Agus Zubiaga (Dec 14 2023 at 16:42):

Brendan Hansknecht said:

Yeah, in my mind there are 3 states, can update minor version and patch, can just update patch version, can not update at all. If we just have those 3, I think it would be enough support.

Do you need the distinction between patch and minor though? There shouldn't be a semantic difference in compatibility

Kevin Gillette (Dec 14 2023 at 16:44):

Agus Zubiaga said:

Interesting. I think detecting that is an extension of the halting problem

Yeah, that's probably true, but I didn't mean that we need to be exhaustive. If the outcome is opportunistically preventing unnecessary patch releases, we only need to make sure we have no false negatives. There are plenty of common manipulations we can choose to check for which have guaranteed zero impact on behavior. If we're unsure about something, we consider it to be behavior-influencing. If a change soley contains changes we're sure have no impact on behavior, then we can be sure the diff as a whole has no impact on behavior, and that doesn't require solving the halting problem.

This could also help with managing build caches if we can do it quickly enough. If there's no change in behavior compared to a cached build, then we don't need to rebuild.

It can also help with running only those tests that could have changed

would someone even try to make a release in that situation?

Yeah, it happens. I've seen all manner of dubious release discipline.

Brendan Hansknecht (Dec 14 2023 at 16:47):

Do you need the distinction between patch and minor though?

This is where, theory and practice diverge in my experience. Often, patch is used for bugfixes and security. As a corporation, I likely want to try to automatically update patch version. Minor version, should be compatible, but sometimes despite having the same api will change things in a way that either:

1. I want to double check their new apis and maybe update my code to the new apis (so good to have manual intervention)
2. I am much more nervous about subtly breaking changes that will break my app (probably in a way I haven't tested)

Brendan Hansknecht (Dec 14 2023 at 16:48):

I think being able to lock either is really useful.

Brendan Hansknecht (Dec 14 2023 at 16:51):

I guess from the spec, semver means nothing essentially before 1.0.0:

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

@Richard Feldman, what are your plans here? If we start at 0.1.0 will we follow semver where apparently any package number change is consider a breaking change. Would we enforce more structure than semver specifies?

The official semver recommendation is to stay below 1.0.0 until your api is essentially stable. Though they do comment that if your code is used in production, it probably should be 1.0.0.

John Murray (Dec 14 2023 at 19:03):

I really like the --replace-deps option, would help me get rid of some of the jank sed replacements i do in roc2nix

Richard Feldman (Dec 14 2023 at 19:44):

Agus Zubiaga said:

Oh, what if you just used the URL for previous? That way you don't have to have it in your cache, and you can copy-paste entirely without having to manually extract the version and hash from it.

that was the initial design I had, but then I realized it would cause problems if you wanted to do things like mirroring (all versions of) a package while keeping all of its hashes the same. If previous tells you "given the URL where you got this, you now have enough info to get the previous package" then that works (and has the same content hash) no matter what URL you found it at.

In contrast, if it hardcodes the URL, then if you host it at a new URL (e.g. because you're the package author and you lost control of the old one - or are forking it going forward - but you want to keep the hashes the same so people can be 100% confident the old versions still work exactly the same way they used to even though the URL is different) then either you have to redo all the hashes or else you have to awkwardly have old versions of the package explicitly mention a different URL from the one where you're getting the current version :sweat_smile:

Richard Feldman (Dec 14 2023 at 19:44):

Kevin Gillette said:

Right, it would be a patch bump if a bump was made. The question is that if we end up with good change detection tooling, should non-behavior-impacting changes even produce a patch bump? i.e. do they warrant a release, or should they be rolled into the next behavior-impacting release?

I don't think we should do this. If you have non-behavior-impacting changes, commit them to source control and don't publish a new release. :big_smile:

Richard Feldman (Dec 14 2023 at 19:48):

Brendan Hansknecht said:

Lock files do require a specific version, but I have seen systems without lockfiles checked in that auto upgrade in all CI runs. I have seen workflows that will fail if the lock file isn't upgraded all the way or that automatically upgrade.

I have seen build shell scripts that just update package minor versions

So there are lots of workflows where people want to stay as updated as possible (sometimes even automatically testing minor and patch version) because they want to ensure that don't miss any sort of security patch or accidentally fall way behind such that upgrades become painful

setting aside the question of whether it's a good idea to do this, I see this as more of a discovery design question, because it requires being able to answer the question "what is the newest version of this package?" - which in turn requires contacting an index.

So for example, I think the way someone might do this is to run roc upgrade (or whatever it ends up being called) that goes and updates versions in your code base and sets them to the latest. That could be done in CI, which would make it possible to do that "fail CI if we aren't upgraded all the way" workflow.

Richard Feldman (Dec 14 2023 at 20:20):

Brendan Hansknecht said:

As for packages just specifying at least a version, I have seen cases where a specific minor version is locked to in a package because the update subtly changes things in a way that brakes the depending package. It also can be defense after a malicious attack like left pad. A package author can just lock to the old version instead of updating to the new patch update that claims to change nothing but leads to all code paths recursing forever or crashing.

I'm glad you brought this up! I have thoughts on this which I should have put in the doc; I'll write them up later tonight

Richard Feldman (Dec 14 2023 at 20:22):

Brendan Hansknecht said:

Question, will 0.1.0 follow the modified version of semver for under 1.0.0? Not actually sure if semver specifically specifies this or not, but it is essentially that it becomes 0.major-version.minor-version in terms of what changes are allowed.

yeah if we did 0.1.0 I think version 0.x.y should mean that x is major, y is minor/patch (and there wouldn't be a distinction there for 0.x.y versions)

Richard Feldman (Dec 14 2023 at 20:24):

Agus Zubiaga said:

Brendan Hansknecht said:

Yeah, in my mind there are 3 states, can update minor version and patch, can just update patch version, can not update at all. If we just have those 3, I think it would be enough support.

Do you need the distinction between patch and minor though? There shouldn't be a semantic difference in compatibility

this is what ComVer does - it's just two digits: one for backwards-incompatible changes and one for backwards-compatible ones.

I like the simplicity of that, and I was actually talking to some people about whether we should use ComVer in Roc, but ultimately it seemed like the SemVer distinction was useful enough to have all 3 digits (except maybe in pre-1.0.0 versions)

Richard Feldman (Dec 14 2023 at 20:24):

I'm loving this discussion btw, thanks for the great comments and questions, everyone! :smiley:

Brendan Hansknecht (Dec 14 2023 at 20:30):

I really like comver with the exception of something like security patches.

Sky Rose (Dec 15 2023 at 02:54):

I like this. Having distributed hosting of static files is such a nice idea.

Some thoughts on SemVer:

maybe some type signatures got less strict ... the minor version stays the same

1. Should this be a minor bump instead? Having a signature that becomes less strict is a lot like adding a new value, which does get a minor bump. It's backwards compatible to upgrade, but not to downgrade.

you can also always ignore it and bump less

2. Should this be allowed? Automatic version bumping could cause compilation failures if patch versions can't be trusted to be backwards compatible. With the automatic version resolution, upgrading one dependency could cause code that doesn't use it to fail to compile, if a transitive dependency gets bumped.
   I guess patch versions could have logically non-backwards compatible changes without type changes, or because there's no central hosting authority you could just upload files without passing the semver checks, so this problem can't be totally avoided, but it should be hard to avoid.

3. I agree on starting at 0.1.0 or 0.0.1. A "1.0" release is an indication of stability that we shouldn't encourage package authors to claim unless they're ready for it.

Sky Rose (Dec 15 2023 at 02:55):

4. How would auth work for non-public packages? (I don't have opinions or experience or stakes, but it seems like it should be included in this discussion.)

Richard Feldman (Dec 15 2023 at 03:04):

Richard Feldman said:

Brendan Hansknecht said:

As for packages just specifying at least a version, I have seen cases where a specific minor version is locked to in a package because the update subtly changes things in a way that brakes the depending package. It also can be defense after a malicious attack like left pad. A package author can just lock to the old version instead of updating to the new patch update that claims to change nothing but leads to all code paths recursing forever or crashing.

I'm glad you brought this up! I have thoughts on this which I should have put in the doc; I'll write them up later tonight

ok, I added it to the doc - it starts with "Buggy or Malicious Releases" and goes up to the Summary section at the end!

Richard Feldman (Dec 15 2023 at 03:09):

Sky Rose said:

4. How would auth work for non-public packages? (I don't have opinions or experience or stakes, but it seems like it should be included in this discussion.)

good question! I didn't put it in the doc, but my default thinking is that if organizations want to restrict access their URLs, they can put them behind a VPN or something so they're still normal URLs but aren't accessible on the public Internet. So at a baseline it's definitely already possible to do this, just by the nature of URLs.

I figure if that doesn't work for some orgs in practice, we can discuss on a case-by-case basis what their constraints are and figure out what to do based on that.

Sky Rose (Dec 15 2023 at 03:13):

Just the VPN isn't very flexible. I guess you could also download the files separately with whatever system you want, and then point to the files on your filesystem (via a relative path, if the path is checked into source control) instead of having roc build download them.

Maybe one common case would be linking to private GitHub repos?

Richard Feldman (Dec 15 2023 at 03:18):

Sky Rose said:

Just the VPN isn't very flexible. I guess you could also download the files separately with whatever system you want, and then point to the files on your filesystem (via a relative path, if the path is checked into source control) instead of having roc build download them.

that's true, but also keep in mind that a lot of orgs use VPNs for other reasons, so it might turn out that the set of orgs interested in private packages overlaps almost completely (or even completely) with the orgs that already use VPNs and don't need anything else besides URLs :big_smile:

Kevin Gillette (Dec 15 2023 at 04:56):

Richard Feldman said:

Brendan Hansknecht said:

Question, will 0.1.0 follow the modified version of semver for under 1.0.0? Not actually sure if semver specifically specifies this or not, but it is essentially that it becomes 0.major-version.minor-version in terms of what changes are allowed.

yeah if we did 0.1.0 I think version 0.x.y should mean that x is major, y is minor/patch (and there wouldn't be a distinction there for 0.x.y versions)

I don't understand this interpretation. iiuc, semver makes no guarantees about compatibility in 0.y.z, but 1.0.0 is, iiuc, definitely the first major version. It's not that the minor is actually a major, it's that there's just a special, limited cut-out in the spec to forgo compatibility rules. I would imagine that in roc tooling this simply can be achieved by not checking for compatibility when the major component is 0.

Kevin Gillette (Dec 15 2023 at 05:00):

Brendan Hansknecht said:

I really like comver with the exception of something like security patches.

You could consider a security patch to be introducing a new feature (security!) or breaking compatibility (from a Hyrum's Law perspective), depending on the audience :thinking:

Kevin Gillette (Dec 15 2023 at 05:09):

Sky Rose said:

maybe some type signatures got less strict ... the minor version stays the same

1. Should this be a minor bump instead? Having a signature that becomes less strict is a lot like adding a new value, which does get a minor bump. It's backwards compatible to upgrade, but not to downgrade.

Another thought here: if a signature becomes less strict, after that point it would then become a breaking change to revert it back to its prior, stricter form.

Perhaps there's something like a _version algebra_ we can consistently apply to determine which component bump a change warrants based on what bump reverting that change would require.

Would a reversion of a patch always be another patch? I don't know in terms of an interpretation of compatibility based on _observable_ behavior. Based on signatures and types alone though, it'd be another patch.

Reverting a minor would definitely require a major. Reverting a major would also require a major.

Inversely, based on signatures and types, if the reversion requires a major, I believe the change being reverted must have been a minor or major, not a patch.

Kevin Gillette (Dec 15 2023 at 05:39):

Richard Feldman said:

ok, I added it to the doc

I believe pinning to an exact version is just as problematic as you describe (i guess it's fine for an application to do so, but not a package).

However, I don't think formal version _exclusions_ are an issue, and, for example, Go does support those.

The essence of minimum version selection is to boil an NP-complete _search_ problem into a linear time _selection_ problem. The main offender that makes it search is expressions of the kind: "I _want_ a version that is _less than_ X"

All minimum version selection expressions are _greater than or equal_.

Exclusions don't cause a later version to be selected, they just pass or fail the version that already would've been selected.

For example, consider that A depends on B and C, and B depends on D v1.0.0 (minimum) and C depends on D v1.5.0... (to illustrate, this is a diamond dependency).

If there's an exclusion among any of A or B for D v1.1.0 through v1.6.0, then the build will fail, since the minimum version is v1.5.0, which as a fast check (linear to number of exclusion ranges for D), is excluded. If there's a v1.7.0 available, and A explicitly requests that version, the build now passes.

Conversely, if the exclusion was for v1.1.0 through v1.4.0, the build succeeds, since the minimum version via C is v1.5.0, which does not run afoul of any exclusions.

Certainly please scrutinize my reasoning on this.

In any case, the tooling does not go _looking_ for a satisfiable version; it could, but then that could devolve into a search problem, and would be problematic. I suppose the tooling could, upon seeing an exclusion error, lookup the next highest version past the minimum overlapping exclusion range and suggest that to the user safely enough, because that's also not a search strategy.

Brendan Hansknecht (Dec 15 2023 at 05:48):

Kevin Gillette said:

I don't understand this interpretation. iiuc, semver makes no guarantees about compatibility in 0.y.z, but 1.0.0 is, iiuc, definitely the first major version. It's not that the minor is actually a major, it's that there's just a special, limited cut-out in the spec to forgo compatibility rules. I would imagine that in roc tooling this simply can be achieved by not checking for compatibility when the major component is 0.

Yeah. So not that this follows the semver ruling, but it is a common things packages do that are before 1.0.0. Essentially many packages still want meaningful versioning while being at 0.x.y so they enforce that y is for non breaking changes (features and bug fixes so equivalent to minor version after 1.0) and x is for breaking changes (equivalent to major changes after 1.0).

I find enforcing this pre 1.0 is really useful for users and not that big of a hassle for package authors, so I prefer it. But yeah, not an official part of semver in any way.

Kevin Gillette (Dec 15 2023 at 08:07):

okay, yeah that makes sense. it definitely wouldn't make much sense for a patch bump to be breaking in 0.x.y or any other formally-major version

Jakob Steinberg (Nov 23 2024 at 08:03):

Sky Rose said:

Just the VPN isn't very flexible. I guess you could also download the files separately with whatever system you want, and then point to the files on your filesystem (via a relative path, if the path is checked into source control)

Maybe also a file:///… URL could be used

Tobias Steckenborn (Feb 10 2025 at 16:03):

I really like the proposal around some sort of epoch value in https://antfu.me/posts/epoch-semver

Last updated: Jun 16 2026 at 16:19 UTC