Sajjad Naveed (Oct 01 2024 at 20:45): Sajjad Naveed (Oct 01 2024 at 20:45):It would be nice to have strictly enforced semantic versioning for packages (and platforms) similar to elm.
Since, Roc packages are fully distributed and can be hosted anywhere, it will be a challenge to enforce this. But I think we can come up with some required structure for packages and platforms to enforce this. (I don't know how yet!) If we can somehow achieve this without having a central repository, that will be a huge plus.
But I think this will be really useful even if we have to go a central repository route!?!
Note: I understand we don't have a versioned release for Roc yet but I think we can at least give it some serious thought as this will have very long term effects.
Sam Mohr (Oct 01 2024 at 20:48):@Sajjad Naveed this might be an interesting thread to check out: https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/Roc's.20semver.20implementation/near/446351240
Sajjad Naveed (Oct 01 2024 at 20:48):Can we just have a central repository that stores hashes of packages and public functions and modules? And the CLI can add this and check against the repo every time a package or platform is added!?!
Sam Mohr (Oct 01 2024 at 20:49):That seems at least somewhat compatible with my understanding of the current plan
Sajjad Naveed (Oct 01 2024 at 20:52):Sam Mohr said:
Sajjad Naveed this might be an interesting thread to check out: https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/Roc's.20semver.20implementation/near/446351240
Thanks for sharing, this seems interesting.
I will give this a little more thought and see if I can contribute in any way on this.
Sam Mohr (Oct 01 2024 at 20:57):We're happy to hear any thoughts on this!
Sam Mohr (Oct 01 2024 at 20:57):You're right to point out the challenges that distributed packages bring to this
Sajjad Naveed (Oct 02 2024 at 08:02):@Sam Mohr Would love to hear your thoughts on this:
I have had the time to think about this in a little more detail and I think it's better to keep things the way they are as much as possible and try to be more like homebrew rather than cargo.
Here's the rudimentary proposal idea (it doesn't answer a lot of questions but I think for now it's better if we can agree in principle over key things.)
To keep things simple all we need is
Configuration File: roc.toml or roc.json with metadata
Publishing Process: roc publish command
Central Repository:
Import Mechanism:
As, I mentioned earlier there are a lot of other things this doesn't cover but I think everything else is secondary if we can agree that Roc will continue to support importing from any URL, and we ideally don't want to host packages and keep things light. (Ideally, I want us to allow cli tools as part of the publishing process and we can allow installing any tool from the CLI itself!?!)
This way some one who doesn't want to publish, doesn't need to worry about anything as nothing will change for them. Yet, it will bring a lot of ease for the normal Roc users.
Of course, we can introduce strict policies on who can publish, either needing a prior account or uploading a special file on the repo to show they have permission to do so.
But the key thing for us to decide is that whether we want to go the cargo route? Have dependencies listed in a single file, manage them and host them in one single place? Or be more like brew with links to the files while utilising the current import resolution and still have enforced semver on published packages and platforms?
Sam Mohr (Oct 02 2024 at 08:54):So the tl;dr of my thoughts come from a goal of Roc that hasn't been made very prominent, but is captured by the status quo and unfortunately not by this design: with the deps being declared at the top of the file, you can just hand someone a single script.roc file and they can run it if they have roc in their path
Sam Mohr (Oct 02 2024 at 08:54):Which makes Roc a great contender for scripting.
Sam Mohr (Oct 02 2024 at 08:55):However, I'm on the same page as you that other languages have spent a lot of thought and effort on figuring out how to do standard package management.
Sam Mohr (Oct 02 2024 at 08:56):So doing basically the Rust method means that we get great discoverability, availability, etc.
Sajjad Naveed (Oct 02 2024 at 08:57):I am slightly confused. How will the scripting not work here?
Instead of just importing from a url, you have an import statement at the top that mentions package name and version? Which in turn, just fetches the url so basically nothing changes (in fact I think it makes it easier for the user)?
Sam Mohr (Oct 02 2024 at 08:59):When I say scripting, I'm referring to a file starting with:
#!/usr/bin/env roc
app [main] { ... }
main = ...
The shebang compatibility
Sam Mohr (Oct 02 2024 at 09:00):With this, you don't have to compile a Roc binary explicitly, you just run ./scripts/my-tool.roc
Sam Mohr (Oct 02 2024 at 09:01):But if we don't embed the packages and their URLs into the header, you need to ask installers to copy a deps file into the same folder
Sam Mohr (Oct 02 2024 at 09:02):Which means instead, we're basically committing to the Rust approach, where in order to avoid the need to do that, we make some ~/.roc/bin that we incentivize users to compile binaries into (from local or the internet) and add that folder to their $PATH
Sam Mohr (Oct 02 2024 at 09:02):Do think that your solution would be able to handle that first case?
Sajjad Naveed (Oct 02 2024 at 09:03):Please pardon my ignorance.
But I feel this will still work because either you have imports here in the script or you don't. And if you do, then they still need to be fetched, all we will be changing here is that url comes from a call in the middle: https://central-repo/?name=weaver&version=1.0.1
And if you don't then that's not a problem.
Sam Mohr (Oct 02 2024 at 09:04):Oh, I think I understand
Sam Mohr (Oct 02 2024 at 09:04):There are two proposals here
Sam Mohr (Oct 02 2024 at 09:04):And I was assuming you wanted both considered at the same time
Sam Mohr (Oct 02 2024 at 09:06):So I'm yammering on with respect to the first one
Sam Mohr (Oct 02 2024 at 09:07):Which is about extracting dependency declaration to a separate file
Sajjad Naveed (Oct 02 2024 at 09:07):The config file that I proposed is for publishing only and doesn't affect people consuming them (we can do without it as well with just cli args). And for usage I am proposing we add a new import statement that just takes name and version (and channel?) all that does is fetch the real URL for the published package.
Sam Mohr (Oct 02 2024 at 09:07):But the second one about using a central repo that catalogs packages by name and semver works here
Sam Mohr (Oct 02 2024 at 09:12):Something I try to keep in mind for all proposals is that the end state should leave Roc with "one way to do a thing". When I look at
It doesn't replaces URL-based imports but we can introduce package name, branch, version based imports as well which in the end just gets the URL from the central repo and rest of the process remains the same
I like having either URLs or structured imports, but I think having both would lead to people using both, and then we get confusion for our users.
Sajjad Naveed (Oct 02 2024 at 09:14):You are right, I think we can find some middle ground.
But my thought process was to find the most frictionless way to achieve this without changing basically anything for the current users. People who want to continue to use URL imports can continue do so, and people who don't want their packages (basically docs and URL) published can still continue to do what they are doing.
But who ever publishes will be forced to have proper semantic versioning.
This way we get the best of both worlds imo.
Sam Mohr (Oct 02 2024 at 09:14):However, reflecting on how many times I've needed to use code based on a specific git commit and not a semver, enforcing that you can only use a semver is not ideal
Sam Mohr (Oct 02 2024 at 09:14):So maybe we'd want the structured import to allow passing strictly one of "semver" or "git commit"
Sam Mohr (Oct 02 2024 at 09:15):I think that providing both options is better for app authors, but worse for app readers. It's extremely minor, but...
Sajjad Naveed (Oct 02 2024 at 09:17):I get where you are saying but I think if we think this kind of a solution will be beneficial for Roc in the long term then it will be easy to iron out minor details.
Sam Mohr (Oct 02 2024 at 09:18):Also, not sure if you've already got plans for this, but our current approach really likes having the hash so you can make sure that the dependency wasn't modified upstream (tutorial link), meaning we'd need to support some kind of locking. That probably falls into "minor details"
Sajjad Naveed (Oct 02 2024 at 09:18):Na I am relying on the current system altogether, the underlying URL still needs to follow all the current rules basically.
Sam Mohr (Oct 02 2024 at 09:19):Well, for something as impactful as package management, a lot of the minor details need to be figured out so we don't shoot off our feet later
Sajjad Naveed (Oct 02 2024 at 09:20):I agree but the structured import is just a syntactic sugar on top to get the real URL which will still have the security hash and everything. So, I don't think basically anything is changing in the true sense.
Sam Mohr (Oct 02 2024 at 09:21):Sajjad Naveed said:
Na I am relying on the current system altogether, the underlying URL still needs to follow all the current rules basically.
Maybe I'm missing something, but if application authors get automatic translation of imports to the real URL, but it's not written anywhere, then it sounds like the code hash is no longer written in their codebase
Sam Mohr (Oct 02 2024 at 09:22):Meaning we lose the security guarantee of the user being mathematically confident that the code they're pointing to hasn't changed under their feet
Sam Mohr (Oct 02 2024 at 09:23):Apologies if you explain any details that I've missed, I'm pretty tired haha
Sam Mohr (Oct 02 2024 at 09:23):Thanks for your patience
Sajjad Naveed (Oct 02 2024 at 09:23):Yeah but it's sitting in the central repo where it can't change. And any one who doesn't trust the central repo can still use the URL imports directly because that option is still not taken away from them.
Sajjad Naveed (Oct 02 2024 at 09:24):Na I haven't slept much myself and I need to do a better job to do a clean write up on this. But I just wanted to run this by someone who knows more than me.
Sam Mohr (Oct 02 2024 at 09:24):Sure
Sajjad Naveed (Oct 02 2024 at 09:29):We can even show the real URL on hover (etc) in the IDE if that helps!?! Or even on the command line which lists all the imports and their true urls.
May be I am mistaken in my understanding here but "most" people should not have any issues with this as all information will be publicly available on the package page and those that might, have an easy way out to go back the URLs.
Sam Mohr (Oct 02 2024 at 09:30):Yeah, I tried skimming for an explanation in Zulip/GitHub history, but I'm not seeing one. I think this comes down to a @Richard Feldman question, which is: how comfortable are we with moving trust on immutability of dependencies from "the URL guarantees it" to "a central repo guarantees it"?
Sam Mohr (Oct 02 2024 at 09:31):Because if we're okay with a central repo making those guarantees, then most people will probably reach for this behavior instead of the URL one, meaning most devs will be putting their trust in a central repo
Sam Mohr (Oct 02 2024 at 09:31):@Sajjad Naveed most people actually don't care about security haha
Sam Mohr (Oct 02 2024 at 09:32):So I think it's somewhat our responsibility to see if we can make their lives more secure even if they don't care until they're personally affected
Sajjad Naveed (Oct 02 2024 at 09:36):That's a good question!
But I feel we can try to allay those fears as much as we can. But at the end of the day the question we need to answer here is if this kind of an approach works at least in principle and provides enough benefit for the end user (either via enforced semver, public docs or no need to always go and find the url). And if the perceived benefit is large enough and the disruptions are low enough for us to take this leap.
Because as it was mentioned in the proposal you shared, sooner or later someone will build a solution for this even if it's just a simple cli tool that adds the urls.
The only difference is that I am proposing to keep things loosely coupled instead of having a full on package manager.
Sam Mohr (Oct 02 2024 at 09:36):You mean like https://github.com/imclerran/roc-start? :smile:
Sajjad Naveed (Oct 02 2024 at 09:37):Lol yeah :rolling_on_the_floor_laughing:
Sam Mohr (Oct 02 2024 at 09:37):The plan, I believe, is to incorporate behavior from roc-start into the main roc binary to facilitate adding deps
Sam Mohr (Oct 02 2024 at 09:37):But anyway, yes, this kind of an approach definitely works in other languages!
Sajjad Naveed (Oct 02 2024 at 09:43):I will let Roc team to be the judge on the best way forward here. I am pretty sure you have thought about Roc and the encompassing ecosystem way longer than someone like me.
But I would love to hear @Richard Feldman 's opinion here if possible :pray:
Sam Mohr (Oct 02 2024 at 09:44):Maybe I have? But fresh ideas from newer contributors are also how we got the spread syntax changes: https://github.com/roc-lang/roc/issues/7091
Sam Mohr (Oct 02 2024 at 09:44):Which I think are very exciting
Sajjad Naveed (Oct 02 2024 at 09:45):Thank you so much @Sam Mohr. I would love to keep bugging you in the future :wink:
Sam Mohr (Oct 02 2024 at 09:45):I appreciate humility, it's a valuable trait, but I think you pushing on this is valuable because we've seemed to just assume it's the right way to move
Sam Mohr (Oct 02 2024 at 09:45):As the zoomers say, poggers
Luke Boswell (Oct 02 2024 at 11:34):I thought Richard had a design that was pretty mature on how package versioning and the central repository will work. I don't think we started building it. It may be another project that is looking for someone to pick up and run with.
Luke Boswell (Oct 02 2024 at 11:38):Ah found it :tada:
https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/package.20versioning/near/407944222
Luke Boswell (Oct 02 2024 at 11:41):@Sajjad Naveed what do you think of this proposal?
Sajjad Naveed (Oct 02 2024 at 12:25):Luke Boswell said:
Ah found it :tada:
https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/package.20versioning/near/407944222
This is an excellent proposal if the goal is just to get the semantic versioning. I don't think it can get better than this tbh.
But as the document itself mentions, upgrade process, package discovery is beyond the scope of the proposal and it will be tackled in a seperate proposal using central index.
Of course, I am biased here and believe that my proposal idea actually addresses those up front with no breaking changes for the current flow. But even ignoring my idea, I think it will be best to look at these problems together rather than individually. Because if we are to have a central index than may be we don't need the previous tag approach!?!
We will most likely still need to implement a version resolution algorithm but I am not fully sure how the current URL imports avoid that problem!?! Because my original idea was to just go with what current URL import process does, as in the end we are just resolving to a URL.
Richard Feldman (Oct 02 2024 at 12:41):it might be helpful to list some design characteristics I think are important:
Sajjad Naveed (Oct 02 2024 at 12:55):Thanks for the response @Richard Feldman. Agreed on all three points. But I "think" my idea follows all these 3 design characteristics.
If you are not immediately disgusted by the approach, I am more than happy to write this into a full proposal. But if you think this can't work then we just move on :)
Richard Feldman (Oct 02 2024 at 12:58):hm, I might be missing something, but let's say I download a .roc file that uses this import syntax exclusively. where do the security hashes appear in that file? :thinking:
Sajjad Naveed (Oct 02 2024 at 13:02):It won't appear in that file by itself but what I am proposing is we can show the resolved URLs in the IDE (as tooltips, annotations etc automatically) and even on the CLI with some basic command if someone wants to look at those URLs. We list every single import alongside the resolved URL (and some other info!?!)
Richard Feldman (Oct 02 2024 at 13:04):Sajjad Naveed said:
It won't appear in that file by itself
if the security hash isn't in the one file I downloaded, where do I get it?
Richard Feldman (Oct 02 2024 at 13:05):I think it's important for security that I have the actual hash; otherwise, wherever I'm getting the hash from might be compromised
Richard Feldman (Oct 02 2024 at 13:06):from a security perspective, having the hash in the actual source code is unbeatable
Richard Feldman (Oct 02 2024 at 13:06):(which is why lockfiles typically have them, but we don't want a separate lockfile)
Sajjad Naveed (Oct 02 2024 at 13:08):Public package listing on the central index will always have the hashes alongside the URLs (and documentation).
Richard Feldman (Oct 02 2024 at 13:08):sure, but the central index can be compromised
Richard Feldman (Oct 02 2024 at 13:09):if the hash is in the source, I don't have to worry about anything being compromised
Richard Feldman (Oct 02 2024 at 13:09):worst case scenario is that the code won't run
Sajjad Naveed (Oct 02 2024 at 13:10):Yeah but that's why we will always show the resolved URLs in the IDE and the CLI? Do you think that won't be enough?
Richard Feldman (Oct 02 2024 at 13:14):where do the "resolved URLs" come from if not the source file?
Richard Feldman (Oct 02 2024 at 13:14):the answer has to be "from a third party server, which can be compromised and therefore lie about the hash"
Sajjad Naveed (Oct 02 2024 at 13:14): Richard Feldman (Oct 02 2024 at 13:15):so the point is that it's meaningfully less secure, which I don't think is worth it
Richard Feldman (Oct 02 2024 at 13:17):I'd rather have the hash in the source, for unbeatable security, and then have the ide collapse it for you
Sajjad Naveed (Oct 02 2024 at 13:26):With all due respect. I don't agree obviously with this as the URL itself is coming from the package publisher with the security hash in it, same as it's now. Central Index lists everything publicly as well but of course if that's compromised itself than that's a problem but for that reason we are trying to surface the URLs to the developer.
In my personal opinion even leaving behind the convenience argument and other benefits, there's a higher likelihood for someone adding a compromised package URL to their file when they go hunting for a package than the risk of central index being compromised.
But I am gonna rest my case now :)
Looking forward to whatever solution you guys end up implementing. And best of luck! :praise:
Sajjad Naveed (Oct 02 2024 at 13:57):Forgot to add one thing (as much I want to let it go my mind is still stuck here), that we won't be taking the standard URL import away. So, anyone who wants to be extra conscious about security, they can still continue to use that!?!
But I get the point of enforcing this for all.
Really looking forward to hearing more about this topic in the future :fingers_crossed:
Sajjad Naveed (Oct 03 2024 at 05:06):@Richard Feldman Do you think it will enough if we allow hash as the optional argument in the import? and/or allow tests to be written for the imports as well?
Both of these will break with automatic minor updates if the user has not specified the exact version e.g. ^1.0.0 but may be that's the exact behaviour we want?
User still gets "notified" of the update being available and then they can chose to either update the hash in the import and/or tests or just go to specifying the exact version they want to stick with?
I understand where you stand and you obviously have a much clear vision of where you want to take Roc. But if Roc is to support central index (let alone a full package manager) one day then this issue will still remain. Unless of course the goal is to never support central indexes?
Richard Feldman (Oct 03 2024 at 10:55):Sajjad Naveed said:
Richard Feldman Do you think it will enough if we allow hash as the optional argument in the import? and/or allow tests to be written for the imports as well?
I think the security hash should be in the source file, so any design in which the source file does not contain the security hash would not satisfy that goal :big_smile:
Richard Feldman (Oct 03 2024 at 10:56):I mentioned this in the design doc, but I separately think automatic updates are a mistake - so regardless of how specifying dependencies works, I think updates should be a thing you opt into, not automatic
Richard Feldman (Oct 03 2024 at 10:57):(basically I agree with Go's update philosophy on that topic)
Sajjad Naveed (Oct 03 2024 at 11:08):I think I get it now, in that case going with the previous tag/line approach from your proposal is all we need if we are just after semver. Because if we need to always have hash being present in the code then the user might as well enter the full URL.
And central index, is secondary and won't serve (or at least need to serve) as a way to get (and update) packages but just package discovery, and documentation if it's every built.
Richard Feldman (Oct 03 2024 at 11:23):this does remind me that I need to ship that second proposal, about central index design! :big_smile:
Sajjad Naveed (Oct 03 2024 at 11:42):Yeah really looking forward to that.
I genuinely believe finding the package (or even the platform) on the internet leaves users' open to spoofing attacks and a central index design is a definitely needed to curb something like that.
But since you believe that index could be compromised itself (which is true), then I don't know what to think/trust any more.
I am sure you will come up with something clever like you did with the semver proposal (this is me sucking up to you in case it isn't obvious :P)
But really appreciate you taking out the time to respond :praise:
@Sam Mohr argued the same point about hash being in the source code but somehow I kept on insisting and arguing because I was just too passionate about the idea, apologies :pray:
Richard Feldman (Oct 03 2024 at 13:06):oh no worries! pushback is always welcome as long as it's civil add friendly like this! :hearts:
Notification Bot (Oct 03 2024 at 13:11):Sajjad Naveed has marked this topic as resolved.
Last updated: Jun 16 2026 at 16:19 UTC