Stream: ideas

Topic: include file in binary

Kilian Vounckx (Sep 07 2022 at 08:18):

Somewhat related, I think a way to include a file in a binary as either a Str of List U8 would be nice
A bit like @includeStr in zig, or include_str! in rust
So not opening files at runtime, but at compile time to have them behave the same as literals

Folkert de Vries (Sep 07 2022 at 09:23):

I think we have talked about being able to import non-roc files as a string/bytes? We don't have good other places in the syntax where we could add this feature (at the moment)

Kilian Vounckx (Sep 07 2022 at 09:40):

Maybe something like how go does it could work? Adding a comment above?

#roc:embed filename.txt
file : Str

This would mean comments need to be parsed though, so probably not an ideal solution

Folkert de Vries (Sep 07 2022 at 10:02):

putting it into the header has the additional benefit that the normal parser does not need to be doing file IO

Folkert de Vries (Sep 07 2022 at 10:02):

(e.g. for including strings, we must check that it is valid UTF8)

Anton (Sep 07 2022 at 10:10):

I think it would make sense to treat it as a "special" import

Georges Boris (Sep 07 2022 at 11:52):

I believe the feature @JanCVanB is talking about is being able to read/write files as effects on the cli platform though (importing files is pretty interesting! but I think these are two separate threads)

Kilian Vounckx (Sep 07 2022 at 11:54):

Yeah my bad, this should be a different thread indeed
Could someone with more zulip knowledge maybe move the parts about importing files to the ideas thread?

Anton (Sep 07 2022 at 11:59):

Moving messages here...

Richard Feldman (Sep 07 2022 at 12:58):

yeah I definitely want to do something along these lines!

Richard Feldman (Sep 07 2022 at 12:58):

one question I'm not sure about is what the scope should be

Richard Feldman (Sep 07 2022 at 12:59):

for example, here are some possible designs:

• only supporting bytes
• supporting bytes and UTF-8 strings (and we give an error at import time if the contents of the imported file aren't valid UTF-8)
• supporting arbitrary values with Decode, and you specify the DecodingFormat when you do the import

Richard Feldman (Sep 07 2022 at 13:00):

that last one would mean you could do something like importing a .json file and access it like a top-level record, with that record (eventually) being stored in the binary rather than having to be parsed out of the JSON when the program starts up

Richard Feldman (Sep 07 2022 at 13:01):

(or lazily and then cached, or something like that)

Richard Feldman (Sep 07 2022 at 13:01):

on the one hand, that would require executing user code before assembling the final binary - so we'd have to sort of compile everything, execute some of it, and then create the final binary

Richard Feldman (Sep 07 2022 at 13:03):

on the other hand, that already separately seems like something we probably want to do for optimized builds anyway (that is, evaluating top-level values ahead of time, so we can store more values in the binary and do fewer heap allocations at runtime) - so if the downside isn't that significant, is the "support arbitrary values with Decode" design best?

Richard Feldman (Sep 07 2022 at 13:03):

but there is a difference there, which is that decoding can fail

Richard Feldman (Sep 07 2022 at 13:03):

e.g. you say "I want to import this .json file and decode it using Json.fromUtf8" but then the file fails to decode because it has a JSON syntax error or something

Richard Feldman (Sep 07 2022 at 13:03):

when/how do you detect and report that decoding error?

Richard Feldman (Sep 07 2022 at 13:04):

in the case of evaluating top-level values, we don't have to consider that because it can't fail

Richard Feldman (Sep 07 2022 at 13:04):

so you could say "we should evaluate it at compile time so we can report any decoding errors involved in imports as a build error, because we don't actually need to run the program to tell you that"

Richard Feldman (Sep 07 2022 at 13:05):

but does that mean that roc check now needs to perform monomorphization, code generation, and some amount of code execution in order to do its job? that's a lot of extra time compared to today where it doesn't need to do any of those!

Richard Feldman (Sep 07 2022 at 13:07):

another possible design is that it could happen only in roc build, but right now we have the invariant that roc build never reports any errors on top of roc check reports, so you're never missing out by running the faster roc check instead; would this change the incentive, such that people would start running the slower roc build all the time, just to make sure they found out about all the errors? (seems likely)

Richard Feldman (Sep 07 2022 at 13:08):

so, putting all that together, I'm inclined towards the "just support importing bytes and strings" design, because even though it's less flexible, it doesn't have these downsides regarding compile times for all programs!

Martin Stewart (Sep 07 2022 at 13:09):

If you support custom decoding at compile time then you essentially have the feature that got discussed here
Qqwy / Marten said:

Maybe at some point we might improve the constant value story in Roc by simultaneously:

• allowing partial functions. (like "either this can be turned into a Nonempty list, or I crash"). The cleanest way might be to introduce an 'absurd' keyword or something.
• but ensuring that they are only allowed to run at compile time (during a constant folding step).

but with the drawback that it's one file per value. So decoding a url string means needing to put the url in a separate file rather than a string literal in the code.

I think I'd start with only supporting bytes and then maybe later add a feature that supports computing arbitrary values at compile time, not just the contents of a file.

Richard Feldman (Sep 07 2022 at 13:12):

there was also another thread about related ideas awhile back: https://roc.zulipchat.com/#narrow/stream/304641-ideas/topic/Compile.20time.20computation/near/258831990

Richard Feldman (Sep 07 2022 at 13:13):

they all have subtly different tradeoffs though

Richard Feldman (Sep 07 2022 at 13:14):

for example:

ensuring that they are only allowed to run at compile time (during a constant folding step)

this means that now we have to do constant folding in dev builds, which we currently don't do, and which would work against the dev build goal of completing as fast as possible

Arya Elfren (Sep 07 2022 at 16:27):

So this would work as a a limited implicit comptime from zig that is only applied to files mentioned in the header?

But then can't you treat import failures like linking failures? So you do them in dev builds but not with check. Does roc check check for errors in execution? Because if you just have a fallible decoder you could treat the imported file as opaque until compilation?

Richard Feldman (Sep 07 2022 at 16:30):

roc check doesn't evaluate any Roc expressions, it just parses and type checks them

Richard Feldman (Sep 07 2022 at 16:30):

if we changed that, it would run significantly slower

Brian Carroll (Sep 07 2022 at 18:00):

I agree that importing bytes is the way to go here. With the way Abilities work, it's pretty trivial to decode that into Roc data. You have to handle a Result but I think that's fair enough.

Anton (Sep 07 2022 at 18:11):

Editor plugins that can convert for example json to a roc record (that you then save to a roc file) would also prevent needing to use a result in many cases.

Brendan Hansknecht (Mar 22 2023 at 17:42):

Poking this old topic because i think it would be really useful for templating ang glue in roc. Makes it easier to just ingest a .zig or .cc file as bytes or a string instead of copying each one into a roc multiline string and losing all tooling related to the original language.

Is the basic form of just importing a file as a string or bytes something we want to add? How would that actually work in practice?

Richard Feldman (Mar 22 2023 at 19:02):

my thinking was something like:

imports [
    "path/to/my/file.txt" as myFile : Str,
    pf.Whatever.{ ...etc },
]

and now you just have myFile in scope and it's a Str. The rule would be that you could only do : Str and : List U8 for these.

Richard Feldman (Mar 22 2023 at 19:04):

I could imagine a hypothetical fancier version where you could give it anything with Decode and then you specify a decoding format and it runs it at compile time, but that seems like a rabbit hole of a discussion :stuck_out_tongue:

Brendan Hansknecht (Mar 22 2023 at 19:16):

yeah, let's save Decode for much in the future.

Joshua Warner (Mar 22 2023 at 19:55):

I'd actually love to be able to include a whole directory that way, e.g. for a semi-static site generator

Brendan Hansknecht (Mar 22 2023 at 20:32):

That would be interesting at some point. Though at some point I guess you have to question if something like a static site generator should just pay the cost of reading the file once from disk.

Brendan Hansknecht (Mar 22 2023 at 20:32):

though something like "path/to/my/" as myDir : Dict Str Str or using a record or list of k v tuples would be quite interesting.

Richard Feldman (Mar 22 2023 at 21:12):

ha, maybe! But yeah I think it's enough of a project to support List U8 and Str, which are the two I feel confident we want :big_smile:

Brendan Hansknecht (Mar 22 2023 at 21:26):

Yeah 100% agree. Those other ideas are interesting for maybe a future project.

Brendan Hansknecht (Mar 22 2023 at 21:27):

I just want to add something like:

    /// e.g "path/to/my/file.txt" as myFile : Str
    IngestedFile(&'a str, TypedIdent<'a>),

Brendan Hansknecht (Mar 22 2023 at 21:27):

That said, a lot of this is code that is very unfamiliar to me.

Brendan Hansknecht (Mar 22 2023 at 21:28):

Haven't touched the parser or imports at all. I think for imports, there are some assumption that will now be broken given this tag has no module.

Last updated: Jun 16 2026 at 16:19 UTC