Stream: ideas

Topic: custom records - public fields, private constructor

Richard Feldman (Jan 16 2025 at 23:22):

a potentially nice use for custom records would be if they had the following design:

• all fields are always public no matter what (this turns out to be a hard design constraint, because otherwise type annotations might be required for code to work correctly, as we saw in #ideas > custom types - inferring record types)
• however, constructing a new instance of the custom record (as well as exhaustive destructuring) would only be allowed in the same scope where the custom record is defined (so, the same module)
• to make constructing them easier in other modules, we can support having gen auto-generating a function named new for you, which takes a structural record of the same shape as the custom record

the scenario this would make nice is the one we can see in roc-lang/http, where backwards compatibility is in tension with ergonomics.

in that module we essentially have a record (representing a Request) that we could easily choose to represent as a structural record (because all of the information in it is public), but if we do that, then every time in the future that we add a field to that record, it's unavoidably a breaking change.

Richard Feldman (Jan 16 2025 at 23:22):

the reason it's unavoidably a breaking change is that anywhere someone is constructing one of those structural records, they will suddenly be missing the new field(s), and will get type mismatches

Richard Feldman (Jan 16 2025 at 23:23):

this module currently addresses this by having a getter or setter function for every field on the record, which really clutters up the public API and also creates a bunch of annoying shadowing situations within the module

Richard Feldman (Jan 16 2025 at 23:24):

but it does solve the problem

Richard Feldman (Jan 16 2025 at 23:24):

the idea here is that we could sort of formalize this scenario

Richard Feldman (Jan 16 2025 at 23:24):

and say that a custom record can only be built from scratch from within the module, and outside the module you're stuck calling a function exposed by that module if you want to get ahold of one of them

Richard Feldman (Jan 16 2025 at 23:25):

but once you have it, you can do all the usual record things with it - read its fields, pattern match on them (although not exhaustively), update them, etc.

Richard Feldman (Jan 16 2025 at 23:25):

so that way, Request can add new fields as a nonbreaking change

Richard Feldman (Jan 16 2025 at 23:26):

because if you have a code base that updates to the new version, nothing breaks because you aren't trying to create one from scratch (which requires specifying every field, including the ones your code base didn't know about because they didn't exist at the time) or to destructure it exhaustively (which also would have required knowing about fields that didn't exist at the time)

Richard Feldman (Jan 16 2025 at 23:27):

however, your calls to Request.from_method : Method -> Request (which is how you actually create one of these) continue to work, because the module will have updated the implementation of that function to include the new fields

Richard Feldman (Jan 16 2025 at 23:28):

finally, in this design if you do actually want a "nominal record" where you can specify all the fields during construction, we can still support that use case easily by having gen support auto-generating a function named new (or init or whatever) which takes a structural record of the same shape, and then returns the custom record

Richard Feldman (Jan 16 2025 at 23:28):

so if you were using that, instead of it being MyRecord.{ field1, field2, ... } it would be MyRecord.new({ field1, field2, ... }) - so almost identical ergonomics

Richard Feldman (Jan 16 2025 at 23:29):

exhaustive destructuring still wouldn't work, but we already don't support that, so doesn't seem like the end of the world :sweat_smile:

Richard Feldman (Jan 16 2025 at 23:30):

I guess alternatively to the gen new idea, if we're having the concept of "custom tag unions either do or don't support their tags being constructed outside the module" we could offer the same thing for records

Richard Feldman (Jan 16 2025 at 23:31):

and you'd choose that option (or not) based on whether you want backwards compatibility vs. exhaustive destructuring

Isaac Van Doren (Jan 16 2025 at 23:38):

Would this be a good enough solution in the HTTP case? It would mean we could add new fields to the types without breaking, but we could never change the type of an existing field, rename it, or remove a field. It seems to me that in many cases if you're worried about backwards compatibility you might want to hide more implementation details than a custom record solution like this would allow.

Richard Feldman (Jan 16 2025 at 23:43):

that's already true though

Richard Feldman (Jan 16 2025 at 23:44):

in that if I change a field's type, that implies changing the getter and setter for it accordingly

Isaac Van Doren (Jan 16 2025 at 23:47):

Not necessarily, you could change the internal representation of the data to use different types but still expose the same public API with an opaque type

Isaac Van Doren (Jan 16 2025 at 23:47):

I don’t know that it would come up with the HTTP example, but it certainly could other places

Anthony Bullard (Jan 16 2025 at 23:53):

I'm seriously not trolling, but would "hanging functions off" the custom type be an option here? I.e, constructors are just module functions, but all "methods" are declared using some syntax that "attaches" it to the definition of the type?

Here's a only-for-bikeshedding example (and feel free to move this out of this thread if it's just noise or unhelpful):

MyRecord := {
    field1 : Str,
    field2: U64,
} with
    totally_not_a_method = |self| { self & field2: self.field2 + 1 }

or even

MyRecord := {
    field1 : Str,
    field2: U64,
} with
    self.totally_not_a_method = || { self & field2: self.field2 + 1 }

These aren't methods, I know! They are "custom-type associated functions".

Anthony Bullard (Jan 16 2025 at 23:54):

And then maybe you don't need the module(a).blah : ... syntax for where statements

Anthony Bullard (Jan 16 2025 at 23:56):

Just where a.blah : ...

Anthony Bullard (Jan 16 2025 at 23:57):

Another idea (going Lua here)

MyRecord := {
    field1 : Str,
    field2: U64,
}

MyRecord.totally_not_a_method = |self| { self & field2: self.field2 + 1 }

Sam Mohr (Jan 17 2025 at 01:56):

This idea sounds good to me, except for gen being a compiler-generated function. It would be easy for custom record authors to just rely on that and get easy breakage all over again.

We could take a page from the Gleam book and have a code action to generate the method def:

MyRecord := {
    foo: Str,
    bar: Bar, # another custom type
}

# generated with a "Generate `new` method" action
new = |{ foo, bar }|
    MyRecord.{
        foo,
        bar,
    }

Sam Mohr (Jan 17 2025 at 01:58):

Basically, it's pretty important that we have custom records, and even if we don't do something like the above, that's fine

Anthony Bullard (Jan 17 2025 at 01:58):

I mean LISP has done pretty well without opaque types.

Anthony Bullard (Jan 17 2025 at 01:58):

/me ducks

Sam Mohr (Jan 17 2025 at 01:59):

Does lisp have a good FFI story?

Anthony Bullard (Jan 17 2025 at 01:59):

Well opaques across a FFI is different than a language facility for defining one

Anthony Bullard (Jan 17 2025 at 02:00):

And Clojure does :stuck_out_tongue:

Sam Mohr (Jan 17 2025 at 02:01):

I'll check it out

Richard Feldman (Jan 17 2025 at 02:44):

Isaac Van Doren said:

Not necessarily, you could change the internal representation of the data to use different types but still expose the same public API with an opaque type

I've heard this is an important use case since 1990s and the number of times I've seen it come up in practice across my entire career has been 0 and counting :sweat_smile:

Richard Feldman (Jan 17 2025 at 02:46):

to clarify, that's for things that have fields on them

Richard Feldman (Jan 17 2025 at 02:46):

for collections, sure :thumbs_up:

Richard Feldman (Jan 17 2025 at 02:46):

but I'd just make them opaque with no public fields anyway

Richard Feldman (Jan 17 2025 at 02:48):

but I've never seen the like "well what if we want to change something from being a public field to instead storing something different internally but then provide backwards compatibility through a new method which computes the old thing as a calculation on the new internal representation?" use case actually come up in practice

Anthony Bullard (Jan 17 2025 at 03:03):

Richard Feldman said:

but I've never seen the like "well what if we want to change something from being a public field to instead storing something different internally but then provide backwards compatibility through a new method which computes the old thing as a calculation on the new internal representation?" use case actually come up in practice

I've had to do this before. It was miserable. 1/5 stars. Do not recommend.

Richard Feldman (Jan 17 2025 at 03:04):

oh? what was miserable about it?

Anthony Bullard (Jan 17 2025 at 03:05):

Because sometimes that calculation is non-trivial or meant you had to keep data around in the data structure that isn't pertinent anymore

Anthony Bullard (Jan 17 2025 at 03:05):

And sometimes you had to just throw an exception for some cases because we just could do it anymore

Anthony Bullard (Jan 17 2025 at 03:06):

It _should_ have just been versioned

Anthony Bullard (Jan 17 2025 at 03:06):

And had a strategy to handle the different versions and/or convert from one version to another

Joshua Warner (Jan 17 2025 at 04:34):

When I was first reading this, the usecase I was imagining was being able to have the constructor private, such that you can make strong guarantees that must be true of any instance of that type

Joshua Warner (Jan 17 2025 at 04:35):

Unfortunately if you can _update_ the fields, that no longer works

Joshua Warner (Jan 17 2025 at 04:36):

(I mean, update the fields in an unconstrained maner from outside the module, without going thru the same validation logic again)

Richard Feldman (Jan 17 2025 at 04:46):

yeah we can't support that for the same reason that we can't support private fields: it would break principal decidable type inference

Joshua Warner (Jan 17 2025 at 04:52):

I didn't follow that argument, could you give an example of something that would break?

Joshua Warner (Jan 17 2025 at 04:54):

This is what I was imagining: the "plain" record update syntax _never_ means a custom record, and MyRecord.{foo & a: 1, b: 2 } always means that (but requires that code be written in the same module)

Joshua Warner (Jan 17 2025 at 04:56):

Then, take a function accessing a field from that, but without a type annotation:

foo = |myrecord| myrecord.a + myrecord.b

The type of that would be inferred to be {a: Num, b: Num} -> Num

Joshua Warner (Jan 17 2025 at 04:57):

We'd make it so record types could always unify with custom record types

Joshua Warner (Jan 17 2025 at 04:59):

(i.e. custom record types are a refinement of plain record types, in the same way that {a: Num, b: Num}{c: Num} is a refinement of {a: Num, b: Num}o)

Joshua Warner (Jan 17 2025 at 05:04):

You could try to sneak past the requirement that such a custom record must be constructed in the module by (in another module) forcing your plain record literal to be unified with a custom record, e.g.

val = if Bool.true then
   { a: 1, b: 2}
 else
   MyRecord.new(42, 43)

... but we can easily recognize that as an attempt to construct a custom record outside of the proper module (post-type-checking)

Joshua Warner (Jan 17 2025 at 05:06):

In the type checker, records would have a new variable that needs to be solved for, the 'custom record name' - which unless constrained would just be a variable (and that'd be fine)

Richard Feldman (Jan 19 2025 at 15:30):

random thought that just occurred to me: even if people used nominal records for absolutely every data type (which I don't think is likely, but hypothetically), structural records would probably still be worth having just for record builders alone :big_smile:

Kilian Vounckx (Jan 19 2025 at 16:03):

Could you make a nominal record builder?
Like

NominalName.{ combine <-
    foo: bar,
    foo2: bar2,
}

Richard Feldman (Jan 19 2025 at 16:27):

I suppose, but then you'd have to define the type up front even though you'd only be using it in one place

Richard Feldman (Jan 19 2025 at 16:28):

that's a recurring annoyance for me in Rust, e.g. when I want a "named arguments" equivalent (also a place where structural records are nice outside of data modeling)

Anthony Bullard (Jan 19 2025 at 18:20):

Structural records (with decidable type inference) are - more than just about anything else - the core of Roc to me

Brendan Hansknecht (Jan 19 2025 at 18:30):

They are super convenient.

Brendan Hansknecht (Jan 19 2025 at 18:32):

Though given that I type all functions eventually, probably not actually that important. Nominal types with type interference would be about the same but more verbose

Anthony Bullard (Jan 19 2025 at 18:38):

Yeah, it feels lovely to just hit a problem without even thinking about the types, letting the compiler guide me, and then concretizing the types after the fact

Anthony Bullard (Jan 19 2025 at 18:41):

Not exactly the same but inferable collections of tags is also killer

Brendan Hansknecht (Jan 19 2025 at 18:42):

Yeah, 100%

Richard Feldman (Jan 19 2025 at 20:48):

Anthony Bullard said:

Structural records (with decidable type inference) are - more than just about anything else - the core of Roc to me

I think they're surper useful for the above use cases plus scripting, but I'm surprised you'd put them that high!

in a bigger code base, where you're choosing to write out the types of structural records with type aliases anyway (just to avoid writing out the type every time in annotations), do you still see them as very different from nominal records?

Anthony Bullard (Jan 19 2025 at 22:47):

Probably not, because at that point you have named them. But I haven't built a real piece of software with Roc yet so I don't know how I would approach things. I know it's great to not have to use a named constructor to build a record everytime but I still get the benefit of typechecking

Sam Mohr (Jan 19 2025 at 22:52):

The "named args" thing is a really strong want for me in Rust when you have unobvious args at the callsite, e.g. check_value(true, 123). What do those mean? A disciplined dev will name their vars to make the callsite, but using a record means you don't have to trust the caller, they need to give context to call the function

Sam Mohr (Jan 19 2025 at 22:53):

And if the caller doesn't want to have to label everything, they can just named the args based on the field names, and punning will make the callsite about as long as it'd be sans record

Anthony Bullard (Jan 19 2025 at 22:53):

Yeah, using a record (especially with optional fields) as a way to force named args is really nice. And it even reminds me of the Dart syntax for specifying named args. And between Roc and LLVM it should have little to no overhead

Sam Mohr (Jan 19 2025 at 22:54):

Since records on their own are all stack-based, it's a "zero-cost abstraction"

Anthony Bullard (Jan 19 2025 at 22:57):

Yeah, but LLVM can sometimes even do struct unpacking if it's used as a function arg

Anthony Bullard (Jan 19 2025 at 22:57):

Not always, but sometimes

Anthony Bullard (Jan 19 2025 at 22:57):

I'm sure Brendan knows the exact rules for it

Brendan Hansknecht (Jan 19 2025 at 22:59):

Llvm only does it with structs that contain 2 register sized elements. So very limited. That said, it will do it if only 2 fields are used from a larger struct and they are both register sized

Brendan Hansknecht (Jan 19 2025 at 23:00):

Otherwise, it is all just const ref on the stack

Anthony Bullard (Jan 19 2025 at 23:02):

So either way, not much of an impact

Isaac Van Doren (Jan 19 2025 at 23:04):

Structural record are very close to the core of Roc for me also. I suspect in a large system I would use nominal records for the "this is an important part of my domain" kind of types and structural records for many other instances where I just want to group some fields together in a one off way to get a job done. In Java I'm constantly frustrated by having to define and name records that are only used in one or two places for some simple intermediate purpose.

Brendan Hansknecht (Jan 19 2025 at 23:06):

I guess if we had named args and default valued args, I probably would be fine without structural records in the long run....but I probably would still miss them for quick prototyping

Sam Mohr (Jan 19 2025 at 23:13):

I think we'd still want them for the same reason we want tuples: structural records serve many functions (pun somewhat intended) that we'd probably need multiple, smaller features to replace them, so it's better to have a single, simple, obvious feature than four little helper features (e.g. named args, etc)

Sam Mohr (Jan 19 2025 at 23:14):

This mirrors how tuples solve both matching on multiple local values and allowing multiple return values without needing two discrete language features

Richard Feldman (Jan 19 2025 at 23:45):

yeah although I think the benefit of structural records is much bigger than tuples :big_smile:

Last updated: Jun 16 2026 at 16:19 UTC