https://github.com/roc-lang/roc/issues/11594
@Richard Feldman, could you summarize some of the downsides and the "can of worms"? I don't have any clue, since it works so great in Zig. Maybe because Zig's type system is simpler?
The lack of this seriously limits the expressivity for some ideas, or require very complex workarounds. If I understood the downsides, maybe we could figure out some solution that doesn't have those downsides, yet enable all the use cases. Not necessarily a 1:1 port of the Zig builtin, just something that enables more concise and readable code.
sure! so here are some of the downsides I've seen in "expose a representation of the type as a value" in other languages:
foo whose type is Str, and also I can use that type info to access the value of that foo field at runtime), that means that module boundaries are no longer strictly enforced like they are today, because there's a workaround you can use to read the "private" data from any value anywhere.|a| typeinfo a without introducing a new type for "a type variable whose typeinfo can be accessed from this scope"a -> a has only one implementation. if you can get the type info of a as a value, you can make that function return a different value depending on what you pass in.List(a) -> List(a) would not longer tell you as much as it does today.these are just the first ones that come to mind; there are others, but this is an example of the can of worms I am reluctant to open :sweat_smile:
This is indeed quite problematic. But it's also quite different from how I imagined it.
I have some ideas that could work, but I want to get a full picture first, so these are not necessarily my main idea, I'm just curious.
What if we restricted reflection to only the public interfaces of each type? Only those that are safe to access anywhere.
StructInfo := {
fields: [Opaque, List(StructField)]
methods: List(StructMethod)
}
I know this does not solve the other problems, e.g. parametricity, but could it solve the visibility problem?
Most of the use cases would be nominal and structural types anyways. I can't think of any use case where reflection of inner fields of opaque types would be useful.
that could work, and a solution to the parametricity problem could be if you have a method called like type_info which returns that
and a solution to the parametricity problem could be if you have a method called like
type_infowhich returns that
Could you provide an example of this?
basically like encoder_for today except it returns TypeInfo
I am thinking about maybe restricting reflection into special functions, which must be called with special syntax, just like effectful functions have their own syntax.
Something like gen_record@(my_schema) (I know this is not the best sigil.)
so a -> a works the same way as today, and if you want to do reflection you'd need to add where [a.type_info : () -> TypeInfo]
This also seems good.
so then it would be reflected in the type if you're trying to do reflection on a type variable
the reason I mentioned in the issue that it makes sense to start from use cases is that at this point we basically have a different encoder_for API :smile:
with a different name, and maybe the API differences are relevant
but that depends on the use case of course
This seems good. It has the additional benefit of explicitly enabling/disabling reflection for a type.
what use cases would this enable that we can't do today?
It has the additional benefit of explicitly enabling/disabling reflection for a type.
The downside is that it's not available for structural types, which cover a significant chunk of use cases.
what use cases would this enable that we can't do today?
A few:
roc-pg. However, SurrealDB has a much more powerful data model, e.g. it supports type unions in the database (but it's not tagged, so it can not use the the official encoder API), and returns nested data in CBOR, compared to the row+column output of Postgres.There are much more, but since it's not currently possible, I instantly discard any idea that could be done with it.
Some of these use cases that are currently possible in a limited way, but this would make them much-much easier and powerful. Reflection in itself is not that much of a tool (except for API endpoint type generation for Elm/TypeScript), but pairing it with field access and maybe with an upgraded struct builder would unlock most of the use cases.
One example is to allow defining persisted objects as arbitrary structs, and automatically enabling constraints for every field that has a ForeignKey type.
A hacky workaround is possible by creating a fake parser, and parsing a fake item. But it's much more complex and fragile, and if we try to shoehorn solutions to APIs that weren't designed for this, it'll be buggy and quickly hit limitations.
On the other hand, if a sufficiently powerful reflection API is added, it could replace the encoder/parser APIs, since those could be implemented in userspace.
And maybe the struct builder too.
When I need some solution like this, I always evaluate some hacky workarounds based on the encoder and the struct builder API, but they are mostly complex and fragile, and usually I hit a limitation. I think that users are going to build hacks around these APIs, which (in my opinion) will be harmful to the Roc ecosystem in the long term.
And importantly: these hacks usually require much more time. That time could be spent on building actually useful things, not on working around artificial limitations.
Some more brainstorming, just a dump of probably bad ideas (but possibly some inspiration to someone). Some are use cases, some are possible solutions:
combine function.Here's an example that's not possible currently in a clean way.
This is a coarse concept for an object persistence library:
Post := {
title: Str,
content: Str,
}
User := {
username: Str,
# Incorrect, should be List(RecordId(Post))
posts: List(Post),
}
Database := PersistenceEngine({
users: User,
posts: Post,
})
If a type has a table, it MUST be linked from other records via RecordId(type), and it's not possible to store the data inline. (Any other supported type is inlined.) Comptime checking would instruct the user to change it to a RecordId(Post) in the above example.
This kind of feature would make an (all-in-Roc) ECS much easier to implement. Today, the platform (or something associated with it) would need to take a list of types that are considered 'components' and generate a new 'world' type composed of them. I had also been considering ways of trying to abuse the encoder machinery for this, but I think it might end up not being sufficient, and if it did work it would boil down to a hacky version of this API. Essentially we run into the same problems as the database client, I think (component = column, world = table).
Having reflection also unlocks more ability for platforms to interact with existing native ECS libraries like flecs or bevy. To use these, the platform fundamentally needs some way to get type-level information to the native library without recompiling the platform.
The ability to do something like
Ok(val) = some_validation(SomeType.reflect_info())
is both very powerful and potentially a bit spooky. Like, if it's not dependent types it's certainly getting closer.
Another use case is writing type-safe migrations for databases. E.g. AddField({name: "posts", type: List(RecordId(User)) }), then the migration checker woud check if applying all the migration history would match the current schema.
Again, could be solved with a fake encoder. Partially. Then possibly find some cases where it's not possible to comptime check via the encoder API, because it's not flexible enough, then resorting to runtime crashes.
notramo said:
It has the additional benefit of explicitly enabling/disabling reflection for a type.
The downside is that it's not available for structural types, which cover a significant chunk of use cases.
we could infer it for those, like we do encode_for
what are the specific features you'd need that you don't have today?
Str, and maybe the type based on a comptime value.These are the goals, not the mechanisms. Could be achieved with multiple approaches.
I have multiple ideas for much simpler, yet much more powerful APIs, but I'm not that familiar with the compiler internals to figure out a concrete solution.
I'll think through what I learned today in this chat, and try to come up with more specific ideas.
like if we had something like:
Type := [
Opaque,
Record(List(RecordField)),
TagUnion(List(Tag)),
]
RecordField := {
name : Str,
type : Type,
}
Tag := {
name : Str,
payloads : List(Type),
}
...that wouldn't be enough because you couldn't:
TypeType into a valueis that it, or are there other missing pieces there?
This could be good for the reflection part, but yes, for most use cases, additional solutions need to be combined with this.
One note about programmatic field access: since the fields can have different types, the field name must be fully comptime, because there are no anonymous unions in Roc.
yeah I think the way that would have to work is something like it gives you back a Try and then for comptime values you can pattern match on Ok exhaustively because it always succeeds
It could “desugar” to a .field access synax.
I'd want to try to avoid introducing new language concepts with this
and just do it all with ordinary types and functions, so the only compiler magic is auto-deriving them like we do with the encode/decode functions today
Zig requires the to be comptime, so basically it needs inline for for iteration, which generates a separate instance of the function call.
yeah, I would explicitly try hard to not introduce a concept of "this only works at comptime" for this :smile:
Isn't the encoder API comptime?
the compiler does not currently have a concept of "you can only use this at comptime" for any language features
including all the current APIs
and I would want to preserve that property
Seems good. The problem could be that it introduces additional runtime logic which makes the resulting binary bigger.
Because runtime type info needs to be carried.
I don't know whether Roc already includes it, or how much overhead would it add.
right, but if you only use it at compile time, you wouldn't pay anything at runtime
because of dead code elimination as usual
yeah I think the way that would have to work is something like it gives you back a
Tryand then for comptime values you can pattern match onOkexhaustively because it always succeeds
Seems good. How would it look with an example? Would it be a derived method? Or a builtin?
one option could be something like:
RocVal := [
Opaque,
Bool(Bool),
Str(Str),
U8(U8),
# (the other number types go here)
Record(Dict(Str, RocVal)),
TagUnion(Dict(Str, List(RocVal)),
]
To address the parametricity problem, it could be a method, e.g. get_field, just like the type info.
and then you have a method like .to_roc_val() that can be auto-derived, which returns one of these representations of self
so then you'd access the fields by using .get on the Dict, which would return a Try, and at comptime you could exhaustively match on Ok etc.
So it would work for every type, not just for structs?
Would it be also possible to reverse this? E.g. a RocVal could be converted to a type i.e. move from the value into the type that the type system uses?
yeah I'd think any reflection API would work on anything but opaque types :smile:
yeah we could have a RocVal.type()
How would the type system access it? It seems tricky.
oh sorry I misunderstood
I mean we could have RocVal.type : RocVal -> Type
what would moving that into the actual type system be for? :thinking:
What wold Type represent?
this Type: #ideas > Type reflection — Zig `@typeInfo()` equivalent @ 💬
from earlier
what would moving that into the actual type system be for? :thinking:
Like @Struct and @Int and others in Zig.
oh, and also we'd need something like a from_roc_val (also auto-derived) for all the things that have to_roc_val
so that you could construct them from one of these
and it would of course also have to return a Try
which you could exhaustively match at comptime
what would moving that into the actual type system be for? :thinking:
Sorry, I was already thinking about the struct type creation. (Creating a struct with a field from a Str.)
I worry it would require a "comptime only" feature.
I think from_roc_val would cover that
like what's a concrete example of how you would use that struct once it had been created?
I mean, some use cases like zig-clap.
Have a DSL, which creates the types.
I think the way that would work in Roc is the other way around
A workaround is possible by inferring the result type from later use (or an annotation), then using the field access API.
first, type inference would infer the structural types based on how you use them in your program
Yes, it seems good. The only problem is lack of autocompletion.
and then you can call to_roc_val or type_info on those inferred types to get whatever you want
Would it be possible to have a set_field builtin method? Also returning a Try?
Then I think these APIs would cover all of my use cases.
we could add conveniences on top of the above API
I don't think that would introduce any new capabilities that the Dict doesn't already cover
or to put it another way, you could already implement set_field in userspace on top of the above
it would just be a matter of whether we offered an auto-derive for it as a convenience so you didn't have to write it yourself :smile:
to set expectations, I'm serious about the feature freeze for 0.1.0, so this would have to be after that, but I'm open to trying it out
This seems good. I think these APIs would cover all my use cases. And it could replace the struct builder and the encoder APIs too, so those can be implemented in userspace.
it doesn't seem like it would break any current guarantees, because it's so similar to encoder_for etc., and if it can unlock more use cases without breaking guarantees, I'm interested in exploring it!
It would be easier for beginners too: there's only one API, and it can do everything (as opposed to the current encoder/parser/struct builder, each limited to separate use cases).
@Dan G Knutson (and anyone else who has use cases in this area that aren't currently implementable) I'm curious to hear your thoughts!
@Richard Feldman, thanks for considering this idea and taking the time to discuss!
thanks for proposing it! :raised_hands:
I'll need to look back at what bevy and flecs would need. And maybe try a pure roc one. I've been in rendering world for a while and haven't been thinking about ecs stuff. Really excited reflection's on the table though!
This could benefit from a throwaway experiment or spike to try out different ideas -- we wouldn't upstream anything until after 0.1 as it's a low priority. But I assume if someone wanted to mess around or try out different ideas these would be helpful to inform a future design proposal or discussion.
3 messages were moved from this topic to #ideas > reifying function bodies as Roc values by Richard Feldman.
Richard Feldman said:
Dan G Knutson (and anyone else who has use cases in this area that aren't currently implementable) I'm curious to hear your thoughts!
struct-of-array (SoA) from a record is a likely contender use case.
Something along the lines of
Particle := {
x : F32,
y : F32,
vx : F32,
vy : F32,
}
ParticleSoA := SoA(Particle)
particles : ParticleSoA
particles = SoA.append(particles, {
x: 1.0,
y: 2.0,
vx: 0.5,
vy: 0.3,
})
Reflection could convert an arbitrary record into a Dict or List of field/value.
This would help the HTML DSL use case. For example in Joy :
https://github.com/niclas-ahden/joy-todomvc/blob/main/app.roc#L156
header(
[class("header")],
[
h1([], [text("todos")]),
input([
class("new-todo"),
placeholder("What needs to be done?"),
value(model.draft),
on_input(|s| UserTypedDraft(s)),
]),
],
)
could become
header(
{ class: "header" },
[
h1({}, [text("todos")]),
input({
class: "new-todo",
placeholder: "What needs to be done?",
value: model.draft,
on_input: |s| UserTypedDraft(s),
}),
],
)
which is closer to HTML and would avoid a bunch of user imports (class(), placeholder(), value(), on_input() which are here to normalize the attribute types) while remaining type checked.
I think, reflection could make HTML templating even simpler. A comptime-parsed string with a custom templating syntax. It could acces the corresponding struct fields and comptime check them.
on the other hand with reflection you can get formFor model as library independent from the type
Can you provide an example of it?
laravel admin panel comes to mind for example, I think Rails also let's you basically say this is an input for type post and just generate the whole thing
it's even better if the source doesn't have to be a roc file or type
like api_client = ApiClient.from_spec("openapi_spec.json") or same but for graphql schema, both with hot reload
so, another use case of this came to mind :)
https://www.qt.io/development/qt-framework/qt-bridges so Qt made some changes so that it's easier to write the backends for QML in other languages.
I think Roc is a great fit - Qt is already built around signals, slots (messages), reactive state and so on. However, to make the editor experience nice, you have to generate a schema file
This should be doable for a package/platform author
I was experimenting with API close to
Counter := {
count: Int32 ? 0,
comment: Str ? "",
since_save: Int32 ? 0
}.{
inc = |{state, ..}, message| {
{ count: state.count + 1, since_save: state.since_save+1, state..}, []
}
inc = |{state, ..}, {}| {
result = { count: state.count -1, since_save: state.since_save+1, state..}
signals = if result.count == 0 {
[CounterAtZero]
} else { [] }
result, signals
}
save! = |{state, logger, ..}, _| {
Sqlite.write(write_query, {counter : state.counter, comment: state.comment}).tee_error(|err| logger.log_error(err, "Save unsuccessful)?
logger.log_information("Saved counter")
{ count: state.count -1, since_save: state.since_save+1, comment: ""}, [CounterSaved(state.count))]
}
qt = {
name: "Counter",
public_props: |{counter, since_save| { current: counter, is_dirty: since_save > 0},
slots: [
Increment => Slot(inc), # you should be able to rename these, so a map is nice
Decrement => Slot(dec),
Save => SlotEffect(save!)
].to_map(), # or better define the map inline, but I can't remember the syntax now
property_setters: |state| {
[state.comment] # this would be nicer with the "reification" feature
},
handle_errors: |ctx, error| { # exhaustive list of errors from effectful slots
Sqlite.Write => {
ctx.slot(Save) # just a stupid exmaple of a retry, because why not
Ok(ctx.state, [])
},
..others => Err(others) # pass forward
}
}
}
then the QML can use the slots, subsribe to signals, the slots and signals auto infer, and are type safe, all the unhandled errors from all components bubble up all the way to main and so on.
For QML editor, I would need to generate a schema based on those inferred types, including listing tags from the tag union and so on
the above is just a first pass, don't too attached to that :D the issue is the same regardless of the API
Last updated: Nov 04 2026 at 18:54 UTC