I have been working on a version of this for a while - in the form of reification - representing the function body as value.
The design: if a function accepts a Code(f) argument, you can pass finto it, where f is a type of any pure Roc function expression. The compiler passes into this argument a Typed AST value that can be translated as necessary within Roc. The function passed can be polymorphic, and the types inside are resolved just like they would in a normal function call site. What you translate the AST into is your business - could be strings for SQL, could be effectful platform calls.
It is beautiful IMO, it unlocks various DSLs that have no syntax overhead - imagine:
# main.roc: the rows are ordinary records, the clauses ordinary Roc functions
Product : { id : I64, title : Str, price : Dec }
products : Sql.Table(Product)
products = Sql.table("products")
is_premium : Product -> Bool
is_premium = |p| p.price >= 20
expect is_premium({ id: 1, title: "Espresso machine", price: 249 })
premium : Try(Str, Sql.Error)
premium = Sql.from(products).filter(is_premium).select(|p| { title: p.title }).to_sql()
# Ok("SELECT products.title AS title\nFROM products\nWHERE (products.price >= 20.0)")
Notice how transforms and predicates are pure Roc and operate on plain Roc struct, yet the result is domain-specific.
Since there's this discussion, I'll publish the prototype in a couple of days - there are some features I wanted to polish first.
line_total : Order, Product -> Dec
line_total = |order, product| product.price * order.quantity.to_dec() * (1 - order.discount)
expect line_total({ id: 1, customer_id: 1, product_id: 1, quantity: 4, discount: 0.25 }, { id: 1, title: "Pen", price: 2.50 }) == 7.5
report_line = |((c, o), p)| { customer: c.name, product: p.title, quantity: o.quantity, total: line_total(o, p) }
purchases =
Sql.from(customers)
.join(orders, |c, o| o.customer_id == c.id)
.join(products, |(_, o), p| o.product_id == p.id)
bulk_lines =
purchases
.filter(bulk_purchase)
.select(report_line)
.order_by(|r| r.total, Desc)
works too
This design only introduces the tag union that describes pure roc and the Code(f) rule for arguments - no new syntax.
3 messages were moved here from #ideas > Type reflection — Zig `@typeInfo()` equivalent by Richard Feldman.
this is a very different idea from #ideas > Type reflection — Zig `@typeInfo()` equivalent so I moved it to its own thread :smile:
Sql.from(products).filter(is_premium).select(|p| { title: p.title }).to_sql()
Isn't solvable with a fluent builder interface? Literally this same syntax, but just using normal Roc values, instead of analysing the function body.
If I write something in this DSL, how do I know that it is not going to run like normal Roc code? The behavior is different than if it were running as Roc, and I guess the type inference is also different.
The types are inferred based on the usual Roc rules - the question is how these types are translated. The idea is the Translator function preserves the semantics of the type - e.g. uses the correct implementation for integer and float divisions, uses appropriate target types for u32 and u64 Roc values etc. This is a pretty basic expectation IMO - if a translator does not respect Roc types it is malformed. I imagine most translators won't implement all types abd primitives; in this case the translator rejects the AST node, and the user gets a nice error with clear code location
notramo said:
Isn't solvable with a fluent builder interface?
This isn't about the builder pattern, that is just for showcase. It's about the functions that, in this example, get passed into builder methods - they are plain Roc. Some of this UX can be achieved with Static Dispatch, but it doesn't come close - the most obvious example are and and or - the static dispatch can't overload these because they have to return Bool. And the code that uses Static Dispatch wraps everything in custom types that don't work directly with plain Roc
notramo said:
Ihow do I know that it is not going to run like normal Roc code?
In the current design this is encoded in the functions you call - in the above examples you just know that .select, .join, .filter reify (because of Code(a) in their type signature), because you are using this SQL DSL package based on reifying.
You could wrap the code that you want to reify with e.g. .filter(reify_code(is_premium)) , but I think it defeats the ergonomics benefits.
To be clear, the code that gets reified is not DSL - it is Roc - the DSL is the function that reifies it. The semantics you express are the Roc semantics, and a well-written DSL (that uses translators that process the AST, resulting from reification) should respect them.
It is not a novel idea - there are similar features in C# (System.Linq.Expressions) and Julia (Quoting), and reification is considered a great feature in both.
Often reification has downsides like loss of types or patchy language coverage, but the beauty of Roc's type system allows for strict typing through-and-through and a clean boundary - all pure Roc (whose types can be fully resolved at the call site) being supported for reifying.
so the new magic in your example is deconstructing the body of is_premium (or the join lambdas in the second example) to convert it to a sql string in a type-safe way?
Exactly
so Code(is_premium) produces a typed AST in a form similar to ?
Function(
[{ name: "p", type: Product }],
GreaterThanOrEqual(
Field(Var("p"), "price"),
DecLiteral(20),
),
)
which corresponds to
is_premium : Product -> Bool
is_premium = |p| p.price >= 20
What you translate the AST into is your business - could be strings for SQL, could be effectful platform calls
Do you translate to another Typed AST then back to code or directly to roc code ?
so
Code(is_premium)produces a typed AST in a form similar to
Basically, only the substitution happens when the function expression is passed into a matching Code(a) slot - no explicit Code(value) call - Code stays type-level.
In the example filter has type filter : Query(row), Code(row -> Bool) -> Query(row), where Query is the usual Roc type that hosts the builder pattern methods and "state".
is_premium's Product fits into the polymorphic slot of the Code(row -> Bool) type, so is accepted and reified into the typed AST - every field name, value type etc. is known - resolved as in a simple Roc call.
The AST is not translated "back" to Roc. The "Translator" - the Roc code that received the Code(a) value and matches/processes the recursive AST tag union - builds whatever representation it wants, or, again, does some effectful calls if it needs to, making the return value a Task. There is no mechanism to get the compiled Roc back from the reified value. You can of course call the function you reify in one place normally in a different place - the reification happens at call site of the reifier function (that triggers the reifying because of the Code in it's type declaration), with types resolved there.
You could pass some toInt : a -> I32 into Code(Str -> a) and have that resolved correctly, and turned into the AST.
oh i see !
does some effectful calls if it needs to
doesn't this break some compile time garanties of roc ?
No, in this case it's simply a translator : Code(a) -> Task whatever. You just get an AST value and do what you want with it
You can now try reification through building the compiler fork at https://github.com/Karakatiza666/roc/tree/reify
It is rebased on latest Roc main. You can consult multiple related docs. Firstly, an overview: docs/reifying-overview.md.
Examples:
examples/reifying/
More docs:
The design - roc-reify-design.md, roc-reify-proposal.md, the user guide - docs/langref/reifying.md
https://learn.microsoft.com/en-us/dotnet/csharp/advanced-topics/expression-trees/
this is exactly how it works in C# and if Roc gets this, the options are limitless!
The example with database is great, EF Core (the dotnet ORM) is amazing.
https://entitygraphql.github.io/library-compatibility/entity-framework/ for example, you can get something like appBuilder.AddGraphQL<DbContext>(), autogenerated with paging, sorting, filterting and so on
Because it's a data structure, it can be built up in runtime
I can give you more exampes later :)
GraphQL is another fun domain, I may vibecode an example once I build out a few more parts of the mechanism (feel free to, too). But it would be generated at compile time in the current impl, I haven't designed for deferring to runtime.
I do not have expertise in language design, but Roc's type inference seems like a great feature to leverage for DSLs - you can have concise code with minimal boilerplate, yet expressive and strongly typed.
so in c# a great use case for this is something like a query builder
var post = dbContext.Posts; // this is IQueryble, so for example Filter takes the Expression<...> which is the "code as structs", auto converter
if(request.HideArchived){
posts = posts.Where( x => x.IsArchived); // this get accumulated
}
if(!request.TitleFilter.IsNullOrEmpty())
{
posts = posts.Where(x => x.Title.Contains(request.TitleFilter, Comparison.InvariantCultureCaseInsensitive);
}
return posts.Select(post => new { id = post.Id, title = post.Title, commnetCount = post.Comments.Count()});
this will produce a VERY good SQL, of course tailored to whatever database provider you are using, whether it's SQLite or Snowflake. The Contains will be translated to something like where title ilike '%{:title}%' (syntax from memory for postgres, so probably not that accurate), multiple Where will become an and. You get the idea :) these days you rarely have to fix that, unless you do something really weird.
There are some functions that don't do anyting in C#, but map to SQL, for example:
posts.Where( post => EF.Functions.Like(post.Name, "%smith"))
With Roc I imagine you can have this verified at compile time whether all the code you used in there actually can be mapped to SQL
For example:
bool IsRecentlyArchived(Post p) => p.IsArchived && p.ArchiveDate > DateTime.UtcNow.AddDays(-7);
posts.Where(IsRecentlyArchived); // this is ok, even the DateTime.UtcNow will be translated to a db instruction
posts.Where(p => p.Author == "someone" && IsRecentlyArchived(p)) // this fails on runtime
post.Where(p => p.Author == "someone").Where(IsRecentlyArchived) // this is fine
thing is, for GraphQL this can be really nicely generate in runtime and cover so many cases
also, since Roc already has a notion of code generation (like is_eq = _ will generate this), I think we should be able to write our own generators
making those generators just produce the AST in the same form as the reification would be super powerful
what concrete use cases (as in, not theoretical or hand-waved, but rather "here is a concrete code example of what it looks like") are there for this other than database queries specifically?
(and I'd count graphql as a database query)
if you do any UI work for example, you could build a dependency matrix for components at compile time (more or less)
I'd count that as a hand-wave :wink:
@Krzysztof Skowronek Could you write a pseudocode example of what leveraging the is_eq = _ syntax would be useful for, and what the generators would be? I'm not sure how that extends the concept of "Translator over the AST" of the current prototype
like what are other things people have done with this in C# for example, besides making database query DSLs or equivalent?
in dotnet we also use that for mocking, so you can say mock.Setup(x => x.SomeFunction(It.IsAny, It.IsBetween(10,100)).Returns(something)
you can then use the lambda as a key in a dictionary
there's already a separate post-0.1.0 plan for (the equivalent of) mocking - what about other production use cases?
For testing - you can make sure that some top-level pure expression results in a certain complexity - number of calls (structure, not runtime property), depth, etc.
also anything config related, like validation
again, I really am only interested in concrete examples here, not generalities :sweat_smile:
like to be clear, I think the cost of what you are describing is astronomically high
hmm, assertions maybe?
Assert.IsTrue(result.Something) can auto-print result.Something wasn't true
any usage of nameof(Something) can be replaced with just Something
to go from "Roc's type checker is the sole authority on what types mean, and you can trust that type inference works" and "a == b is just syntax sugar for a.is_eq(b)" to "userspace code determines what types get inferred, and they can make mistakes, so if you get a type mismatch error you now permanently have to wonder if it's because a userspace thing did something wrong" and "a == b might mean anything" is an extreme degradation in the quality of the language imo
so the bar for the benefit to outweigh that downside is correspondingly astronomically high
and "composable SQL queries look nicer" is not even remotely close to being in that tier, which is why I'm asking for other examples :sweat_smile:
well, if you want to have composable SQL queries (and in general I'd say you do), you end up with a query builder that then "renders" whatever it built into a string, having the database engine type as another parameter
the AST thing just makes the builder normal Roc code
other use cases are a bit more C# specific, like two way bindings for the ui, auto hooking up PropertyChanged events, automatic logging contexts and so on
right but that's what I mean
Krzysztof Skowronek said:
well, if you want to have composable SQL queries (and in general I'd say you do), you end up with a query builder that then "renders" whatever it built into a string, having the database engine type as another parameter
the AST thing just makes the builder normal Roc code
but that's what I mean baout "composable SQL queries look nicer"
@Karakatiza I took it from the webiste https://www.roc-lang.org/examples/EncodeDecode/README
there are other examples where you write something = _ and that gets generated
obviously they are implementable using normal method calls; no language features are needed to implement a composable SQL builder :smile:
the language feature would, at best, make it look nicer
I know I sound like a fanboy here :D
but LINQ (the iterator operations) just works tm on databases in C#, and it's amazing, and that's thanks to this language feature
so the same code goes with an array and with a database, I know it doesn't sound like a lot :D
but it is
I hear what you're saying, I just don't think the upside justifies the downside (and I don't think it's remotely close) if all we are talking about is database query DSLs :sweat_smile:
hence my wondering what else there is
well, the sky is the limit, in c# it ended up being used mostly for "around the database" world, but since every single patform comes with SQLite built in, maybe it's a good argument
Roc's types are common, and pure code is, well, pure - which makes pure Roc expressions truly universal - so Roc syntax can be used in a huge range of domains
to be honest, most of these things were recently replaced by custom source generators either way
these are all things people use them for in C# in practice? :thinking:
what are custom source generators?
// System.Text.Json: serializer metadata generated at compile time
[JsonSerializable(typeof(Order))]
partial class AppJsonContext : JsonSerializerContext { }
so before, you would use runtime reflection to just say give me all public properties of Order, this is an Expression of how to serialize it, compile it at runtime
now with the push for AOT (Ahead of time compilation, basicaly you want to generate a self contained binary Go/Roc style), runtime reflection doesn't work that well, because unused types are trimmed out
so the above generate at build time a file AppJsonConte.cs.g (that you can actually open and see the c# code if you want) that has whatever the reflection would produce
Well, all these have implementation in at least C# or Julia. There are Haskell/Python -> FPGA translators
so you are running reflection code at build time and generate a string that contains your C#
these are incremental, and pretty much instant in output
and you can write one that produces python if you want
or let's say OpenAPI descriptor into a file, and then have another source generator that gives you an API client based on OpenAPI spec
@Krzysztof Skowronek well if I understand you correctly that's how current prototype works - it compiles AOT
Karakatiza said:
Well, all these have implementation in at least C# or Julia. There are Haskell/Python -> FPGA translators
are people using these in practice though? Like I've certainly heard of tons of LINQ use in production, but I've never heard of people writing real FPGAs in Haskell.
yeah, it's all over the place
// loggin without boxing, a very c# specific case
public static partial class Log
{
[LoggerMessage(Level = LogLevel.Information, Message = "Order {OrderId} shipped in {Days} days")]
public static partial void OrderShipped(ILogger logger, int orderId, int days);
}
// UI specific thing, wrapping properties and methods with event-full things with a single attribute
public partial class OrderViewModel : ObservableObject
{
[ObservableProperty] private string _customerName; // generates CustomerName property + change notification
[RelayCommand] private async Task SaveAsync() { ... } // generates SaveCommand
}
// object mapping
[Mapper]
public partial class OrderMapper
{
public partial OrderDto ToDto(Order order);
}
there are DI (dependency injection) frameworks that have compile tie checks whether they can construct every type requested - that's all source generated
the mapping for example ties nicely to the AST expressions, you can have a source generated mapping from your db object into a DTO, and that will get translated into a SELECT in the db, just like that
no I mean using the bullet list of use cases form above, e.g. "Spreadsheet formulas (Excel, Google Sheets)" "Gradients for ML training, made from a loss function (like JAX and Julia's Zygote)" etc.
(the partial on the class means there can be another file with more members implemented, on the method it means it is implemented in another file, and the generators provide that implementation)
oh, I missed that message completely :D
Claude says Haskell Clash (for FPGAs) is used by company QBayLogic in prod and Google research in bittide/bittide-hardware project
I am certain this says more about the popularity of Haskell rather than about ergonomics of the feature
The SQL DSLs are obvious examples, and CUDA stuff is used actively in Julia
This feature in Roc potentially has better UX than C# and Julia counterparts. We need more voices on this, but I'm pretty sure this is much more than a party trick for Roc
3 of current 4 latest #ideas threads are in some form about this: reflection, generating code based on the types and database access
Karakatiza said:
- GPU kernels (CUDA and stuff)
- Shaders (GLSL)
- Verilog, FPGA
these three use cases (and similar) feel to me like they are very different from the SQL query string builder examples.
I'm most familiar with (although still very inexperienced with) the GLSL example, but what I think they all have in common is that I don't think you'd want an AST as the input; I think in all three cases you'd want something along the lines of a different backend than machine code or wasm - like you take Roc code as an input, you let the usual parsing/type-checking/etc. process run, and then instead of generating machine code you generate GLSL or FPGA or CUDA or whatever
so the pitch for that would be something like "what if you could write your shaders in Roc, and share code with your Roc application, and then the compiler could spit out [working, but not likely to be nice to read] shaders, possibly alongside your compiled Roc application machine code binary"
or maybe even generate Roc code on the client, execute on the server, validate it's only pure functions
but then again, it's also the case that GLSL is kinda like C but without heap allocations, and certainly Roc code can do lots of heap allocations...but maybe that's fine if the host just does them as stack allocations, given that the entrypoints would be set up to not accept heap allocations anyway? I'm not totally sure on that one, but it sounds like it could potentially fit shaders' constraints.
I think there could still be potential problems actually...like if you had an intermediate function that returned something on the heap, unless we did some sort of Goose-like optimization for internal functions :sweat_smile:
this is interesting to explore but it definitely feels like a minefield of nonobvious and serious downsides
#ideas > Type reflection — Zig `@typeInfo()` equivalent @ 💬
I am reading through the other thread, and this came up
with the expression AST you could just "make up" a function that is |some_struct| some_struct.some_property
I'd say just because we can't come up wiht something huge right now doesn't mean that this whole thing (reflection + convert code to AST (so like anti-reflection) + source generation) is useless
the partial keyword was kind of useless in C# for a decade, and then suddenly boom, source generators are built around it
feel to me like they are very different from the SQL query string builder
That's the point - reification is a general and universal mechanism
I think in all three cases you'd want something along the lines of a different backend than machine code or wasm - like you take Roc code as an input, you let the usual parsing/type-checking/etc. process run, and then instead of generating machine code you generate GLSL or FPGA or CUDA or whatever
That's what reification is - in the current prototype the Roc compiler does the parsing and type inference, the reifier produces and normalizes the AST, then the user's translator module walks the AST and generates the target program.
so the pitch for that would be something like
So that's exactly what the current design offers
It's just the generation (translator) is written in Roc, instead of e.g. Zig. And why not if it's more ergonomic?
again, I can't stress this enough: the downside here is huge
the bar for it to be a good idea to add to the language is way higher than "not useless" :smile:
I think the database query part is completely irrelevant here. If we want the most powerful and easily readable queries in Roc, a more powerful string template syntax could be added to the language, which could be used to write the queries in the query language of the database. But it's already achievable to some extent, see roc-pg. If #ideas > Type reflection — Zig `@typeInfo()` equivalent is added, it would bring even more flexibility.
And it would retain the same consistent behavior of the type system and control flow.
which part has the biggest downside?
source generation (so macros in another sauce), comp time reflection or comp time "make ast from this code"?
or the thole "generate AST at runtime and run it as an actual roc code"?
becuase each of them separately are bad, but combined become quite awesome
one more cool usage of reflection is something in dotnet we call convention test
you can have a unit test that does "find all rabbitmq message handlers and check if they return Result<_, MessageQueueError>"
or check if every module/ use case/ sub-system has the same shape (as in has a class that is SomethingHandler, SomethingValidator and so on), since many things are auto-registered it is quite usefull
Reification offers leveraging Roc syntax and semantics in other execution contexts. A "more powerful string template syntax" feature sounds like language-in-language - which possibly is a good feature, but I feel like this is a bigger stretch than reification UX
Karakatiza said:
I feel like this is a bigger stretch than reification UX
No. Because it's a string. It's obvious that the Roc syntax constructs are not applied inside strings, because strings are opaque to Roc type system and control flow.
comp time reading AST nodes in userspace is a huge downside. It means you can no longer look at a snippet of Roc code and even have confidence that you understand basic properties like what the control flow is anymore. Anything could mean anything.
macros have the same downside, and I've had a longstanding goal of never putting macros in the language - this is one reason why.
writing AST nodes is less of a downside but I'm not really sure what the benefit of that would be compared to normal function composition. It sounds like a solution in search of a problem but maybe I'm missing something.
I see that, but in principle that's already broken - the decoders/encoders are macros that use reflection
just not available to user space
It means you can no longer look at a snippet of Roc code and even have confidence that you understand basic properties
Noone should be using a package that doesn't respect basic semantics of Roc expressions :smile:
A "more powerful string template syntax" feature sounds like language-in-language
It's probably not even necessary. If #ideas > Type reflection — Zig `@typeInfo()` equivalent (which has a bit more support from Richard) is implemented, no new string syntax is required, as the current string syntaxes could be used with comptime and type reflection, providing full comptime type checking.
Basically, the language-in-language could be implemented entirely in userspace, without changing how the Roc type system or control flow works.
having functions that behave differently based on their types is something any language with parametric polymorphism (aka "generics") already has, so then the question becomes what the APIs are for determining the behavior. We're already accepting reflection in some form for encoding and decoding (and similar), so the question there is mostly what the API looks like.
Reification doesn't introduce new syntax, change type system or control flow, too
I don't know what that means, can you rephrase?
I think a good way to put it is reification is not about functions behaving differently, it's about functions having the same behavior in a different execution context
In a certain way it is a complement to platforms
I still don't understand the point, sorry.
I think his point is that with reification you don't introduce new syntax
posts.filter(|x| x.title.contains("something") is still normal Roc code, even if it will get translated into SQL (or shader and so on)
basically, there is a big difference between the category of "implementation is determined in part based on type information" (which we already have) and "implementation is determined in part based on AST" (which we have never had, on purpose, because the downside is huge and the benefits have never seemed close to outweighing the downsides)
Krzysztof Skowronek said:
I think his point is that with reification you don't introduce new syntax
posts.filter(|x| x.title.contains("something")is still normal Roc code, even if it will get translated into SQL (or shader and so on)
I think you're presenting that as if it's an upside, but I definitely view it as a downside :sweat_smile:
Again, what is the downside? The code being in user space has no bearing whether it's good or bad
another way of saying it is "in this world, there's no longer an such thing as 'normal Roc code' because it could turn out to be interpreted in so many different ways - it could even be SQL!"
The lambda is expressed in Roc and has Roc semantics which is extremely convenient. And you can use it both in Roc code and in a SQL query, with the same meaning. That's the power
I see, that other wording sounds closer to a downside, but having worked with C# for 10 years, it really isn't "as bad" - but I guess I will stop trying to convince you
like posts.filter("x") today is a function call. I know how those work. I know that if I rename posts to blah nothing will break unless that introduces shadowing, in which case I'll get a warning immediately. If I want to know what filter does, the only missing information is what the type of posts is, at which point I can go look up the docs for that function. And then the body of that function will follow all the same rules.
I think, this conversation leads to nowhere.
If a prototype is available, let's start building libraries/programs with it, and if it turns out to be useful in practice, then restart the discussion about merging it into the language.
Because I think this would be a disaster, not only for the reasons Richard listed, but for a bunch of additional reasons too.
yeah, I get it - explicit interpreters are the way to go in roc
I'd still say, being able to generate what they interpret is EXTREMELY valuable
well that's why I'm asking for specific examples haha
I'm looking to understand what value could justify the drawbacks - which I see as severe :smile:
Let's say the example is posts.filter(|a| a.includes("x")). a is array data structure, .filter() filters it, "x" is a substring. You express and care about what all these mean, not in what order they introduce heap allocations - this is Roc and not Zig. Reification allows translating this semantics into any compatible target. It wouldn't compile/translate for GLSL because that one doesn't deal with strings
A simple .filter() is a very weak argument to demonstrate the power and flexibility of a query builder. It's trivial in basically every ORM system in every language.
The prototype is out there, with improvements in short order, with enough docs to read or feed to LLM for experiments
I am providing examples to extrapolate from =)
But the extrapolation is not that easy, because if you write nontrivial queries, you'll very soon hit limitations.
Do you have any libraries/programs that already utilize the features in the prototype?
I have a SQL ORM example and GLSL one that feeds tha shader into roc-ray. Later this week I'll vibecode a GraphQL generator
I would like to have an ergonomic query builder in Roc. Currently the reason I'm not using it for web backends is that SQL is a horrible language, and there's no other database available.
I haven't built them out into "complete" packages, but it will be easy with Claude. For the next day or two I will spend my vibe-coding bandwidth to close a few gaps in the implementation
I would gladly try an ORM based on this feature, to provide real-world feedback.
so I looked for this, because I remebered that it existed: https://www.nuget.org/packages/AdminForge
their example for defining a generated data table:
panel.Table<Todo>(t => t
.Columns(x => x.Title, x => x.Status, x => x.AssigneeId, x => x.DueAt) // the list page, in this order
.Column(x => x.Title, c => c.Validate(s => s.Length >= 3, "At least 3 characters."))
.Column(x => x.Status, c => c.Status(s => s.Tone(TodoStatus.Done, Tone.Success)))
.Column(x => x.AssigneeId, c => c.Title("Assignee").LinksTo<User>())
.Column(x => x.Description, c => c.Content(Content.Markdown))
.Column(x => x.CreatedAt, c => c.ReadOnly().Format("yyyy-MM-dd"))
.Hide(x => x.UpdatedAt)
.Computed("due", x => DuePhrase(x.DueAt))
.Computed<AuditLogStore, int>("edits", (log, x, ct) => log.CountAsync(x.Id, ct), c => c.InList())
.DisplayMember(x => x.Title));
it's a bit convoluted, but it combines the expression trees (so the AST from code) with reflection to provide proper types of inputs when you edit a row
I'll try to complete a more or less complete SQL ORM this week, then. For now, I can only offer to look around https://github.com/Karakatiza666/roc/tree/reify/examples/reifying/sql
Although I think the best solution is compile time strings and the original query language of the database. But I'm curious about other approaches.
Could you possibly write a SurrealDB query builder instead of an SQL builder? It has a much better data model than Postgres or MySQL. I think that the features of SurrealDB queries would really stress-test this Roc feature.
and it's all type safe, so if you rename something, it's all still up to date, the title says the property name based on x => x.Status, and you probably also say t.AllPropertiesAsColumns().Hide(x => x.UpdatedAt)
Could you possibly do a SurrealDB builder
I may yolo and try to vibecode a Kysely-like thing which differentiates dialects, just PostgreSQL and SurrealDB for now. It will just be a day or two more worth of tokens, and I have plenty
it's a bit convoluted
With reification in Roc it would be much cleaner hehe
My two cents is that the thing that would drive this conversation forward is working code without the ORM. Show the pain. And you need a concrete case (to repeat what Richard Feldman said).
working code without the ORM
As in, a different example, or just the core feature itself?
Whatever problem you're trying to solve, show the current Roc. If you're just theorizing about problems to solve, then it's just a waste of time to speculate.
https://github.com/Karakatiza666/roc/tree/reify/examples/reifying
I may yolo and try to vibecode a Kysely-like thing which differentiates dialects, just PostgreSQL and SurrealDB for now. It will just be a day or two more worth of tokens, and I have plenty
I don't think Postgres and SurrealDB should be shoehorned into one system. SurrealDB is not an SQL dialect. It has a completely different data model, and if the query builder is limited to what both systems support, it would not be possible to demonstrate the advanced features of SurrealDB.
I haven't looked into it yet. If it's very different then yes, it makes sense to do a dedicated translator. I am interested in making an example for you =)
The problem with ORMs is that you sacrifice the expressiveness of the database's native language, even for a relatively primitive and ancient language like SQL. I don't know SurrealDB, but I agree with the gist of what notramo says.
not really - I keep saying EF Core is the best and I mean it :D
public record MonthlySales(int Year, int Month, decimal Total);
var report = await db.Database
.SqlQuery<MonthlySales>($"""
SELECT YEAR(OrderDate) AS Year, MONTH(OrderDate) AS Month, SUM(Total) AS Total
FROM Orders
GROUP BY YEAR(OrderDate), MONTH(OrderDate)
""")
.Where(r => r.Year == 2026)
.OrderBy(r => r.Month)
.ToListAsync();
you can just pluck raw SQL wherever you want, just type the output and you can keep chaining that with your normal LINQ
SurrealDB supports type unions in the schema, which would be excellent to use as Roc tagged unions. It's a must-have feature for me.
in roc the Monthlysales record could be anonymous and inferred
notramo said:
SurrealDB supports type unions in the schema, which would be excellent to use as Roc tagged unions. It's a must-have feature for me.
that's also covered by inheritance hierahies in EF Core - you can have a table for Shape, and put Rectangles and Triangles in there, and choose whether you want table per hierarchy, or table per type
I want them embedded inside the record.
No joins required.
Not as record types, but field types, inline, without having to link to a different table for every separate type.
you can do that also :D
if you have
class Door {
[Owned] // this is the magic
public Shape HoleShape {get;set;}
}
it will add all the columns needed for the Shape to the Doors table and add a Discriminator. You can also slap something like [UseJson] and put the whole thing in a json column
Does it support constraints at the database level?
by whole thing, I mean that HoleShape property will be stored in a json column with a discriminator
How would it look in Roc?
yes, of course https://learn.microsoft.com/en-us/ef/core/modeling/keys?tabs=data-annotations
if you don't like attributes, you can also define it as code: https://learn.microsoft.com/en-us/ef/core/modeling/#use-fluent-api-to-configure-a-model
in roc I imagine it would be wrapping in tags like { id: Id(i64), hole_shape: Owned(Shape)}
SQL does not support putting a foreign key inside a field of a JSON object. SurrealDB does support links inside any arbitrarily nested field, with the same constraints as a top level field.
then you keep it as a column and define a constraint
Keep a separate column for every type of a nested type union?
Then how to ensure that only certain combinations of these columns are set at once?
And all of the columns have to be nullable at the database level, losing database-level constraints.
I am pretty sure you can do that with custom converters: https://learn.microsoft.com/en-us/ef/core/modeling/value-conversions?tabs=data-annotations#configuring-a-value-converter
you can basically tell it how to convert the runtime object into something that will actually be saved in the db, and you can do really whatever you want with that
But this does not offer database-level constraints.
I would like to have an ergonomic query builder in Roc. Currently the reason I'm not using it for web backends is that SQL is a horrible language, and there's no other database available.
Although I think the best solution is compile time strings and the original query language of the database. But I'm curious about other approaches.
@notramo Am I understanding you correctly that you'd like something like roc-pg but for SurrealDB then? You agree with the approach of roc-pg but not with SQL as a language, essentially?
Yes, absolutely.
I've also been curious about SurrealDB so that may be something I look into later :)
if a database supports the thing, you can create it with EF Core
modelBuilder.Entity<Order>(e =>
{
e.Property(o => o.CustomerId)
.HasComputedColumnSql("((data->>'customerId')::int)", stored: true);
e.HasOne(o => o.Customer)
.WithMany()
.HasForeignKey(o => o.CustomerId);
});
this is a database enforced foreign key in JSON column in postgres
and this config is also a nice example of the AST from code in action
I looked it up, and Postgres does not fully support these.
I mean, it's a work around, irrelevant to the ORM though
Anyways, I did not mean it as a debate about which database system is better. The reason I brought it up is that I think, a Roc reification-based SurrealDB query builder should definitely expose something like this.
If all SurrealDB features are exposed in the builder, it would really stress-test the usability of this Roc feature.
yeap, for sure. with EF Core you also get engine specific functions and configs, so in Roc it would be the same
Some of these can be solved with Postgres too, but it's not trivial, and just makes the schema unreadable, complex and buggy.
I haven't kept up with this thread. A use case I have, that I'm using string templates for, is generating a type. Like, I have a (non-Roc) file, and I want to run some code that reads that file, does something with it, and creates a Roc module. This matches a couple different game dev asset processing things.
It's fine if that kind of asset processing keeps needing string templates with some external CLI. I'd be excited about a way to do it all in Roc, where possible. Say, read an imported JSON config file and generate a Roc record with matching keys or something (where that json is probably itself a preprocessing artifact of one kind or another). But, neither struct reflection nor function body ASTs address this. So if I'm wishing for codegen features, I'd want to be able to create Roc types, rather than the function body thing.
Dan G Knutson said:
read an imported JSON config file and generate a Roc record with matching keys or something
You can do that today using basic-cli no?
Yeah, if I'm using string templates to make a Roc file, then they could come from wherever. In practice, I'd be writing the Rust that writes the JSON, so I can generate a Roc file there instead.
What I mean is, if the language adopted a feature more like rust's proc macros, where a parsed function body becomes something else, then I'd rather get the high-powered version of that, that can create structs/records and submodules without going through the filesystem. Like, why pay the full cost of a macro system without getting the full benefit.
Dan G Knutson said:
So if I'm wishing for codegen features, I'd want to be able to create Roc types, rather than the function body thing.
I'd suggest starting a separate #ideas thread for that!
It sounds like you just want a Roc CLI app that parses an assets file at runtime and outputs a Roc module with type declarations, I don't see any form of reflection needed for this
I'm personally very against out of band apps that generate roc code with a string template
The workflow should be 'git clone & roc build ./main.roc' and that's it. With dev giving you hot reload to all generators.
And the "source" for the generator should be both roc code (as in types) and other files (openapi schema), and the target should also be roc code and other files (again, for example openapi schema)
Then you can generate types from SQL schema, and for example generate an admin panel.
Any UI grid control now gets the automatic column mode
Hell, just with that you can make something like DBT in roc
I think this would need to be designed very carefully. It could be a compiler feature, or could be a build toolchain feature - so the various tools you may need are orthogonal to the compiler. Like webpack/vite plugin ecosystem in JS
Dan G Knutson said:
So if I'm wishing for codegen features, I'd want to be able to create Roc types, rather than the function body thing.
We discussed this in the type reflection thread, and the solution was that it's done in Roc in a reversed way: the type is inferred from usage and then validate that type against the original string (which is possible with type reflection).
You have to use the resulting struct and fields later (otherwise there's no point in using a struct), so it can be inferred.
As long as you can clone a repo and type in 'roc dev' and it works, the mechanism not that important.
Like with python you have to first set up a venv, run uv sync (or one of the dosen alternatives), activate it, and then maybe it works of your base python version was correct
I'm node you usually have to run the install first, then regenerate all generated code (unless it's checked in, which I don't like), only then you can run things
notramo said:
Dan G Knutson said:
So if I'm wishing for codegen features, I'd want to be able to create Roc types, rather than the function body thing.
We discussed this in the type reflection thread, and the solution was that it's done in Roc in a reversed way: the type is inferred from usage and then validate that type against the original string (which is possible with type reflection).
That doesn't sound like a great experience for generating API clients for example, or generating UI for a given model
So in C# there is for example a library called refit: https://www.reactiveui.net/documentation/refit/
In short, you define an interface with attributes for urls, query params, body type, response type, and it generates code around HttpClient (which it can create on its own, or you can supply with some custom settings)
There is another library that gives you a source generator that "generates code from a string template" to convert an openapi schema into this interface, live. You paste new schema, and your editor lights up in red if it changed things you are using.
There are others that generate everything right away, refit is the standard, especially if you want AOT compilation
notramo said:
Dan G Knutson said:
So if I'm wishing for codegen features, I'd want to be able to create Roc types, rather than the function body thing.
We discussed this in the type reflection thread, and the solution was that it's done in Roc in a reversed way: the type is inferred from usage and then validate that type against the original string (which is possible with type reflection).
You have to use the resulting struct and fields later (otherwise there's no point in using a struct), so it can be inferred.
Yeah, I did read that. The roc-pg method requires that your preprocessing is pure, so you can't (for example), run shader code through a cpp library to get the types out without going through the file system. I might be able to do the roc-pg "checked at usage" way by writing the type & serialization metadata to JSON.
Krzysztof Skowronek said:
I'm personally very against out of band apps that generate roc code with a string template
The workflow should be 'git clone & roc build ./main.roc' and that's it. With dev giving you hot reload to all generators.
I'm talking about the feature because I would also prefer to avoid an out-of-band CLI where possible. But some kind of preprocessing is necessary here, and that kind of workflow is fairly common in game dev. You have some other program that produces a binary (3D model, sprite sheet, shader bytecode) and some metadata, and you want to be able to check/gather the metadata and hot-reload the binary. So there's going to be an out-of-band step, because there's at least one other program you depend on, and maybe a chain of them.
Generating source code based on the metadata is a bit weirder, but people do it in cases where they want the type checking. The cases I've heard of are shared headers for shaders and generating APIs for sprite animations.
@notramo
It's obvious that the Roc syntax constructs are not applied inside strings, because strings are opaque to Roc type system and control flow.
If that's how you see the extended string templates + "zig-like" type reflection - then reification is a superset of that, and it already allows this in the form sql(| | "SELECT something FROM x WHERE a == ${value.fieldA}") - If I understood the kind of UX you are describing. Don't know why I didn't catch that earlier. But I find using the target language (e.g. SQL) verbatim as an opaque string, with no type checking, just to be able to reference Roc values inside an underwhelming UX.
Importing and compiling text files of a target language then becomes thin syntax sugar on top of the reification mechanism
Reification would be an overkill just for that, of course, but it is a general mechanism
SurrealDB example is in progress, it would be done in ~3 hours if not for a bug in the reifier
Hell, just with that you can make something like DBT in roc
Haha
I heard about DBT earlier and liked it; looked into it now and this could in principle replace Jinja and its template syntax for SQL- for a better UX: things like compile-time errors for SQL, better dependency graph. The only cost to eat would be having to write SQL-like DSL instead of plain SQL.
@notramo let's see if https://github.com/Karakatiza666/roc/blob/reify/examples/reifying/surreal/main.roc converts you. It has the features you were interested in:
Instructions. This example is more or less feature-complete, so you can paste the package into a basic-webserver app and try doing something with it (building the compiler fork, of course).
@Krzysztof Skowronek this is the most involved example so far; since you have a lot of experience with Entity Framework would be great to learn your opinion after glancing over the syntax of main.roc
Dan G Knutson said:
I'm talking about the feature because I would also prefer to avoid an out-of-band CLI where possible. But some kind of preprocessing is necessary here
I think a first class transformation hook in build step would be nice, so we don't need to hack around and race, especially for hot reload scenarios. In roux we handshake with host for sqlite checking, so not pure. I think possibley my template-> bytecode compilation could be pure.
@Karakatiza, thanks, I'll check it out tomorrow.
@Karakatiza this looks pretty good!
so a quick comparison with EF Core:
The function names are the same in EF, so you have select for Surreal, and map for normal Roc collections
Cool trick in EF is that you can easily convert the IEnumerable (that would be Iter in Roc) to IQueryable (which uses Expression<Func<_>>, so reification, while IEnumerable takes normal Func<_>), so can have have a single function that operates on both.
your blocks could have looked something like:
# I am not sure how to tacle this type, but it would probably "just work"
get_follow_ups = |query| query.keep_if(needs_follow_up).map(outcome_line))
block(
"-- query: follow_ups",
[],
Surreal.from(visits) |> get_follow_ups().to_list(),
Surreal.literal(seed_visits |> Surreal.as_query() |> get_follow_ups().to_list()),
),
another thing is that like in your case, the queries are lazy, so the database trip happens exacty where you think it is. In your code it's bind(), in EF it's .ToListAsync() or for foreach(var pattient in dbContext.Patients.Where(x => x.Name.Contains("Steve"))
So, iterating the query triggers it. You can even make it query.ToAsyncEnumerable() which would give you something like Roc Stream, and a buffered read, so you can foreach over millions of rows as long as 2 or 3 fit in memory at once.
You can also do dbContext.Patients.Where(x => x.Name.Contains("Steve").ToAsyncEnumerable().Where(x => x.Id %2) - the first Where will happen in the database, the second at runtime in C#.
Also, defining the tables is a bit funny in your example, they seem to float around as global objects (you picked quite a domain to show this off), in EF you define a DbContext that has properties of type DbSet<SomeClass> and every type that is reachable from there will become a database table (unless configured as Owned type or JSON column)
EF also does change tracking, but since we are working with immutable structures, that should not be a thing - and that's great actually
I am also not sure how you designed your links, but in EF if there is a link to another table, by default it's not loaded unless use it in your query
so
dbContext.Posts
.Where(p => p.Likes.Count() > 15) // this is not including the Likes, they are just in the WHERE
.Include(x => x.Comments.Where(c => c.Likes.Count() > 10 && c.Dislikes.Count() < 5)
.Select(p => new { Id = p.Id, TopComments = p.Comments.ToArray()})
.ToListAsync() ; // this triggers the db trip
@Karakatiza, a few quick questions:
Is it building a huge query and runs everything at once, or lazily loads something, so multiple database queries happen if I want to access some data structures?
Is time series and geo supported?
Are functions supported? These are very useful to do the computation inside the query, to avoid extra roundtrips.
@notramo I guess the bind call pulls records from the db for that specific query
I haven't focused on the runtime properties yet, only on syntax; the execution is eager at bind site - no lazy queries on access
Just dump all your feature requests in one message so I make sure they're in
Krzysztof Skowronek said:
I am also not sure how you designed your links, but in EF if there is a link to another table, by default it's not loaded unless use it in your query
Well links are structural, so in the demo they are a part of the table definition, so when it comes to queries links are already available, strongly typed
No, that's not what I meant.
If you do 'dbContext.Posts.ToListAsync()', then 'posts[0].Comments == null'
If you do 'dbContext.Posts.Include(x => x.Comments).ToListAsync()' then posts will have loaded comments. Basically the include causes a join
I don't know surrealdb enough (or at all :D), but I imagine you can get a pet without pulling the owner via the link, correct?
This needs to be possible to express in the query
Just dump all your feature requests in one message so I make sure they're in
Is it possible to enable issues on the repo? I would open tickets for these.
These are just examples, I will but you should only open issues regarding reification itself :smile:
I will release the SurrealDB example as a package if you are going to be using it, and you'd create isses there
Yes, that's even better.
I'll enable issues now, for reification itself
@Krzysztof Skowronek ah, I see. In terms of this links works the same way as you describe for EF - link can be used in a join without pulling in that table. EF's .Include(x => x.Comments) here is .fetch(|p| p.owner). Currently loaded/unloaded tables share the same type so it's a runtime error, I can possibly can protect against accessing rows not actually loaded at the type level.
I prepared what I find to be an interesting prior-art comparison.
https://github.com/Karakatiza666/roc/blob/reify/roc-reify-prior-art-comparison.md
The headline claim: on most design properties Roc matches or improves on the strongest peer, and no known system offers them all together.
Last updated: Nov 04 2026 at 18:54 UTC