Kingfisher Webplatform · show and tell

I want to introduce the Kingfisher platform. It is a platform to build websites, that follows a different approach then the basic-webserver-platform.

There is no SQL-Database. Instead, you define your own Model. It can be any Roc type and is held in memory.

main : {
    decodeModel : [Init, Existing (List U8)] -> Model,
    encodeModel : Model -> List U8,

    handleReadRequest : Request, Model -> Response,
    handleWriteRequest : Request, Model -> (Response, Model),
}

handleReadRequest gets a representation of the request and the Model and can only return a Response. handleWriteRequest has the same arguments, but can also return a new Model. The distinction, if its a read or write request is done on the HTTP method. GET requests can not manipulate the Model and are guarantied to be idempotent.

The platform can handle many read requests at the same time. Write requests can not run in parallel. So there can be no conflicts between them.

The persistence of the data is similar how redis does it. In redis, all data is in memory, but there are two ways to persist it on disk. There are dumps (in redis called RDB) or redis can write every write-command to an append only file (in redis called AOF). There are different strategies you can do with this two methods.

The current implementation of kingfisher saves an encoded representation of the model on shutdown. It also writes all write requests to an append only file. If the platform crashes, the model can be recreated by rerunning all the write requests. Since there are no side effects, you always get the the Model, when you run the same write requests. In the future, there could be setting to define the persistence strategies.

This approach defensively has an upper limit on scalability. But I think, there are a lot of projects, where this approach is good enough. On the other hand, kingfisher gets an easy entrance to building a website. You don't need any knowledge about SQL or other databases. Just create your Model in Roc and manipulate it, as you would in any other Roc application.

The platform is called Kingfisherbecause a Kingfisher is a beautiful bird and bird names could be a good fit for Roc modules.

The platform is written in Go. It is in a very early state. Expect memory leaks and unexpected crashes. But I am interested to hear, what you think about this approach.

Richard Feldman (Apr 29 2024 at 10:06):

wow, really cool design - love to see a fresh take on how to make a webserver, and the "GETs can't change the Model" guarantee is awesome!

Norbert Hajagos (Apr 29 2024 at 10:28):

This is the power of the platform-application separation and a creative solution! Nice job :tada:

Isaac Van Doren (Apr 29 2024 at 12:22):

Johan Lövgren (Apr 29 2024 at 18:48):

I love the precedent of using bird names for packages. This bodes well for Raven DB:grinning_face_with_smiling_eyes:

Alf Richter (Apr 30 2024 at 08:28):

@Oskar Hahn , very cool! Just as a side note: the new open source project golem is using a similar approach, but it is based on WASM’s simple execution and memory model, every (!!) interaction with the outside (side effect) and each memory change is recorded. It works out of the box for most wasm programs:

Eli Dowling (May 02 2024 at 03:58):

@Oskar Hahn how would you imagine handling caching in this case? Say if your GET request requires you to fetch some data or do a database query or some such and you wish to cache it?

Oskar Hahn (May 02 2024 at 06:15):

The database is in memory. So there are no database queries. The platform does not support Task on purpose, so there can not be any data fetching or any kind of IO. The only thing a GET request can do is heating up the CPU and creating an HTTP-Response. The only reason for caching would be to save CPU and bandwidth. For most cases, the normal HTTP-Cache headers should be adequate. The platform could also read the HTTP-Cache-Headers and do some caching like any http proxy.

If this is not enough, it would be possible to add the caching feature, where the application can return the HTTP headers, it depended on (for example the url and Authorization). The platform could cache the response. If there is another request, where the specific headers are the same, the cached response could be returned by the platform without calling the application. This cache could be invalided on every write request.

Eli Dowling (May 02 2024 at 10:44):

ahh, sorry I missed that part, If you're not performing any IO anyway there isn't any need for caching. My bad, thanks for the clarification though :)

Oskar Hahn (May 27 2024 at 06:27):

I created a new release, that support the legacy and the surgical linker. The release contains files for mac, windows and linux.

Niclas Ahden (May 27 2024 at 06:40):

Luke Boswell (May 27 2024 at 07:02):

Oskar Hahn (Jan 06 2025 at 11:30):

The main purpose of the new release is to support effects. Of course, with the new purity inference.

Model :  ANYTHING
init_model: Model
update_model: Model, List (List U8) -> Result Model _
handle_request! : Http.Request, Model => Result Http.Response _

The host still makes a distinction between read requests (like GET) and write requests (like POST). There can either be multiple read requests in parallel or one write request. But the new API does not let the app observe this behavior.

To update the model, you have to pattern match on the request method. For example:

when request.method is
        Get ->
            ...

        Post save_event! ->
            ...

The method on write requests has a function save_event! as payload. You can use this function to let the host save an event. After handle_request! is done, the host calls update_model with all new events to update the model.

The reason for this new change is that the old version had to save every write request, even when the authentication failed or they were sent to an unknown URI. The reason was that the platform could not know if the Model changed. With the new API, the event handling has to be explicit by the application.

This new API is in flux. A friend of mine does not like that he cannot just save the new model, but has to create events. I don't like that the Event is a List U8. I would more like it if it could be a tagged union you can pattern match on. For example Event: [CreateUser {username: Str, ...}, DeleteUser {user_id: U64}]. But it is currently not possible for Roc effects to have a generic type that the application provides to the platform.

At the moment, there are only effects to write a line to stdout, get the current time and a very simple version of sending a GET request. But this new API has the potential to support many more effects. I will add them when needed.

The idea of the kingfisher app is to have a webserver platform that can hold state in memory without the need for writing SQL statements. Now this is possible, but also doing effects.

If you've tried kingfisher or are interested in using it, I'd love to hear your thoughts on the new API design.

Norbert Hajagos (Jan 07 2025 at 15:33):

I haven't tried it, but I can tell that this is one of the coolest use of Roc I have seen.
What's your thought on adding some kind of "versioned model & updaters" api? Would benefit greatly this project i think. Not talking about migrating the binary format of events, but rather the case of "I want a new version of the model (like swapping is_admin from aBool to user_level tag), so I will write the v2 of the update function and also a migrate_model_from_v1_to_v2 function that will transform the model to the new format so that the v2 update function will work on it". These migration functions would work as a bridge that run once per initialization.

Oskar Hahn (Jan 07 2025 at 16:00):

I don't think a migrate function is necessary. But if there is a usecase, I have no problem with adding it.

In the current version, the model is not saved at all. On startup, the platform calls init_model and afterwards update_model with all events. So all events get a playback on every start. I have two small go-projects that work an the same principle, where this playback is so fast, that it feels instant. If startup times will be a problem, I could implement some sort of snapshots.

Since all events run every time, you can rewrite the update_models function to interpret then differently. For example, if you have an update_user- Event with a {is_admin: Bool, ...} payload, you can rewrite the update_models function to set user.user_level to Admin. For future updates, you can rename new events to update_user_v2 : {user_level: [...]}. So you can make an distinction between the update-events before your change and the new update-events. In my private go-eventstore-projects, I had a very good experience with migrations in that way.

I am not an expert on event stores. But if I understand the idea, then you don't save data, but the information, what happed. The information user X has clicked on "make user Y an admin" can never change and will never need an migration. Only the way, how you interpret the event can change.

Norbert Hajagos (Jan 07 2025 at 18:49):

Well yes, I didn't think about that solution as I've never used such an architecture. Problem solved then, great work and thanks for the explanation!

Stream: show and tell

Topic: Kingfisher Webplatform

Oskar Hahn (Apr 29 2024 at 06:40):