Eric Rogstad (May 29 2024 at 21:32): Eric Rogstad (May 29 2024 at 21:32):Suppose I want to write a Roc app that handles Kafka messages rather than HTTP requests. Would the basic-webserver still be the best platform to use?
Brendan Hansknecht (May 29 2024 at 22:43):Gut feeling is that a custom platform that understands Kafka could be really powerful here, but if you fundamentally need a webserver and want to build something today, basic-webserver is really the only thing ready in Roc.
Brendan Hansknecht (May 29 2024 at 23:30):Oh, though with Kafka queues, it is more like a CLI that polls the kafka queue and the post results, right? So it isn't really a web server at all. So might fitter better to a a CLI repeatedly making http requests across multiple threads
Luke Boswell (May 29 2024 at 23:48):After skimming through this tutorial... https://developer.confluent.io/get-started/go/#create-project
I think it might be possible to make a Go platform to do this.
You could have the app provide the ConfigMap on startup for all the properties, and Topics to subscribe to etc.
The app could provide a handler and the platform exposes an API with effects or things to do on each message.
Luke Boswell (May 29 2024 at 23:50):@Oskar Hahn has made a lot of progress recently with Go platform development, and upgraded the template, so I think we have most of the parts needed to get something like this working.
Luke Boswell (May 29 2024 at 23:51):https://github.com/lukewilliamboswell/roc-platform-template-go
Luke Boswell (May 30 2024 at 00:01):I also want to mention that I have no experience with Kafka other than a quick google.
Eric Rogstad (May 30 2024 at 05:05):Thanks guys. That's helpful!
Let me come at this from another angle though — what if what I really want is to be able to handle a bunch of requests (or queue messages) in parallel, and take advantage of the server's many cores — is there a platform that will run Tasks on multiple threads by default? Or that makes it easy to do so?
Eric Rogstad (May 30 2024 at 05:08):The context is that I've got a Rust project that uses https://github.com/fede1024/rust-rdkafka for interfacing with Kafka, and uses tokio tasks and channels to split up the work of processing the messages across multiple cores.
I'm trying to think about how I'd organize an analogous project in Roc.
Oskar Hahn (May 30 2024 at 05:09):As far, as I know, currently, you can not run Tasks in parallel. This will probably change in the future.
What you can do today is writing your own platform. The host (the part of the platform, written in rust, go etc) can then call the exported roc function in parallel.
Eric Rogstad (May 30 2024 at 05:11):Correct me if I'm wrong, but is it fair to say that the Task pattern, with managed effects, should be pretty amenable to running in parallel? Especially since everything in Roc is immutable.
Oskar Hahn (May 30 2024 at 05:17):Here is the proposal to change, how Task works: https://docs.google.com/document/d/1-h9bNNCLuYV2wSvjQA58SsGHOJivH9NHGr4wU_VF5I0/edit
Here is the discussion, to implement it: https://roc.zulipchat.com/#narrow/stream/395097-compiler-development/topic/Task.20as.20builtin
With this change, Task can run in parallel.
Brendan Hansknecht (May 30 2024 at 05:31):To clarify, basic webserver runs multiple workers running tasks in parallel
Brendan Hansknecht (May 30 2024 at 05:31):What is missing is starting with a single threaded task and then spawning many things to run in parallel from that single threaded start
Brendan Hansknecht (May 30 2024 at 05:32):In this specific case, a bespoke platform for kafka that is built on top of https://github.com/fede1024/rust-rdkafka would make a lot of sense (but of course be more work).
Brendan Hansknecht (May 30 2024 at 05:33):Could also be built upon any similar tooling/libraries. You would define the worker functions in roc and something else would deal with mapping to/from kafka queues
Brendan Hansknecht (May 30 2024 at 05:34):Otherwise, with existing roc platforms, I'm not really sure something would work quite right. Basic cli is only single threaded. Basic webserver is not really made for queue processsing workers. It is made for url based web requests
Eric Rogstad (May 30 2024 at 05:37):Hmm, yeah I think what I'd be looking for is something like a more generic version of basic-webserver, where it runs multiple workers in parallel, but the workers aren't necessarily handling HTTP requests, and also they can pass messages to each other.
Brendan Hansknecht (May 30 2024 at 05:39):Yeah, and that doesn't have to be tied directly to kafka, but roc really lets you play with the idea of tying directly to something like kafka if that would make a better experience for some forms of apps
Last updated: Jul 05 2025 at 12:14 UTC