The New Wave of the Streaming Log Technologies
When Kafka compatibility is no longer a goal.
Apache Kafka was open-sourced in 2011, and it made a huge impact a few years after that. The Streaming log became a very popular abstraction.
Today, 15 years later, I believe we’re experiencing the third wave of streaming log technologies. The abstraction is still the same (and as useful as before), but many new implementations raise and try to answer really hard questions.
A Bit of History
Was Apache Kafka actually in the first wave? Probably.
Before Kafka, the idea of an Enterprise Message Bus was quite common. We had messaging tools (IBM MQ, RabbitMQ, ActiveMQ, ZeroMQ, TIBCO, Solace and many more) of all shapes and flavours. Messaging and streaming share many primitives and concepts. And I do believe we needed to explore what was possible with messaging before arriving at the streaming log idea.
But Kafka had a unique twist: the log is durable, the fanout reads are cheap (you can have hundreds or thousands of consumers), and write performance is critical. Making it open-source contributed to its success as well.
After a while, many companies started offering a packaged Kafka product. Sometimes with a lot of customizations. I’d still categorize these products (Confluent, Aiven, AWS MSK, etc.) as the first wave: in the end, they made Kafka operationally easy, but they haven’t drastically changed the technology itself.
The second wave was started by Redpanda. It showed that it’s possible to re-implement such a complex project from scratch and maintain compatibility. Later came WarpStream and Bufstream, building on top of the object storage abstraction. Apache Pulsar is worth mentioning as well - it wasn’t created with Apache Kafka support, but it was eventually added.
I previously wrote about Redpanda and WarpStream here and here.
The Third Wave
Apache Iggy
Apache Iggy is a high-performance streaming platform. It supports many primitives that can be found in Apache Kafka: topics, partitions, consumers, producers, etc. However, it doesn’t support the Kafka protocol, and it’s quite intentional. This fact alone made me very curious about the project.
Performance is the main goal of the project. For example, it uses Rust, thread-per-core architecture, io_uring, etc. This makes it somewhat similar to Redpanda (minus the Rust part).
It comes with a few distinct features:
It supports more than one protocol: not just TCP, but also QUIC, HTTP and WebSocket (!). Supporting the last two means that it’s (theoretically) possible to consume or produce data directly in your browser (without any intermediaries). But I haven’t seen a working demo of that yet.
A very easy way to build and load connectors: just compile a dynamic library and load it at runtime on the broker side. I used a similar approach for Streamling, and I’m convinced it’s one of the best ways to extend a streaming system1.
You can check more features here.
Btw, there is now a company behind this project: LaserData, which offers a managed version of Iggy.
S2
S2 is a low-level API for real-time streams. It tries to answer the question: can interacting with a streaming log be as simple as reading and writing data with S3? There are no consumers and producers. There are no partitions. Instead, you have a few simple primitives:
Streams (similar to topics).
Appending to a stream.
Reading a stream.
That’s it. Interacting with streams can be done over HTTP.
If you think that it’s not enough functionality, here’s an example of an end-to-end pipeline writing data from Postgres to ClickHouse with Streamling, leveraging S2 as the streaming log.
Durability is an important guarantee S2 offers. From one of the customers:
S2 solved our problem with reliably streaming long-running AI sessions. Before, connection drops could cause lost data and broken streams. S2's bottomless storage means our customers can stream for hours and network hiccups don’t matter.
This means that the simple low-level interface allows S2 to stream literally anything, not just conventional events/messages.
OpenData Log
OpenData is a family of open-source, object-native databases, and the Log is the latest addition.
The announcement blog post argues that the streaming logs are supposed to support two different types of workloads:
Funnelling: collecting data from high-cardinality sources and coalescing them onto a low-cardinality pipe.
Routing: delivering events from addressable sources to addressable destinations.
And Apache Kafka failed to properly support routing workloads. The post explains that using partitions for routing leads to large read amplification: e.g., you need to read the whole topic to find messages with certain keys.
OpenData Log also supports HTTP protocol with several endpoints for reading and writing records.
Honorable Mentions
I have to mention two more projects:
Apache Fluss, lakehouse-native streaming storage. I wrote about it here. Imagine a Kafka cluster, but every topic is a table with a defined schema. That’s Fluss (it offers much more than that, of course). Why mention it here? Arguably, it’s an example of a streaming log, but it’s specialized to serve data analytical use cases.
Apache Pulsar 5. The next major version of Apache Pulsar (announced here) will support Scalable Topics. A Scalable Topic is a partition-less topic that dynamically splits and merges its keyspace. Choosing the number of partitions is one of the main challenges in Kafka, and, as we saw above, keys can be a better abstraction for routing use cases. Why mention it here? 5.x is not GA yet 🙂.
Summary
The new wave of streaming log technologies doesn’t support the Kafka protocol. Instead, they try to approach the log design differently.
In retrospect, S2 and OpenData Log are very similar: simple, low-level API endpoints over HTTP, Object Storage as a primary backend. OpenData Log decided to bet on routing use cases and designed the API around record keys.
I believe simplicity is the main goal behind the projects. Offloading a lot of complexity to object storage makes sense; you can cheaply get high durability and scalability guarantees.
Iggy stays very close to the Kafka protocol, but it’s not constrained by it. Having performance as the number one goal can be attractive for many potential users. And rewriting all your consumers and producers could be an easier sell nowadays, thanks to LLMs.
I still wonder if this new wave of systems will complement existing Kafka deployments, or try to replace them. Kafka is no longer a single project - it’s a protocol, and protocols are tenacious.
Advanced Apache Flink
Advanced Apache Flink is an on-demand course focused on Flink internals, production deployment best practices, and advanced patterns.
Running connectors in the broker can be a bad idea if isolation is not properly implemented.
@yaroslav Tkachenko - Soon Apache Iggy is supporting Kafka Compatibility. It is in works and progress can be tracked at https://github.com/apache/iggy/issues/3560. Please stay tuned.