In our experiment around 9 months ago, for a topic with certain number of partitions , when we created a Sink connector using Benthos - the number of consumer threads spawned exceeded the number of partitions of the topic and operation wise it was sluggish and slow waiting for very very long for a very simple sink operation
Great article, thanks for sharing it! I think you'd find this discussion that I started in the Bento github repo to be interesting - it seems to me that Bento should integrate/absorb Conduit (which is essentially Kafka Connect written in Golang) into it. https://github.com/warpstreamlabs/bento/discussions/396
Though, the bento/warpstream/confluent team surprisingly hasn't really shown much interest. Consequently, I'm going to move to a Debezium -> NATS -> Bento/RP Connect -> wherever stack, which might be better for my use case anyway... Still, it seems to me to be an idea with enormous potential
In our experiment around 9 months ago, for a topic with certain number of partitions , when we created a Sink connector using Benthos - the number of consumer threads spawned exceeded the number of partitions of the topic and operation wise it was sluggish and slow waiting for very very long for a very simple sink operation
Very interesting! Would you mind sharing any specific details?
I love the benthos project, it's fast (thanks to Go) and well written. This drama though...
Great article, thanks for sharing it! I think you'd find this discussion that I started in the Bento github repo to be interesting - it seems to me that Bento should integrate/absorb Conduit (which is essentially Kafka Connect written in Golang) into it. https://github.com/warpstreamlabs/bento/discussions/396
Though, the bento/warpstream/confluent team surprisingly hasn't really shown much interest. Consequently, I'm going to move to a Debezium -> NATS -> Bento/RP Connect -> wherever stack, which might be better for my use case anyway... Still, it seems to me to be an idea with enormous potential