Kafka Share Groups and Parallelizing Consumption — Part 1: Tuning max.poll.records (opens in new tab)

All tests were executed against Kafka 4.2.0 using Dimster. In the last post we measured the overhead that the mechanics of share groups adds, and saw that it is pretty small. Likewise we saw that raw throughput was also comparable to consumer groups and even saw it exceed consumer group throughput on one test. In this post we’re going to simulate processing time in the consumers to make these benchmarks more realistic and show the utility of share groups (namely the ability to parallelize pro...