Queues, Ordering, and Long-Running Tasks in Kafka with Atleon

At Indeed, we help people get jobs.

Our team supports this mission by ingesting data about employers’ hiring needs, enabling them to manage this data, and transforming it into searchable job advertisements—commonly known as job posts. The many different asynchronous processes involved in solving these and related challenges can be complex, widely diverse, and in many cases, I/O dependent.

Consequently, our team implements a wide swath of asynchronous use cases on top of Kafka. This includes conventional idiomatic use cases, but just as important, use cases that are historically tricky on top of Kafka’s architecture. Such use cases involve “treating Kafka like a queue” and implementing long-running (i.e. minutes, hours) per-record consumption tasks.

This talk will dive into how we at Indeed tackle these challenges with Kafka and Atleon, an OSS project, as well as explore the following topics:

• The difficulties involved with “treating Kafka like a queue”
• How “share groups”, introduced by KIP-932, may (or may not) address needs for higher consumer concurrency
• What reliable queue-like consumption without loss of ordering looks like with vanilla Kafka consumer groups
• Facilitating long-running per-record consumption tasks
• Accomplishing per-key processing order and concurrency