Apache Kafka

Apache Kafka is a distributed event streaming platform. Producers append records to topics, which are split into partitions and replicated across brokers. Consumers read from those logs at their own pace, tracking their position with an offset.

Why it matters

Unlike a traditional queue that deletes a message once consumed, Kafka retains records for a configured time or size. Several independent consumer groups can read the same data, and a new service can replay history from the start. This makes it popular for activity tracking, log and metrics pipelines, stream processing and connecting systems through change events. Ordering is guaranteed only within a single partition, so the partition key matters.

Operating a cluster takes real expertise, and many teams use managed offerings.

Example

kafka-topics.sh --create --topic orders --partitions 6 \
  --replication-factor 3 --bootstrap-server broker1:9092

Records with the same key, such as a customer id, go to the same partition and are read in order.

Common mistakes

• Choosing too few partitions, which caps consumer parallelism and is awkward to change later.
• Using Kafka for a simple job queue where RabbitMQ or SQS would be easier.
• Not monitoring consumer lag, the gap between latest and processed offsets.

Related

• Publish/subscribe
• Message queue
• Data pipeline
• RabbitMQ

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.