Apache Kafka Explained

What is Kafka?

Kafka is a distributed event streaming platform. Think of it as a massive, persistent log where services publish events and other services consume them. Unlike a traditional message queue where messages are deleted after consumption, Kafka retains events for a configurable period (days, weeks, or forever).

When a user makes a payment, the payment service publishes an event to Kafka. The notification service picks it up and sends a receipt. The analytics service picks it up and updates dashboards. The fraud detection service picks it up and checks for suspicious activity. Each consumer works independently.

Core Concepts

• Topics - Named channels for events. "payments", "user-signups", "order-updates"
• Producers - Services that publish events to topics
• Consumers - Services that read events from topics
• Consumer Groups - Multiple instances of a consumer sharing the work
• Partitions - Topics are split into partitions for parallelism
• Brokers - The Kafka servers that store and serve events

Producer - Publishing Events

A producer sends events to a topic. The key determines which partition the message goes to - all events with the same key land in the same partition, preserving order for that user.

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'payment-service',
  brokers: ['kafka-1:9092', 'kafka-2:9092'],
});

const producer = kafka.producer();

async function sendPaymentEvent(payment) {
  await producer.connect();
  await producer.send({
    topic: 'payments',
    messages: [
      {
        key: payment.userId,
        value: JSON.stringify({
          type: 'PAYMENT_COMPLETED',
          amount: payment.amount,
          currency: payment.currency,
          timestamp: new Date().toISOString(),
        }),
      },
    ],
  });
}

Consumer - Processing Events

Consumers belong to a consumer group. Kafka ensures each partition is consumed by only one member of the group, so messages are not processed twice. If a consumer crashes, Kafka reassigns its partitions to surviving members.

const consumer = kafka.consumer({
  groupId: 'notification-service',
});

async function startConsumer() {
  await consumer.connect();
  await consumer.subscribe({
    topic: 'payments',
    fromBeginning: false,
  });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const event = JSON.parse(message.value.toString());
      console.log(`Payment received: ${event.amount} ${event.currency}`);

      // Send notification, update analytics, etc.
      await sendEmailReceipt(event);
    },
  });
}

Running Kafka Locally

The easiest way to get started is Docker Compose. This gives you a single-broker Kafka cluster for local development.

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1