Kafka Source

Consume messages from Kafka topics using a consumer group.

Kafka Source

The kafka source consumes messages from one or more Kafka topics using a consumer group.

Configuration

sources:
  - type: kafka
    name: kafka-main
    topics:
      - orders
      - users
    config:
      brokers: "localhost:9092"
      group_id: "litejoin-consumer"

Field	Type	Required	Description
brokers	string	yes	Comma-separated list of Kafka broker addresses.
group_id	string	yes	Consumer group ID.
topics	string[]	yes	List of Kafka topics to consume from.

Message Mapping

Kafka messages are mapped to LiteJoin messages as follows:

Kafka Field	LiteJoin Field
Topic	topic
Key	key
Value	payload
Timestamp	timestamp

Example

Consume orders and users from Kafka, join them, and forward to a webhook:

sources:
  - type: kafka
    name: kafka-main
    topics:
      - orders
      - users
    config:
      brokers: "broker1:9092,broker2:9092"
      group_id: "litejoin-prod"

joins:
  - name: order-user-join
    query: |
      SELECT o.key, o.payload, u.payload
      FROM orders o
      INNER JOIN users u ON json_extract(o.payload, '$.user_id') = u.key
      WHERE o.timestamp > (strftime('%s', 'now') - 3600)
    sink: webhook-out

sinks:
  - type: http
    name: webhook-out
    config:
      url: "http://localhost:9000/webhook"

When to Use

Use the Kafka source when:

• You already have a Kafka cluster producing events
• You want LiteJoin to enrich or join Kafka streams with other data sources
• You need consumer group semantics (offset management, partition assignment)

For systems without Kafka, use the API Source or HTTP Source.