> ## Documentation Index
> Fetch the complete documentation index at: https://docs.springwinter.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy Kafka Producers and Consumers on Springwinter

> Deploy Kafka producers, consumer groups, and stream processors on Springwinter with safe offsets, retries, idempotency, rebalancing, and lag monitoring.

Springwinter can run containerized Kafka producers and consumers while Amazon MSK or another reachable Kafka service owns the brokers. Use a web server for request-driven producers and a worker for long-running consumers.

*Updated October 9, 2026.*

**Consumer correctness depends on when offsets are committed, not only whether the container stays running.**

## Producer deployment pattern

An HTTP producer validates a request, creates a stable event ID, publishes with acknowledgements enabled, and returns only after the required durability level is met. Configure broker addresses, topic names, authentication, compression, request timeouts, and retry limits through environment variables.

For ordered events, choose a stable message key such as customer ID or aggregate ID. Kafka guarantees ordering within a partition, not across an entire topic.

## Consumer deployment pattern

A Springwinter Worker can join a consumer group and continuously poll records. Each replica should use the same group ID when instances divide work. Use a different group ID when applications need independent copies of every event.

Commit offsets after the business side effect succeeds. If a database write and offset commit cannot be atomic, make the write idempotent with an event ID or use an outbox-compatible design.

## Handle shutdown and rebalancing

When the container receives `SIGTERM`:

1. Stop polling for new records.
2. Finish or abandon in-flight records before the deployment drain limit.
3. Commit only completed offsets.
4. Close the consumer so partitions rebalance promptly.
5. Allow unfinished records to be processed again.

Assume duplicate delivery. Exactly-once marketing language does not remove application-level side effects outside Kafka.

## Deploy with Springwinter

<Steps>
  <Step title="Build one explicit process">
    Create a Docker image whose command starts either the producer API or the consumer. Avoid running unrelated process types in one container.
  </Step>

  <Step title="Add runtime configuration">
    Configure bootstrap brokers, authentication, topic, group ID, concurrency, and log level as environment variables.
  </Step>

  <Step title="Choose the resource type">
    Deploy HTTP producers as Web servers. Deploy consumers and stream processors as Workers.
  </Step>

  <Step title="Observe lag and failures">
    CloudWatch container metrics show resource pressure, but Kafka lag requires client or broker metrics. Alert on lag age, failed processing, rebalance frequency, and dead-letter volume.
  </Step>
</Steps>

## Retry strategy

Do not block a partition forever on one bad event. Use bounded local retries for transient failures, then publish the record and error context to a retry or dead-letter topic. Preserve the original key, timestamp, topic, partition, offset, and event ID.

## Capacity planning

Consumer parallelism cannot exceed useful partition parallelism. Ten worker replicas do not provide ten-way processing when the topic has three partitions. Increase partitions cautiously because partition count affects ordering, open files, metadata, and future rebalancing.

## Frequently asked questions

<AccordionGroup>
  <Accordion title="How many Springwinter Workers should a Kafka consumer use?">
    Start with enough replicas to meet throughput and availability goals, but no more active consumers than useful partitions in the group. Measure processing time, lag, memory, and downstream limits before increasing replicas.
  </Accordion>

  <Accordion title="Should consumers enable automatic offset commits?">
    Automatic commits are simple but can acknowledge work before the side effect finishes. Explicit commits after successful, idempotent processing provide clearer failure semantics for important workloads.
  </Accordion>

  <Accordion title="Where should failed Kafka messages go?">
    Route permanently failing records to a recovery topic with error metadata. Keep the original payload and coordinates so operators can inspect, fix, and replay safely.
  </Accordion>
</AccordionGroup>

## Sources and further reading

* [Apache Kafka consumer configuration](https://kafka.apache.org/documentation/#consumerconfigs)
* [Amazon MSK best practices](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html)
* [Springwinter Workers](/deploy/workers)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.