Producer deployment pattern
An HTTP producer validates a request, creates a stable event ID, publishes with acknowledgements enabled, and returns only after the required durability level is met. Configure broker addresses, topic names, authentication, compression, request timeouts, and retry limits through environment variables. For ordered events, choose a stable message key such as customer ID or aggregate ID. Kafka guarantees ordering within a partition, not across an entire topic.Consumer deployment pattern
A Springwinter Worker can join a consumer group and continuously poll records. Each replica should use the same group ID when instances divide work. Use a different group ID when applications need independent copies of every event. Commit offsets after the business side effect succeeds. If a database write and offset commit cannot be atomic, make the write idempotent with an event ID or use an outbox-compatible design.Handle shutdown and rebalancing
When the container receivesSIGTERM:
- Stop polling for new records.
- Finish or abandon in-flight records before the deployment drain limit.
- Commit only completed offsets.
- Close the consumer so partitions rebalance promptly.
- Allow unfinished records to be processed again.
Deploy with Springwinter
1
Build one explicit process
Create a Docker image whose command starts either the producer API or the consumer. Avoid running unrelated process types in one container.
2
Add runtime configuration
Configure bootstrap brokers, authentication, topic, group ID, concurrency, and log level as environment variables.
3
Choose the resource type
Deploy HTTP producers as Web servers. Deploy consumers and stream processors as Workers.
4
Observe lag and failures
CloudWatch container metrics show resource pressure, but Kafka lag requires client or broker metrics. Alert on lag age, failed processing, rebalance frequency, and dead-letter volume.
Retry strategy
Do not block a partition forever on one bad event. Use bounded local retries for transient failures, then publish the record and error context to a retry or dead-letter topic. Preserve the original key, timestamp, topic, partition, offset, and event ID.Capacity planning
Consumer parallelism cannot exceed useful partition parallelism. Ten worker replicas do not provide ten-way processing when the topic has three partitions. Increase partitions cautiously because partition count affects ordering, open files, metadata, and future rebalancing.Frequently asked questions
How many Springwinter Workers should a Kafka consumer use?
How many Springwinter Workers should a Kafka consumer use?
Start with enough replicas to meet throughput and availability goals, but no more active consumers than useful partitions in the group. Measure processing time, lag, memory, and downstream limits before increasing replicas.
Should consumers enable automatic offset commits?
Should consumers enable automatic offset commits?
Automatic commits are simple but can acknowledge work before the side effect finishes. Explicit commits after successful, idempotent processing provide clearer failure semantics for important workloads.
Where should failed Kafka messages go?
Where should failed Kafka messages go?
Route permanently failing records to a recovery topic with error metadata. Keep the original payload and coordinates so operators can inspect, fix, and replay safely.