> ## Documentation Index
> Fetch the complete documentation index at: https://docs.springwinter.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# How to Use Kafka with Springwinter and Amazon MSK

> Connect Springwinter web servers and workers to Amazon MSK for production Kafka without running stateful brokers inside application containers.

The safe production architecture is Amazon MSK for Kafka brokers and Springwinter for producers, consumers, and stream-processing applications. Current Springwinter resources do not provision a Kafka cluster or durable broker volumes.

*Updated October 9, 2026.*

**Do not run a production Kafka broker as an ordinary Springwinter Worker. Use MSK and deploy Kafka clients on Springwinter.**

## Why brokers and clients need different infrastructure

Kafka brokers require stable identities, durable replicated storage, controlled rolling maintenance, partition rebalancing, and private multi-broker networking. A long-running application container is not a substitute for that stateful operating model.

Springwinter web servers and workers are a strong fit for Kafka clients because clients are replaceable compute. Producers publish events. Consumer groups divide partitions among replicas. Stream processors rebuild local state from Kafka topics and checkpoints.

## Recommended architecture

1. Create an Amazon MSK provisioned or serverless cluster in the same AWS Region and VPC used by the applications.
2. Keep broker endpoints private.
3. Permit network traffic from application task security groups to the MSK security group on the configured broker ports.
4. Choose IAM, SASL/SCRAM, or mTLS authentication.
5. Deploy HTTP producers as Springwinter web servers.
6. Deploy consumers and stream processors as Springwinter workers.
7. Monitor consumer lag, rejected connections, broker storage, and processing errors.

## IAM authentication

MSK supports IAM authentication for Java and non-Java clients. ECS applications should receive AWS credentials through a task IAM role rather than static access keys.

A Java client commonly uses:

```properties theme={null}
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```

Grant only the cluster, topic, group, and transactional actions that the application requires. Springwinter's current public configuration does not expose arbitrary per-resource task-role authoring, so verify that the deployed task role supports your chosen MSK authentication before committing to this design.

## Deploy the application clients

<Steps>
  <Step title="Create the MSK cluster">
    Provision MSK separately and record bootstrap broker endpoints, authentication mode, and security group.
  </Step>

  <Step title="Build the producer or consumer">
    Package the application in a Docker image. Read broker endpoints, topic names, group IDs, and authentication settings from environment variables.
  </Step>

  <Step title="Deploy on Springwinter">
    Use a Web server for HTTP-facing producers and a Worker for continuously polling consumers.
  </Step>

  <Step title="Validate failure behavior">
    Restart a consumer, revoke network access in a test environment, and confirm retries, lag alerts, and idempotent processing.
  </Step>
</Steps>

## Production checklist

* Use at least three Availability Zones when the selected MSK mode and Region support it.
* Set replication and minimum in-sync replica policies deliberately.
* Avoid automatic topic creation in production.
* Use idempotent producers where ordering and duplicate resistance matter.
* Track consumer lag by group and topic partition.
* Cap retries and route poison messages to a recovery topic.
* Test client compatibility before broker-version upgrades.

## Frequently asked questions

<AccordionGroup>
  <Accordion title="Can Springwinter deploy Kafka brokers today?">
    Not as a first-class durable resource. Springwinter Workers are replaceable ECS services and do not provide Kafka's required broker identity, replicated disk topology, or cluster lifecycle management.
  </Accordion>

  <Accordion title="Should I choose MSK Serverless or provisioned MSK?">
    Serverless reduces broker capacity management for compatible workloads. Provisioned MSK provides more explicit broker, storage, configuration, and scaling control. Compare protocol requirements, throughput shape, networking, and regional pricing.
  </Accordion>

  <Accordion title="Can Kafka clients run on Fargate?">
    Yes. Producers and consumers are ordinary containerized applications. They need network reachability, authentication, sufficient CPU and memory, and a controlled restart strategy.
  </Accordion>
</AccordionGroup>

## Sources and further reading

* [Amazon MSK documentation](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html)
* [Configure MSK clients for IAM access](https://docs.aws.amazon.com/msk/latest/developerguide/configure-clients-for-iam-access-control.html)
* [ECS task IAM roles](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.