May 15, 2026

Mocking Kafka Producers Without a Real Broker

Standing up Kafka for tests is overkill. Mocking it badly is worse. Here's how to fake the broker convincingly enough that your producer and consumer code actually exercises the wire protocol — without the operational tax.

Three levels of fake

Before you write code, decide which level you need. Each has a different cost and a different blast radius:

Mock the client library. Replace kafka.Producer with a spy. Cheapest. Doesn't exercise serialization, partitioning, or the broker protocol — so a producer config error slips past your tests entirely.
Use an in-memory testcontainers Kafka. Real broker, ephemeral. Catches everything but adds 15–30 seconds of startup per test and pins you to Docker in CI.
Use a wire-compatible mock broker. Speaks the Kafka protocol, runs in-process or as a sidecar, no JVM. Exercises serialization, partitioning, and consumer-group behavior; doesn't try to be a real broker for performance tests or log mechanics. The sweet spot for unit and integration tests.

This post is about option three. If you want option one, your client library has a mock package already and you don't need a blog post.

What "wire-compatible" actually means

A wire-compatible Kafka mock implements the parts of the Kafka protocol your client cares about — produce, fetch, metadata, group coordination — well enough that the client's unmodified code dials it and works. It usually doesn't implement replication, log compaction, or anything that requires real durability. That's the trade: complexity gone, realism kept where it matters for tests.

MockForge's Kafka listener falls in this category. So does Confluent's kafka-rest for the REST proxy case, and parts of the Kafka Streams test utilities if you're in the JVM world.

Setting it up

With MockForge installed (cargo install mockforge-cli), bring up the Kafka listener with topic auto-creation enabled and (optionally) seed messages your tests will consume:

# mockforge.yaml
kafka:
  enabled: true
  port: 9092
  auto_create_topics: true
  default_partitions: 3
  default_replication_factor: 1
  seed_messages:
    orders.created:
      - key: "order-001"
        value: '{"order_id":"order-001","total":4299}'

$ mockforge serve --config mockforge.yaml
[kafka] Listening on 0.0.0.0:9092 (auto_create_topics: true)
[kafka] Seeded 1 message across 1 topic at broker startup

Pointing your producer at it

Use whatever Kafka client you already have. The mock advertises localhost:9092 through its metadata response so your client doesn't notice it's not a real broker. In Python with confluent-kafka:

from confluent_kafka import Producer

p = Producer({"bootstrap.servers": "localhost:9092"})
p.produce("orders.created", key="order-002", value='{"order_id":"order-002"}')
p.flush()

The producer's serialization, partitioning, and ack handling all run as they would against a real broker. If your config is wrong (a missing key.serializer, a bad compression setting), the mock surfaces the same error the real broker would.

Testing the consumer side

Seeded messages from the config show up on the topic at offset 0 immediately — a consumer reading from beginning sees them without any setup. You can also publish from the test itself:

def test_consumer_handles_order_created():
    # Mock broker already has order-001 seeded from mockforge.yaml
    handler = OrderHandler()
    handler.consume("localhost:9092", "orders.created", group_id="test-1")
    assert handler.processed_count == 1
    assert handler.last_order_id == "order-001"

Same key always lands on the same partition (Kafka's hash-on-key contract), so seeded records distribute exactly the way real produced records would. Each test starts the broker fresh, so the only messages a consumer sees are the ones in seed_messages plus anything the test publishes inline.

Failure-injection: where mock brokers earn their keep

The reason to use a mock instead of a real broker for tests isn't speed — it's that you can introduce specific failures deterministically. With MockForge, fault injection is config:

kafka:
  port: 9092
  faults:
    - topic: orders.created
      partition: 1
      kind: produce_throttle
      delay_ms: 2000
    - topic: orders.shipped
      kind: offset_out_of_range
      probability: 0.05
    - topic: critical-pipeline
      kind: produce_not_leader

Your retry-and-backoff code actually gets exercised. The 2-second produce throttle on partition 1 forces your client through its retry path. The 5% out-of-range error on orders.shipped surfaces consumer reset behavior. The produce_not_leader rule makes the client re-fetch metadata and retry. None of these is easy to reproduce on a real broker.

Wiring into CI

GitHub Actions example. Background the mock, wait for the port to open, run the tests:

- run: cargo install mockforge-cli
- name: Start mock Kafka
  run: |
    mockforge serve --config mockforge.yaml &
    timeout 30 bash -c 'until nc -z localhost 9092; do sleep 0.5; done'
- run: pytest tests/integration/

When this isn't enough

Wire-compatible mocks don't replicate the actual log mechanics — they're not the right tool for testing log compaction behavior, replica leadership transitions, or performance under load. If your test specifically depends on those, you want a real broker via testcontainers and you're going to pay the startup tax.

For the case where you mostly want to know whether your producer config works and whether your consumer handles the messages it gets, a wire-compatible mock turns a flaky 30-second CI step into a reliable one-second one.

Kafka Docs GitHub