Scene 09

Design canvas — pick the knobs

Designing a Kafka pipeline is choosing within a triangle of trade-offs (throughput, durability, simplicity); each knob — partition count, RF, MIR, acks, group strategy, EOS — tilts the triangle and forces a deliberate choice rather than a default.

Previously

You've seen every knob — log shape, partitions, replication, durability, compaction, leader epoch, rebalance, exactly-once. Time to assemble them for a real workload, and tilt the throughput / durability / simplicity triangle deliberately rather than by default.

Scene 09

Design canvas — pick the knobs

Watch
Try it
Predict
Capture

What to watch for

This is the architecture-interview question. The defaults — P=3, RF=2, MIR=1, acks=1, single group, EOS=off — are wrong on purpose. Read the warnings, watch the trade-off bars, and feel where the design is exposed before you start tuning.

Implementation

Highlighted lines are the ones running in the diagram right now.

producer.properties

producer-side dials — durability contract & retry shape

1bootstrap.servers=broker-1:9092,broker-2:9092
2acks=0              # 0 fire-and-forget
3acks=1              # 1 leader-only ack
4acks=all            # all in-ISR replicas
5 
6enable.idempotence=true     # PID + seq# dedupe retries
7transactional.id=clickstream-tx-1
8                            # cross-session epoch fencing
9 
10compression.type=lz4        # batch-level; cheap throughput
11linger.ms=20                # fill batches before sending
12max.in.flight.requests.per.connection=5

server.properties (broker)

broker-side dials — replica policy & election safety

1broker.id=1
2log.dirs=/var/lib/kafka/data
3 
4default.replication.factor=<RF>
5min.insync.replicas=<MIR>   # floor on the ISR
6                            # rule of thumb: MIR = RF - 1
7unclean.leader.election.enable=false
8                            # silent loss > unavailability
9 
10replica.lag.time.max.ms=30000
11                            # ISR eviction threshold
12log.retention.ms=604800000  # 7d default; per-topic overrides
13log.cleanup.policy=delete   # not compact: events expire

kafka-topics --alter (per-topic)

per-topic overrides — what the brief actually demands

1kafka-topics.sh --bootstrap-server broker-1:9092 \
2  --create --topic clickstream \
3  --partitions <P> --replication-factor <RF> \
4  --config min.insync.replicas=<MIR> \
5  --config cleanup.policy=delete \
6  --config retention.ms=604800000 \
7  --config compression.type=producer \
8  --config segment.ms=3600000
9 
10# downstream consumer (warehouse / dashboard):
11isolation.level=read_committed
12                            # skip aborted txn markers

Sources

Where this sits in Build Kafka

Scene 09 of 13, in the Guarantees act — Three monotonic counters; one design canvas.. Capstone: apply scenes 2-8 to a fresh problem and articulate the trade-off you took.

All 13 scenes in Build Kafka · Every curriculum

Designs that use this

Built with Arqly

Every scene in Build Kafka builds on the one before it.All 13 Build Kafka scenes