Kafka Streaming Engineering Roadmap

Build the mental model of a replicated partitioned log. Connect every configuration to an application ordering, durability, replay, or cost requirement.

• Explain records, offsets, topics, partitions, leaders, followers, and in-sync replicas
• Use stable aggregate keys for required per-entity ordering
• Distinguish partition order from nonexistent topic-wide total order
• Choose partition counts from measured parallelism, throughput, skew, and operational cost
• Understand replication factor, acks, and minimum in-sync replicas as a trade-off
• Design time and size retention around replay and compliance needs
• Use compaction only for keyed latest-state use cases with tested tombstones
• Inventory internal, business, quarantine, replay, and processor-state topics
• Measure record size, compression, batch behavior, disk use, and network transfer
• Create a failure matrix for broker, controller, disk, network, and under-replicated state

Make every retry and commit point explicit. Separate producer retry deduplication from business idempotency and Kafka-atomic transactions from external side effects.

• Configure durable acknowledgments, idempotence, retries, delivery timeout, batching, and compression
• Handle asynchronous callbacks and classify retriable, fatal, authorization, and serialization failures
• Understand group assignment, useful parallelism, coordinator state, heartbeats, and poll liveness
• Choose offset reset behavior for new, expired, and out-of-range positions
• Commit only completed work and handle revocation and shutdown safely
• Make external writes idempotent with stable event identity and atomic deduplication where possible
• Use Kafka transactions for consume-transform-produce workflows and downstream read_committed
• Document why transactions do not automatically cover databases, emails, or payment-like calls
• Bound retries, pause partitions, and apply backpressure without losing group ownership
• Inject timeouts, retries, crashes, rebalances, and destination outages at every commit boundary

Treat a schema and partition key as durable contracts. Then build keyed state that can be recovered, upgraded, and rescaled without inventing stronger guarantees than the source and sink support.

• Choose Avro, Protobuf, or JSON Schema using current format-specific rules
• Define subject naming, ownership, versioning, and compatibility policy
• Test backward, forward, full, and transitive behavior with retained records
• Coordinate producer and consumer deployment order with the chosen compatibility mode
• Use Kafka Streams keys, repartition topics, state stores, changelogs, and processing guarantees deliberately
• Use Flink keyed state and understand key groups and rescaling
• Enable durable Flink checkpoints and measure checkpoint versus replay trade-offs
• Assign stable Flink operator UIDs before relying on savepoint restore
• Test state schema and topology evolution rather than assuming restore compatibility
• Validate late, out-of-order, duplicate, window, and terminal-state behavior against known truth

Operate streaming as a distributed application. Restrict every principal, correlate asynchronous work safely, detect stale processing, test quotas and overload, and attribute retained and replicated bytes.

• Use TLS with server verification and approved client authentication
• Grant distinct principals least-privilege topic, group, transactional-ID, schema, and admin access
• Test denied reads, writes, group use, schema changes, and administration
• Use quotas and bounded client configuration to contain noisy producers and consumers
• Observe broker requests, replication, controllers, storage, networks, and throttling
• Observe producer errors, retries, record rates, latency, and delivery outcomes
• Observe consumer assignments, rebalances, commits, lag, processing time, and event age
• Follow current OpenTelemetry messaging conventions and use links for batches and fan-out
• Exclude payloads, credentials, and unbounded business IDs from metrics and routine telemetry
• Attribute partitions, replicas, retention, compute, state, transfer, schemas, and telemetry costs

Turn retention into a controlled recovery capability. Preserve poison-record evidence, isolate backfills, measure asynchronous replication, fence writers, translate consumer positions, restore processor state, and reconcile every uncertain event.

• Classify transient and permanent failures before retry, stop, quarantine, or skip
• Preserve original topic, partition, offset, event ID, schema identity, and safe reason
• Use dedicated replay groups, rate limits, idempotent outputs, and suppressed irreversible effects
• Reconcile input, unique event IDs, quarantine, terminal state, and output before cutover
• Design explicit topic replication direction, allowlists, naming, loop prevention, and retention
• Measure replication lag, heartbeat freshness, checkpoint state, and consumer offset translation
• Set honest RPO and RTO targets and test broker, connector, schema, network, and processor failures
• Fence or freeze source writers before active-passive failover and failback
• Restore Flink state with stable UIDs and validate source and sink consistency
• Dispose topics, schemas, snapshots, credentials, telemetry, clusters, and retained lab data