Streaming (Kafka) Interview Questions

Sample Answer

Producers with acks=all will temporarily fail writes for partitions that cannot maintain at least 2 in-sync replicas, while partitions that still have 2 ISR members will continue accepting writes after leader failover. When the leader broker dies, the controller triggers leader election from the ISR; with unclean election disabled, Kafka will not elect an out-of-sync replica, so it prefers availability with no data loss only where ISR is sufficient. Consumers will see a short rebalance or fetch retry while metadata updates, then continue from the new leaders, with no message loss, but with possible lag increase during the failover window. If ISR shrinks below 2 for a partition, producers block or error until a replica catches up and rejoins ISR.

Sample Answer

You could add partitions or add replicas with follower fetching. Partitions win when you need more parallelism per consumer group and higher aggregate write throughput, but they change ordering scope and increase metadata and rebalance costs. Replicas plus follower fetching win when writes must stay stable and you want to scale reads across brokers, but you pay extra replication IO and you must ensure clients are configured to read from followers safely. In practice, if the bottleneck is many groups reading the same data, replicas with follower reads are often the cleanest way to scale reads without changing partitioning semantics.

Sample Answer

First, the old controller session expires and a new broker becomes controller via the metadata quorum, then it takes over managing partition leadership and ISR changes. Next, the controller rebuilds cluster state and reissues leader and ISR assignments as needed, but it usually does not move leaders unless there was an in-flight change or a broker failure. Clients may see transient metadata errors like NOT_CONTROLLER or stale metadata, then they refresh metadata and continue producing and consuming. You expect brief latency spikes, possible consumer group rebalances if coordinators move, and produce retries, but not data loss. The duration is typically bounded by session timeouts and client retry settings, so poor configs can amplify the incident.

Sample Answer

You could pull more per poll by increasing max.poll.records, or you could make each fetch return fuller batches by increasing fetch.min.bytes and allowing a bit more fetch.max.wait.ms. max.poll.records wins when your poll loop overhead is high and you can process bigger chunks without exceeding max.poll.interval.ms. fetch.min.bytes and fetch.max.wait.ms win when network overhead dominates and you want fewer, larger fetch responses, but you must watch added tail latency if traffic is bursty. In both cases you validate by checking time in poll loop, request rate to brokers, and whether you are flirting with rebalances due to slow polling.

Sample Answer

First, the rebalance hint says you are not calling poll often enough, so the coordinator kicks you out when you exceed max.poll.interval.ms. Next, you reduce the time between polls by shrinking the work done on the poll thread: hand off records to a worker pool and keep polling, or lower max.poll.records so each batch is faster to process. Then you set max.poll.interval.ms and session.timeout.ms to match your worst case processing time, but only after you have removed long blocking work from the poll loop. Finally, commit strategy matters: if you process asynchronously, you commit offsets only after processing finishes, otherwise you risk skipping work on restart.

Sample Answer

This question is checking whether you can connect partitioning and consumer group assignment to lag and utilization, then choose mitigations that preserve ordering constraints. Today, you can change the producer keying scheme to spread traffic, for example use a composite key or add controlled salting if strict per original key ordering is not required. On the consumer side, you can isolate the hot partition into its own group or dedicated instances so it does not throttle unrelated work, and you can raise parallelism inside that consumer if processing is parallelizable after fetch. Longer term, you add partitions and revisit key design so the hash distribution matches traffic, because consumer group parallelism is capped by partition count.

Sample Answer

First, idempotent producer dedupes retries per producer session using PID and sequence numbers, so it prevents duplicates from network retries to the same partition. Next, a broker failover can force a producer to restart and get a new PID, so the broker cannot correlate the new session with the old one, and duplicates can appear across sessions. Also, idempotence does not give atomicity across multiple partitions, so partial writes can be observed. To prevent duplicates from being visible, you layer in transactions, produce within a transaction, and have consumers use read_committed so aborted or partial results are not read.

Sample Answer

This question is checking whether you can describe exactly once semantics as atomic visibility plus atomic offset advancement, not magic no-duplicate guarantees everywhere. You wrap your output writes and your consumer offset commits in the same Kafka transaction, so either both are committed or both are aborted. If you die after producing but before committing, the transaction never commits, so consumers with read_committed will not see those output records. On restart, you reprocess from the last committed offset, and because the previous output was aborted, you do not double-apply it.

Sample Answer

The standard move is to treat Kafka as at least once, then make the sink idempotent by writing deterministically, for example by using a unique object key based on topic-partition-offset, and only then committing offsets. But here, retries and multipart uploads matter because you can still create multiple objects or partial objects unless you ensure atomic replace semantics and a single final key. Also, offset commit after S3 write does not prevent duplicates when a crash happens after the write, so consumers will replay and attempt the same write again. You call it exactly once effects only if your S3 layout and compaction logic makes replays overwrite or become no-ops.

Sample Answer

This question is checking whether you can translate a join requirement into bounded state with clear time semantics and operational guardrails. You should implement a keyed stream stream join on session id using event time, store one side in state and match as the other arrives, and set a join window and TTL at 10 minutes plus a safety margin. To handle skew, you add hot key mitigation, for example shard state by adding a salt or split sessions into subkeys if the business logic allows, and you cap per key state with spill to RocksDB plus metrics and alerts. To hit p99 under 2 seconds, you prioritize local state access, avoid cross partition lookups, tune consumer fetch and processing threads, and define what happens to late events, either update attribution with upserts or route to a late events topic.

Sample Answer

The standard move is to use an approximate distinct structure like HyperLogLog to keep state small and updates cheap. But here, determinism for audits matters because approximate sketches can vary with implementation and merges, so you may need exact distinct with a bounded set or a two tier design: exact within a shorter horizon, plus a batch audited store for the full 24 hours. You size state by estimating cardinality per merchant and retention, for example $\text{state} \approx \sum_m |\text{cards}_m| \cdot \text{bytes per entry}$, and you enforce TTL aligned to event time with a small grace period for lateness. For recovery, you rely on changelog topics and periodic snapshots, and you validate that replay plus exactly once processing yields identical counts under redeploy.

Sample Answer

Get this wrong in production and you get a positive feedback loop: consumer lag increases, windows hold more buffered data, RocksDB compaction spikes, and your output topic becomes bursty and knocks over downstream services. The right call is to make backpressure explicit: cap in flight records, tune max poll and processing threads, and shed non critical work, for example reduce window hop frequency or disable early emits during overload. You also protect state by bounding retention and using event time grace correctly, monitor and tune RocksDB (block cache, write buffers, compaction), and ensure changelog topics have adequate partitions and throughput. Finally, you define an overload policy tied to SLOs, for example temporarily route late and non critical events to a side topic for deferred processing while keeping critical aggregations exact.

Sample Answer

The standard move is to add the field as optional with a default, ship producers first, then update consumers, then later enforce requiredness via validation and policy. But here, the deadline matters because slow consumers will not pick up required field handling in time, so you avoid hard schema enforcement that would break them and instead enforce compliance at the edge, for example reject or quarantine messages missing the new field in a side topic or via stream processing. You keep compatibility at BACKWARD or FULL and use schema references or versioned contracts so producers can publish both old and new representations during the transition. When all critical consumers have upgraded, you can tighten rules with a new major version or a new topic and deprecate the legacy path.

Sample Answer

Get this wrong in production and consumers either start reading nulls or crash on missing fields, and you silently lose join keys and attribution. The right call is to treat renames as additive, keep "userId" and add "user_id" with a default, then have producers write both and consumers read either with precedence. In Avro you can also use field aliases so old readers can map the new name, but you still rollout carefully because not all tooling respects aliases equally. After adoption, you deprecate the old name in documentation and linting, then eventually remove it only with a new major version or a new topic.