Spark & Big Data Interview Questions

Sample Answer

Spark will recompute lost work if it can rerun the failed tasks using lineage and the failure is within configured retry limits, it will fail the job if retries are exhausted or the failure is non-recoverable like repeated fetch failures or a Driver-side exception. The Driver owns scheduling, it tracks task attempts per stage and resubmits failed tasks to available executors, including newly launched ones. Whether recomputation is possible depends on RDD/DataFrame lineage, shuffle state, and whether shuffle files are still available, especially without external shuffle service. Settings like max task failures, stage retries, and fetch failure handling control when the scheduler gives up and aborts.

Sample Answer

You could cache after the join or do nothing and let each action build its own job, caching wins here because Spark will otherwise execute two separate jobs and recompute upstream stages for the second action. Without cache, the Driver will submit the first action's job, run the join stages, then submit a second action's job that rebuilds the same stages again. With cache persisted, the first action materializes the cached partitions on executors, and the second action reads those cached blocks, skipping the expensive join stages. You pick the storage level based on executor memory pressure, because eviction can silently force recomputation.

Sample Answer

You call collect(), that is the action that makes the Driver ask the scheduler to build a job from the logical plan. Spark plans operators, then splits execution into stages at shuffle boundaries, explode and pre-aggregation run in parallel tasks on executors, then groupBy triggers a shuffle so executors exchange partitions by key. After the final aggregation, every partition's result is sent back over the network to the Driver because collect() requires the full result locally. The Driver OOMs because it must hold the entire collected dataset in its JVM or Python process, even if executors did most compute, so you should prefer write, take($n$), or reduce-style aggregation that returns small results.

Sample Answer

You could keep a typed Dataset for compile-time safety, or use an untyped DataFrame for maximum Catalyst freedom. DataFrame wins here because it avoids frequent JVM object materialization from encoders and keeps data in Tungsten's binary format longer, which improves whole-stage codegen and reduces GC pressure. Dropping to RDDs only makes sense if you truly need non-relational, custom stateful logic that Catalyst cannot express, otherwise you add serialization overhead and lose optimizer-driven join and aggregation strategies.

Sample Answer

First, you check the Spark UI plan to see which physical join is chosen and where spills occur, then you decide if the plan is fundamentally shuffle-heavy. If the join is a sort-merge join, spilling is usually from insufficient memory for sorting and aggregation buffers, not from Catalyst itself, so you try to change join strategy, like broadcasting the smaller side or pre-filtering to reduce shuffle volume. If GC is high, you look for UDFs or Dataset encoder paths that force row materialization, because that breaks Tungsten efficiency and increases object churn. Your first change is usually to remove row-wise UDFs, ensure selective filters are pushed before the join, and make the smaller table broadcastable if feasible.

Sample Answer

First, groupBy(user_id) requires all rows for the same user_id to land in the same partition, so Spark does a shuffle to repartition by user_id. Next, the count happens within each partition after the shuffle, producing one row per user_id. Then you join that aggregated result back to the original data, and because neither side is co-partitioned on user_id, Spark typically triggers another shuffle to align partitions for the join. If the aggregated side is small enough it could be broadcast, which removes the second shuffle, but the groupBy shuffle still remains.

Sample Answer

This question is checking whether you can recognize skew as a data distribution problem, not a cluster sizing problem. You detect it by looking at per-partition input sizes and task durations in the Spark UI, and by sampling key frequencies to confirm a hot key. You mitigate it by changing the join plan, for example salting the hot key on the large side and expanding the small side to match, or using Spark AQE skew join handling when applicable. Simply raising shuffle partitions often just creates more tiny tasks while the hot partition stays hot.

Sample Answer

The standard move is to co-locate data by the join keys so Spark can do a shuffle-free or reduced-shuffle join, by bucketing both tables on (country, day) with the same number of buckets, or by writing them out partitioned and then reading with those partitions preserved. But here, file layout and compatibility matter because bucketing only helps if both sides are bucketed identically and Spark can actually recognize and exploit it. It also does not help if the join keys are heavily skewed, because you still get one or a few huge buckets. If your workload is ad hoc with varying keys, pre-bucketing can become wasted write cost and operational complexity.

Sample Answer

Get this wrong in production and you trigger shuffle spill storms, executor OOMs, and long tail retries that make the pipeline miss its SLA. The right call is to prevent the multiplicative join when you do not need it, for example pre-aggregate or deduplicate the right side to enforce one row per key, or filter it down before the join. If you do need all matches, you make the join more stable by broadcasting the smaller side when safe, increasing memory overhead for shuffle, and enabling AQE so Spark can change join strategies at runtime. You also validate the expected output cardinality up front, because a 50x expansion is often a data quality bug, not a tuning problem.

Sample Answer

This question is checking whether you can map GC symptoms to object churn, oversized cached or shuffle data, and inefficient join strategies instead of randomly raising executor memory. First, use the SQL tab and stage metrics to see whether the pressure is from cached storage, shuffle, or join build side, look at executor peak memory, GC time, spill, and whether broadcast joins are happening. If GC is driven by large shuffles, increase shuffle parallelism and consider AQE, if it is driven by a huge broadcast, cap `spark.sql.autoBroadcastJoinThreshold` or force a shuffle join, and if it is driven by storage, stop caching or cache smaller projections. Config-wise, keep executor cores modest to reduce per-JVM contention, and size memory so the on-heap overhead and execution memory are realistic for your task footprint.

Sample Answer

The standard move is to confirm skew by comparing task time distribution and per-task shuffle read or records, you will see one or a few tasks with massively higher input size and runtime. But here, the mitigation depends on whether skew is on a join key and whether the other side is small enough to broadcast. If it is a skewed join, enable AQE skew join handling and verify it is splitting skewed partitions, or apply salting on the skewed key to spread the hot key across $n$ buckets, then desalt after aggregation if needed. If one side is small, broadcasting that side often avoids the skewed shuffle entirely, but you must validate broadcast size and executor memory headroom to prevent GC and OOM.

Sample Answer

Get this wrong in production and you accumulate thousands of tiny files, metadata overhead grows, and every read pays extra planning and IO cost even if the data volume is stable. The right call is to control file sizes on write, use reasonable `maxRecordsPerFile` or adaptive coalescing, and periodically compact, for Delta that can be `OPTIMIZE`, for Parquet it can be a compaction job. You also revisit partitioning, avoid over-partitioning by high-cardinality columns, and prefer clustering techniques like Z-ordering where available for common predicates. Validate by tracking file count and average file size per partition, plus query planning time and input file count in the Spark UI.

Sample Answer

The standard move is to use dropDuplicates on event_id with a watermark on event_time, so Spark stores seen ids only for the watermark horizon. But here, days-late resends matter because a small watermark will let duplicates re-enter after state eviction, and a huge watermark can blow up state. You align the watermark to a business TTL for dedup, or you move dedup to the sink with an idempotent upsert, for example a Delta MERGE keyed by event_id. If you must dedup in-stream, you also monitor state store size and tune state TTL, shuffle partitions, and checkpoint durability.

Sample Answer

Get this wrong in production and you ship silent overcounting, which is worse than an outage. The right call is to recognize that Spark can provide exactly-once processing with checkpointing, but the sink must be transactional or idempotent, otherwise retries reapply side effects. You fix it by using a sink with atomic commits (Delta, Kafka with transactions, or a database transaction per batch), or by making writes idempotent using a deterministic key plus upsert, and recording a processed batch id to prevent replays. If the sink cannot support this, you at best get at-least-once end-to-end even if Spark itself recovers correctly.

Sample Answer

Windowed aggregation sounds reasonable but breaks under true session logic because you need dynamic boundaries and custom state per user. mapGroupsWithState does not work because it emits at most one output per key per trigger, which can be limiting for session close plus intermediate updates. That leaves flatMapGroupsWithState with EventTimeTimeout, where you update per-user state on each event and emit zero or more rows, including a final summary when the timeout fires. You tie the timeout to event time and set a watermark so Spark can advance event-time and eventually trigger timeouts for inactive users.