Databricks interviews for data engineers rarely stop at Spark syntax. They ask what happens inside a Delta table when two jobs write at once, why one task in a stage runs for forty minutes, how a streaming job recovers after a crash and who is allowed to see which column. Databricks is a platform, and the questions follow its moving parts.
Candidates tend to slip in one of two ways. Some know open-source Spark well but have never thought about the Delta transaction log, VACUUM retention or Unity Catalog privileges. Others learned Databricks a few years ago and still describe Z-ORDER as the default layout and all-purpose clusters as the place for nightly jobs. Product names have moved too: Delta Live Tables is now Lakeflow pipelines, built on Apache Spark Declarative Pipelines, and Workflows is now Lakeflow Jobs.
The 30 questions below are grouped into five areas, each with a short note on what interviewers look for. Every solution states the answer, the reason and the trap, with SQL or PySpark where code makes the point clearer. Defaults and names were checked against the Databricks, Delta Lake and Apache Spark documentation in October 2026. A prep plan, a quick quiz and answers to common questions follow the questions.
Intermediate30 questions
Databricks Interview Questions
Practice questions on Delta Lake, data layout, Spark performance, Unity Catalog, Auto Loader, Lakeflow pipelines and compute choices, the areas Databricks interviews test most.
Data EngineerAnalytics EngineerMachine Learning EngineerData ArchitectData ScientistPlatform Engineer
Almost every table on Databricks is a Delta table, so interviewers expect you to know what happens underneath a write. They ask about the transaction log, concurrent writers, recovering from a bad job and why deleted rows are not gone yet, because these are the incidents data engineers actually get paged for.
The one idea to carry through this section: a Delta table is a set of Parquet files plus an ordered log that says which files make up each version. ACID commits, time travel, VACUUM, MERGE and deletion vectors are all consequences of that log.
Delta Lake Transactions, Time Travel and MERGE
A job appends 500 files to a Delta table while a dashboard queries it. Explain, in terms of files on storage, why the dashboard never sees half of the new data.
EasyDelta Lake Transactions, Time Travel and MERGE
Sample Answer
A Delta table is a directory of Parquet files plus a _delta_log folder. Each commit writes one numbered JSON file (00000000000000000042.json) listing the data files added and removed, and periodic Parquet checkpoints summarize the log so readers don't replay thousands of JSON files. The writer first writes all 500 data files, then publishes them in a single atomic commit of the next version file. A reader picks the latest version it can see and reads exactly the files that version lists. Until the commit lands, the new files exist but belong to no version. The trap: deleting or copying files directly in cloud storage bypasses the log and can corrupt the table.
Two scheduled jobs run MERGE into the same Delta table at the same time, and one fails with ConcurrentAppendException. Why does Delta fail the job instead of locking, and how do you reduce these conflicts?
MediumDelta Lake Transactions, Time Travel and MERGE
Sample Answer
Delta uses optimistic concurrency control. Each writer reads a snapshot, does its work, then tries to commit the next version. Before committing it checks whether transactions committed in the meantime touched data it read; if they did, it fails rather than produce a result based on stale data. The default isolation level is WriteSerializable, under which blind appends never conflict, but two MERGEs reading overlapping files can. Fixes: make the conditions explicit and narrow (add the date or partition range to the ON clause) so each job reads disjoint files; enable deletion vectors, which turn on row-level concurrency so writers to different rows in the same file no longer conflict (partitioned tables don't get it); and retry with backoff.
A buggy job overwrote yesterday's partition of sales.orders two hours ago. Walk through finding the damage and rolling back.
MediumDelta Lake Transactions, Time Travel and MERGE
Sample Answer
Start with DESCRIBE HISTORY, which lists each version with its operation, user, timestamp and metrics such as rows written, in reverse chronological order. Find the last good version, compare it with the current one using time travel (VERSION AS OF), then roll back with RESTORE TABLE ... TO VERSION AS OF. RESTORE is itself a new commit, so history is kept and you can undo the undo. Time travel needs both the log entries and the data files for that version: the log is kept for delta.logRetentionDuration (30 days by default) and VACUUM removes unreferenced files older than delta.deletedFileRetentionDuration (7 days by default). Downstream streams reading the table will see the restore as a change, so check them afterward.
SQL
1DESCRIBE HISTORY sales.orders LIMIT5;23-- compare the bad version with the last good one (here 41)4SELECTcount(*)FROM sales.orders VERSION ASOF415WHERE order_date =current_date()-INTERVAL1DAY;67RESTORETABLE sales.orders TO VERSION ASOF41;
To save storage, someone disabled the retention check and ran VACUUM events RETAIN 0 HOURS. That night a long-running job failed with missing-file errors and time travel stopped working. Explain what happened and what the safe settings are.
HardDelta Lake Transactions, Time Travel and MERGE
Sample Answer
VACUUM deletes data files that no current table version references and that were removed from the table longer ago than the retention threshold. The default threshold is 7 days, set by delta.deletedFileRetentionDuration, and Databricks refuses shorter values unless spark.databricks.delta.retentionDurationCheck.enabled is false. With 0 hours, files that a running job's older snapshot still needed were deleted under it, so its reads failed. Every version before the vacuum lost its data files, so time travel and RESTORE to those versions are gone too. Keep at least 7 days, longer than your longest job or stream lag. Use VACUUM ... DRY RUN first, and note that LITE mode uses the log while FULL (the default) scans the directory.
A daily customer extract is upserted with MERGE INTO customers USING updates ON t.id = s.id. It fails because multiple source rows matched the same target row. Fix it, and also delete customers that no longer appear in the extract.
MediumDelta Lake Transactions, Time Travel and MERGE
Sample Answer
MERGE requires each target row to match at most one source row; otherwise the update is ambiguous and Delta fails the statement. Extracts and CDC batches often hold several versions per key, so deduplicate the source first, keeping the latest change per id with ROW_NUMBER() or QUALIFY. MERGE only deduplicates new rows against the table, never within the source, so duplicates in the source get inserted twice when nothing matches yet. For deletes, WHEN NOT MATCHED BY SOURCE THEN DELETE removes target rows with no source match. Use it only when the source is a complete snapshot, and add a condition when it covers part of the table, as the docs recommend; otherwise every unmatched row is deleted and far more of the target is rewritten.
SQL
1MERGEINTO customers t
2USING(3SELECT id, name, email, updated_at FROM updates
4 QUALIFY ROW_NUMBER()OVER(PARTITIONBY id ORDERBY updated_at DESC)=15) s
6ON t.id = s.id
7WHENMATCHEDTHENUPDATESET*8WHENNOTMATCHEDTHENINSERT*9WHENNOTMATCHEDBY SOURCE THENDELETE;
A privacy request deletes one user with DELETE FROM events WHERE user_id = 42. A week later an auditor finds the user's rows in the Parquet files. What is going on, and how do you guarantee physical removal?
HardDelta Lake Transactions, Time Travel and MERGE
Sample Answer
With deletion vectors enabled, DELETE, UPDATE and MERGE do not rewrite whole Parquet files. They write a small file marking which rows in a data file are deleted, and readers skip those rows at query time. The data stays in the original file until it is rewritten. Also, older table versions still reference the original files, so time travel can read them. To purge: run REORG TABLE events APPLY (PURGE), which rewrites files that carry deletion vectors, then VACUUM once the retention period has passed after the purge so the old files leave storage. OPTIMIZE and auto compaction also rewrite files, but compaction alone doesn't guarantee every soft-deleted row is gone.
SQL
1DELETEFROM events WHERE user_id =42;-- soft delete via deletion vectors2REORG TABLE events APPLY(PURGE);-- rewrite files that carry them3-- after the retention window (7 days by default):4VACUUM events;-- remove the old files from storage
Upstream added a coupon_code column. The nightly append to a Delta table now fails with a schema mismatch error. What do you change, and what should you avoid?
MediumDelta Lake Transactions, Time Travel and MERGE
Sample Answer
Delta enforces the table schema on write, so a DataFrame with an extra column is rejected instead of silently creating a drifted table. When the new column is expected, enable schema evolution for that one write: .option("mergeSchema", "true") on the append, or MERGE WITH SCHEMA EVOLUTION (Databricks Runtime 15.4 LTS and above) for upserts. New columns are added at the end of the schema and older rows read as NULL. Avoid the session-wide spark.databricks.delta.schema.autoMerge.enabled setting in production; Databricks calls it legacy because it lets every write in the session change schemas, including typos. A renamed or retyped column is a different problem and needs an explicit ALTER TABLE plan.
Python
1(df.write.format("delta")2.mode("append")3.option("mergeSchema","true")# this write only4.saveAsTable("sales.orders"))
Query speed on a lakehouse is mostly decided by how many files a query has to open. Interviewers probe layout because the fixes (compaction, clustering, statistics) are cheap when chosen well and expensive when chosen by habit, such as partitioning a 50 GB table by user.
The idea to anchor on: Databricks skips files using per-file min and max statistics, so a layout is good when rows that are filtered together live in the same files. Liquid clustering is the current default for arranging that, and it replaces both partitioning and Z-ORDER for new tables.
Data Layout: Files, Clustering and Change Feeds
A streaming job has written to a Delta table every minute for six months, and simple queries on it are slow. Diagnose the likely cause and fix it.
EasyData Layout: Files, Clustering and Change Feeds
Sample Answer
Every micro-batch writes at least one file per task, so a minute-level stream produces hundreds of thousands of small files. Each file costs a cloud storage request and a Parquet footer read, and statistics on tiny files skip poorly. DESCRIBE DETAIL shows numFiles and sizeInBytes, which makes the average file size easy to check. Run OPTIMIZE to compact small files into larger ones, and keep it running: on Unity Catalog managed tables, predictive optimization schedules OPTIMIZE and VACUUM automatically and is on by default for accounts created since November 2024. Optimized writes and auto compaction reduce small files at write time. Compaction rewrites data, so it also needs VACUUM later.
A teammate proposes partitioning a 200 GB events table by user_id because most queries filter on one user. What do you say?
MediumData Layout: Files, Clustering and Change Feeds
Sample Answer
Partitioning by a high-cardinality column creates millions of directories with tiny files, which hurts every query, including the ones it was meant to speed up. Databricks guidance is blunt: don't partition tables under 1 TB, and when you do partition, each partition should hold at least 1 GB. For a 200 GB table filtered by user_id, use liquid clustering with CLUSTER BY (user_id). It co-locates each user's rows in a few files so min and max statistics can skip the rest, without a directory per value. If date filters are also common, cluster by both columns. Partitioning still has a narrow role: a low-cardinality column used for whole-partition deletes or overwrites on very large tables.
An events table is filtered by customer_id and by event_date. It currently uses OPTIMIZE ... ZORDER BY (customer_id). Would you move it to liquid clustering, and how?
HardData Layout: Files, Clustering and Change Feeds
Sample Answer
Yes. Z-ORDER rewrites data to interleave the chosen columns, but it is not incremental, is tied to the partition layout and has to be re-run as data arrives, which tends to rewrite large parts of the table. Liquid clustering is incremental: OPTIMIZE only rewrites what is not yet clustered, keys can change with ALTER TABLE ... CLUSTER BY without rewriting old data, and up to four keys are allowed. Clustering is not compatible with partitioning or ZORDER on the same table, so stop the ZORDER job. CLUSTER BY AUTO lets predictive optimization pick keys from the query history. After changing keys, run OPTIMIZE ... FULL once if old data must reflect the new layout.
SQL
1ALTERTABLE events CLUSTER BY(customer_id, event_date);2OPTIMIZE events FULL;-- recluster existing data once (DBR 16.4+)3OPTIMIZE events;-- routine runs are incremental45-- or let Databricks choose keys from query patterns6ALTERTABLE events CLUSTER BY AUTO;
A query filtering on device_model, the 40th column of a wide external table, reads every file even though the table is well clustered on that column. Why?
MediumData Layout: Files, Clustering and Change Feeds
Sample Answer
File skipping uses per-file statistics (minimum, maximum, null count, record count) collected at write time. On external tables, stats are collected only for the first 32 columns of the schema by default, so the 40th column has none and nothing can be skipped. Either name the columns explicitly with delta.dataSkippingStatsColumns, which overrides the count, or raise delta.dataSkippingNumIndexedCols. New writes get the new stats; recompute for existing files with ANALYZE TABLE ... COMPUTE DELTA STATISTICS. On Unity Catalog managed tables, predictive optimization can collect stats on frequently filtered columns beyond the first 32. Long strings are truncated in stats, so they skip poorly anyway.
SQL
1ALTERTABLE events SET TBLPROPERTIES (2'delta.dataSkippingStatsColumns'='event_date,customer_id,device_model'3);4ANALYZETABLE events COMPUTE DELTA STATISTICS;-- backfill stats for old files
A downstream team re-reads the whole 3 TB customers table every hour to find what changed. How would you give them only the changes?
MediumData Layout: Files, Clustering and Change Feeds
Sample Answer
Enable the change data feed on the table with delta.enableChangeDataFeed = true (newer runtimes with Unity Catalog can provide it automatically). Delta then records row-level changes per commit, and consumers read them with table_changes('customers', start_version) in SQL or readChangeFeed in batch or streaming. Each row carries _change_type (insert, update_preimage, update_postimage, delete), _commit_version and _commit_timestamp. Consumers store the last version they processed and start from the next one. The trap is treating the feed as permanent history: change files follow the table's retention and VACUUM removes them. If the team needs a full audit trail, write the changes to their own table.
SQL
1ALTERTABLE customers SET TBLPROPERTIES (delta.enableChangeDataFeed =true);23SELECT id, _change_type, _commit_version
4FROM table_changes('customers',1052)-- last processed version + 15WHERE _change_type !='update_preimage';
Most Databricks performance interviews are Spark interviews with a platform twist. You get a slow job, a Spark UI screenshot or a cost complaint, and you are expected to locate the shuffle, the skew or the UDF that explains it before suggesting a bigger cluster.
Keep one model in mind. Narrow transformations run within a partition; wide ones (joins, groupBy, distinct) shuffle data across the network and start a new stage. Nearly every tuning answer reduces the data shuffled, spreads it evenly, or keeps work inside the optimized engine instead of Python.
Spark Performance on Databricks
A job reads a table, filters, joins it to another table, aggregates by country and writes the result. The Spark UI shows three stages. Explain where the boundaries come from.
EasySpark Performance on Databricks
Sample Answer
Spark splits a job into stages at shuffle boundaries. Reading and filtering are narrow transformations, so they run inside the scan stage without moving data. A sort-merge join needs rows with the same key on the same executor, so both sides are shuffled by the join key: that creates an exchange and a new stage. The groupBy("country") aggregation shuffles again by country, so the final stage aggregates and writes. Spark also does a partial aggregation before that shuffle, so it moves one partial row per country per task rather than every row. In the plan, Exchange nodes mark each shuffle. A broadcast join would remove the join shuffle entirely, leaving two stages.
A 2 TB fact table joins a 40 MB dimension table, and the plan shows a sort-merge join that shuffles both sides. How do you fix it, and when is broadcasting a mistake?
MediumSpark Performance on Databricks
Sample Answer
Broadcast the small side. Each executor gets a full copy of the dimension and joins locally, so the 2 TB side is never shuffled. Spark broadcasts automatically below spark.sql.autoBroadcastJoinThreshold (10 MB in Apache Spark) based on size estimates, and on Databricks, adaptive query execution can switch to a broadcast at runtime when actual sizes fall under 30 MB. For 40 MB, add an explicit broadcast() hint. It becomes a mistake when the broadcast side is large: it is collected to the driver and copied to every executor, which can cause out-of-memory failures, and Spark rejects broadcasts over 8 GB. In a left outer join, only the right side can be broadcast.
Python
1from pyspark.sql.functions import broadcast
23orders = spark.table("sales.orders")# about 2 TB4stores = spark.table("ref.stores")# about 40 MB5joined = orders.join(broadcast(stores),"store_id")6joined.explain()# expect BroadcastHashJoin, no Exchange on the orders side
One task in a join stage runs for 40 minutes while the other 199 finish in under a minute. Diagnose it and fix it, including the case where adaptive execution does not help.
HardSpark Performance on Databricks
Sample Answer
That pattern is skew: one key, often a null or a default value such as user_id = 0, holds a large share of rows. Confirm in the stage's task metrics, where max shuffle read size dwarfs the median. Adaptive query execution handles skewed sort-merge and shuffled hash joins: a partition larger than 5 times the median and above 256 MB is split into smaller tasks. It does not split skewed aggregations, and AQE does not run for streaming queries at all. Fixes: filter or separately process null keys, broadcast the small side, or salt the hot key by adding a random suffix on the large side and replicating the small side once per salt value.
Python
1from pyspark.sql import functions as F
23facts = spark.table("events")# skewed on user_id4dims = spark.table("ref.users")5N =166big = facts.withColumn("salt",(F.rand()* N).cast("int"))7small = dims.crossJoin(spark.range(N).withColumnRenamed("id","salt"))8joined = big.join(small,["user_id","salt"]).drop("salt")
A job processes 5 GB with spark.sql.shuffle.partitions = 200, and another processes 5 TB with the same setting. What goes wrong in each, and what do you change?
MediumSpark Performance on Databricks
Sample Answer
200 shuffle partitions is the Apache Spark default and it suits neither job. At 5 GB, 200 partitions means tiny tasks where scheduling overhead dominates, and if the result is written straight out, 200 small files. At 5 TB, each partition holds about 25 GB, which spills to disk or fails with memory errors. Adaptive query execution coalesces small shuffle partitions toward spark.sql.adaptive.advisoryPartitionSizeInBytes (64 MB by default), which fixes the first case. For the second, raise the number so partitions land near 100 to 200 MB, or on Databricks set spark.sql.shuffle.partitions to auto for auto-optimized shuffle. To control output files, coalesce(n) merges partitions without a shuffle, while repartition(n) shuffles to get even sizes.
An analyst calls .cache() on every intermediate DataFrame in a notebook, and the cluster gets slower and starts spilling. Explain what caching does on Databricks and when to use it.
EasySpark Performance on Databricks
Sample Answer
df.cache() is lazy: it marks the DataFrame for caching at the default MEMORY_AND_DISK level, and the data is materialized only when an action runs. Caching everything fills executor memory with data that is used once, which evicts useful blocks and causes spills. Cache only a DataFrame that is reused by several actions and expensive to recompute, then call unpersist() when done. Databricks also has the disk cache, which automatically keeps local copies of Parquet and Delta files on worker SSDs after the first read and stays consistent when files change. For repeated reads of tables, Databricks recommends relying on it rather than Spark cache. Serverless compute supports cache() but not CACHE TABLE.
Photon was enabled on a job cluster but one ETL job got no faster, while the cost per hour went up. How do you find out why?
MediumSpark Performance on Databricks
Sample Answer
Photon is a native vectorized engine that executes supported Spark SQL and DataFrame operators, while Catalyst still plans the query. Operators it does not support fall back to the regular Spark engine. Python UDFs, RDD code and typed Dataset APIs are not supported, and for streaming only stateless queries benefit. Check where time went: on classic compute, the Spark UI SQL tab shows Photon operators separately from regular ones, and on SQL warehouses and serverless the query profile reports the share of task time spent in Photon. If a UDF sits in the middle of the plan, rewrite it with built-in functions. Queries finishing in a couple of seconds gain little. Photon compute consumes DBUs at a different rate, so measure cost per run.
A Python UDF that normalizes phone numbers dominates the runtime of a 2-billion-row job. What are your options, from best to worst?
HardSpark Performance on Databricks
Sample Answer
A row-at-a-time Python UDF serializes each row from the JVM to a Python worker and back, blocks Catalyst optimizations and forces Photon to fall back. Best is rewriting with built-in functions (regexp_replace, substring, when), which run in the engine and stay eligible for Photon. If the logic needs Python libraries, use a pandas UDF, which moves data in Arrow batches and runs vectorized pandas code, typically far faster than a plain UDF. A plain UDF with Arrow enabled (useArrow=True) helps the transfer cost but still calls Python once per row. Keep a plain UDF only for rare columns or small tables.
Python
1from pyspark.sql import functions as F
23clean = F.regexp_replace("phone",r"[^0-9]","")4users =(users.withColumn("digits", clean)5.withColumn("phone_e164",6 F.when(F.length("digits")==10, F.concat(F.lit("+1"), F.col("digits")))7.otherwise(F.concat(F.lit("+"), F.col("digits")))))
Unity Catalog is where Databricks keeps permissions, lineage and table ownership, and governance questions now show up in data engineer and platform interviews alike. Interviewers want to see that you can grant least-privilege access, protect sensitive columns and trace a bad number back to its source.
The concept everything hangs on is the three-level namespace, catalog.schema.object, with privileges that must be granted at each level. Most access bugs come from a missing USE CATALOG or USE SCHEMA rather than from the table grant itself.
Unity Catalog and Governance
An analyst was granted SELECT on finance.reporting.revenue but still gets a permission error. What is missing, and how would you grant access properly?
EasyUnity Catalog and Governance
Sample Answer
Unity Catalog objects live in a three-level namespace, catalog then schema then table, and reading a table requires privileges at every level: USE CATALOG on finance, USE SCHEMA on finance.reporting and SELECT on the table. Granting only SELECT leaves the analyst unable to reach the table. Grant to account groups, not individual users, so access follows team membership. Privileges are inherited downward, so SELECT granted on a schema covers current and future tables in it, which is convenient for a reporting schema but too broad for one holding raw PII. Check what a principal has with SHOW GRANTS.
Regional managers should see only their region's rows in sales.orders, and only the compliance group should see the full email column. Implement both without copying the table.
MediumUnity Catalog and Governance
Sample Answer
Unity Catalog supports row filters and column masks, both written as SQL functions attached to the table. A row filter returns a boolean per row, and the table shows only rows where it is true. A column mask returns the value a user should see in place of the real one. Inside each function, is_account_group_member() checks the caller's groups, so one table serves everyone with no per-region copies. For many tables with the same rule, attribute-based access control (ABAC) policies created with CREATE POLICY at the catalog or schema level scale better than per-table functions. The trap is building per-group views: they multiply quickly and are easy to bypass if someone still holds SELECT on the base table.
A revenue figure on an executive dashboard is wrong, and the gold table behind it is built from a dozen upstream tables. How do you use Unity Catalog to find the culprit?
MediumUnity Catalog and Governance
Sample Answer
Unity Catalog captures lineage automatically for queries run through Spark DataFrames or Databricks SQL, at both table and column level. Open the gold table in Catalog Explorer and follow the column's upstream graph, or query the lineage system tables (system.access.table_lineage and system.access.column_lineage), which keep a rolling one year. From the upstream tables, check DESCRIBE HISTORY for recent writes that line up with when the number went wrong. Know the gaps: RDD code is not captured, renaming a table or column breaks its lineage, and lineage records which tables fed which, not whether the logic was right.
Someone dropped a managed table in production by mistake. Another team dropped an external table. What happens to the data in each case, and what can you recover?
EasyUnity Catalog and Governance
Sample Answer
For a Unity Catalog managed table, Databricks owns the files. DROP TABLE removes it from the catalog, and the data files are purged after a recovery period, 7 days by default. Within that window, UNDROP TABLE restores it with its history. For an external table, DROP removes only the catalog entry; the files stay in your storage location, so you can re-create the table over the same path but must clean up files yourself if you wanted them gone. Databricks recommends managed tables as the default because they also get predictive optimization (automatic OPTIMIZE, VACUUM and statistics). Non-tabular files belong in volumes, addressed as /Volumes/catalog/schema/volume/path.
Design the Unity Catalog layout for three teams with dev, staging and prod environments. Production jobs must not be editable by individual engineers.
HardUnity Catalog and Governance
Sample Answer
Use catalogs for environment boundaries (dev, staging, prod, or per-domain catalogs such as prod_sales when teams need strong isolation) and schemas for domains or medallion layers inside them. Bind catalogs to workspaces so the prod catalog is only reachable from the prod workspace. Grant privileges to groups, never to individuals, and run production jobs as a service principal that owns the prod tables. Engineers get read access in prod and full rights in dev, and changes reach prod through CI that deploys job definitions as code. Keep raw PII in its own schema with tighter grants plus masks. Avoid one catalog per engineer or per project, which multiplies grants and makes lineage hard to read.
The last area covers getting data in, keeping it correct and paying a sensible amount to run it. Interviewers ask about Auto Loader, streaming checkpoints, pipeline expectations and compute choices because these decide whether a pipeline survives restarts, bad input and the monthly bill.
The idea that links them: incremental processing depends on state stored somewhere durable. Auto Loader and Structured Streaming keep it in a checkpoint, Lakeflow pipelines manage it for you, and almost every serious incident in this area starts with someone deleting, sharing or changing that state.
Ingestion, Pipelines and Compute
New JSON files land in an S3 bucket all day. Build ingestion into a bronze Delta table that handles new columns and runs as a scheduled job rather than a 24/7 stream.
MediumIngestion, Pipelines and Compute
Sample Answer
Use Auto Loader (cloudFiles) with a schema location and an availableNow trigger. Auto Loader records every discovered file in a RocksDB store inside the checkpoint, which gives exactly-once ingestion across restarts. With cloudFiles.schemaLocation set, it infers the schema from a sample (up to 50 GB or 1,000 files) and tracks changes. JSON columns are inferred as strings unless cloudFiles.inferColumnTypes is true. The default evolution mode, addNewColumns, stops the stream when a new column appears, updates the schema, and continues after a restart, so run it as a job with retries. Values that don't fit the schema land in _rescued_data instead of being dropped. For discovery, file events beat directory listing on large buckets.
A streaming aggregation was changed to group by an extra column and restarted from the same checkpoint, and it failed. A colleague suggests deleting the checkpoint. What are the consequences, and what is the safe path?
HardIngestion, Pipelines and Compute
Sample Answer
A checkpoint stores source offsets, a commit log and, for stateful queries, the state store, which is laid out by the grouping keys and the shuffle partition count. Changing the grouping keys or aggregates of a stateful operation between restarts is not allowed, and the shuffle partition count is fixed by the checkpoint. Deleting the checkpoint makes the query start over: depending on the starting options it either reprocesses everything, producing duplicates in an append sink, or skips data. Safe path: start the new query version with a new checkpoint and a defined starting point, backfill the gap explicitly, and make the sink idempotent. In foreachBatch, Delta's txnAppId and txnVersion options let a retried batch be skipped instead of written twice.
In a Lakeflow pipeline, the silver orders table must drop rows with a null order_id, keep but count rows with negative amounts, and stop the update entirely if the currency code is unknown. Implement it.
MediumIngestion, Pipelines and Compute
Sample Answer
Lakeflow pipelines (the product formerly called Delta Live Tables) express data quality as expectations attached to a dataset. There are three actions: @dp.expect keeps violating rows and records metrics (the default behavior), @dp.expect_or_drop removes them, and @dp.expect_or_fail fails the update. The expect_all variants take a dictionary of rules. Pass and fail counts appear in the pipeline event log and UI, which is the monitoring hook. A common pattern for dropped rows is a quarantine table defined with the inverse condition, so bad records are kept for review instead of disappearing. Python pipeline code now imports from pyspark import pipelines as dp; the old import dlt still works.
Python
1from pyspark import pipelines as dp
23@dp.table(name="silver_orders")4@dp.expect_or_drop("has_order_id","order_id IS NOT NULL")5@dp.expect("non_negative_amount","amount >= 0")6@dp.expect_or_fail("known_currency","currency IN ('USD', 'EUR', 'GBP')")7defsilver_orders():8return spark.readStream.table("bronze_orders")
Explain the medallion architecture to a new team member, and say what belongs in each layer for an orders pipeline.
EasyIngestion, Pipelines and Compute
Sample Answer
Medallion is a naming convention for refinement stages in a lakehouse. Bronze holds raw data as it arrived, append-only, with ingestion metadata such as file name and load time, so you can always replay. Silver holds cleaned and conformed records: deduplicated orders, typed columns, joined reference data and quality rules applied. Gold holds business-level tables built for a use case, such as daily revenue by region or a customer lifetime value table. The trap is treating it as a rule that requires exactly three copies of everything. Small sources may go straight to silver, and gold should exist only when someone consumes it.
A nightly ETL job runs on an all-purpose cluster that stays up all day, and the bill is high. Which compute would you move it to, and what about the BI dashboards on the same cluster?
EasyIngestion, Pipelines and Compute
Sample Answer
All-purpose compute is meant for interactive work and bills at a higher DBU rate than jobs compute. Scheduled ETL should run on jobs compute, created for the run and terminated when it ends, or on serverless compute for jobs, which needs no cluster configuration. Serverless jobs default to performance-optimized mode; standard mode tolerates a 4 to 6 minute start and can cut cost by up to 70%, which suits a nightly batch. BI dashboards belong on a SQL warehouse. Serverless warehouses start in seconds and include Photon, predictive I/O and intelligent workload management, while Pro and Classic start in about four minutes. Set auto-stop so idle compute shuts down.
Design a daily Lakeflow Job: ingest, then three independent transforms, then a quality check, then publish. One transform fails on Tuesday. How do you rerun without redoing everything?
MediumIngestion, Pipelines and Compute
Sample Answer
Model it as a multi-task job with dependencies: the three transforms depend on ingest and run in parallel, the quality check depends on all three, and publish depends on the check. An if/else condition task can route to publish or to an alert based on the check result. When one transform fails, a repair run re-executes only the failed task and the tasks downstream of it, using the current task settings, and the successful tasks are not repeated. Because a repaired task starts from the beginning, its writes must be idempotent (overwrite a partition or MERGE), or the repair duplicates data. Instead of a fixed schedule, a file arrival or table update trigger can start the job when its input actually lands.
Time travel, VACUUM, MERGE conflicts and deletion vectors all follow from how the _delta_log lists files per version. Create a small table, run a few writes and read the JSON commit files yourself. Table formats: Iceberg, Delta and Hudi explains the log, and Small files, compaction and schema evolution covers what OPTIMIZE and schema changes do to it.
Read Spark plans until shuffles jump out
Run explain() on every join and aggregation you practise and point at each Exchange. Then predict which join strategy Spark will pick before checking. Read an execution plan and Shuffles build this habit, and Broadcast joins shows when broadcasting backfires.
Have a skew playbook ready
Skew questions come up in nearly every senior loop. Be able to name the symptom in the Spark UI, what adaptive execution fixes on its own, and when you need salting or separate handling of null keys. Practise with Data skew and Adaptive execution.
Prefer clustering talk over partitioning habits
Interviewers notice when a candidate partitions every table by date out of habit. Know the 1 TB and 1 GB partition guidance, how file statistics drive skipping and why liquid clustering replaced Z-ORDER for new tables. Partitioning and pruning and Clustering and sort keys cover the trade-offs across engines.
Practise streaming recovery out loud
Explain what a checkpoint stores, which query changes it tolerates and how to backfill after a breaking change. Then cover cost: jobs compute versus serverless versus SQL warehouses. Work through Checkpoints and recovery and Watermarks, then Cut a 500K per month warehouse bill for the cost side.
How Ready Are You for Databricks Interviews?
1 / 6
Delta Lake
A table was overwritten by mistake yesterday. What is the best first move?
Frequently Asked Questions
Is Databricks asked in data engineering interviews?
Yes, at companies that run on it, and lakehouse concepts appear even where it isn't used. Expect questions on Delta Lake internals, Spark performance, streaming and governance rather than UI clicks. Interviewers care whether you can explain why a job is slow or a table is wrong. Knowing Databricks-specific features such as Auto Loader, Unity Catalog and Lakeflow pipelines is a strong plus when the job description names the platform.
Do I need a Databricks certification to get hired?
No. Certifications can help a resume pass a keyword screen, especially for consulting roles, but interviews test reasoning that a multiple-choice exam does not. Hiring managers respond better to a project you can explain in depth: a pipeline you built, a job you made faster and how you measured it. If you have time for only one, build the project and talk through its trade-offs.
How long does it take to prepare for a Databricks interview?
With solid Spark and SQL experience, two to three weeks of focused work is usually enough: Delta Lake internals, layout, Unity Catalog and streaming recovery. Coming from pandas or a traditional warehouse, plan on five to eight weeks so you can learn Spark execution first. Databricks Free Edition, the successor to Community Edition, is enough to practise MERGE, time travel and Auto Loader on real tables.
What level of Spark do I need for a Databricks role?
You should read a physical plan, explain shuffles and stages, choose join strategies, diagnose skew from the Spark UI and write clean DataFrame code without UDFs where built-ins exist. Senior roles add streaming state, checkpoint recovery and cost trade-offs. You rarely need RDD APIs or Scala, although some teams still use Scala for older jobs. PySpark and SQL cover most interview prompts.
Are Databricks interview questions different from Snowflake ones?
The warehouse fundamentals overlap: columnar storage, pruning, clustering, cost control and access control. Databricks interviews lean harder on Spark execution, open table formats, streaming and notebooks or code-first pipelines. Snowflake interviews lean toward SQL, virtual warehouse sizing and features such as streams and tasks. If you prepare the shared concepts once, switching between the two mostly means learning each platform's vocabulary.
Which Databricks product names should I use in 2026?
Use the current names but recognise the old ones. Delta Live Tables became Lakeflow pipelines, built on Apache Spark Declarative Pipelines, and the Python API is now from pyspark import pipelines as dp. Workflows became Lakeflow Jobs. Unity Catalog, Auto Loader, Photon and Delta Lake keep their names. Interviewers will not penalise an old name, but mixing in current ones shows you have used the platform recently.
Written by
Dan Lee
Data & AI Lead
Dan is a seasoned data scientist and ML coach with 10+ years of experience at Google, PayPal, and startups. He has helped candidates land top-paying roles and offers personalized guidance to accelerate your data career.