Databricks Interview Questions

Dan Lee's profile image
Dan LeeData & AI Lead
Last update
Databricks Interview Questions

Databricks interviews for data engineers rarely stop at Spark syntax. They ask what happens inside a Delta table when two jobs write at once, why one task in a stage runs for forty minutes, how a streaming job recovers after a crash and who is allowed to see which column. Databricks is a platform, and the questions follow its moving parts.

Candidates tend to slip in one of two ways. Some know open-source Spark well but have never thought about the Delta transaction log, VACUUM retention or Unity Catalog privileges. Others learned Databricks a few years ago and still describe Z-ORDER as the default layout and all-purpose clusters as the place for nightly jobs. Product names have moved too: Delta Live Tables is now Lakeflow pipelines, built on Apache Spark Declarative Pipelines, and Workflows is now Lakeflow Jobs.

The 30 questions below are grouped into five areas, each with a short note on what interviewers look for. Every solution states the answer, the reason and the trap, with SQL or PySpark where code makes the point clearer. Defaults and names were checked against the Databricks, Delta Lake and Apache Spark documentation in October 2026. A prep plan, a quick quiz and answers to common questions follow the questions.

Intermediate30 questions

Databricks Interview Questions

Practice questions on Delta Lake, data layout, Spark performance, Unity Catalog, Auto Loader, Lakeflow pipelines and compute choices, the areas Databricks interviews test most.

Data EngineerAnalytics EngineerMachine Learning EngineerData ArchitectData ScientistPlatform Engineer

Delta Lake Transactions, Time Travel and MERGE

Almost every table on Databricks is a Delta table, so interviewers expect you to know what happens underneath a write. They ask about the transaction log, concurrent writers, recovering from a bad job and why deleted rows are not gone yet, because these are the incidents data engineers actually get paged for.

The one idea to carry through this section: a Delta table is a set of Parquet files plus an ordered log that says which files make up each version. ACID commits, time travel, VACUUM, MERGE and deletion vectors are all consequences of that log.

Delta Lake Transactions, Time Travel and MERGE

A job appends 500 files to a Delta table while a dashboard queries it. Explain, in terms of files on storage, why the dashboard never sees half of the new data.

EasyDelta Lake Transactions, Time Travel and MERGE

Sample Answer

A Delta table is a directory of Parquet files plus a _delta_log folder. Each commit writes one numbered JSON file (00000000000000000042.json) listing the data files added and removed, and periodic Parquet checkpoints summarize the log so readers don't replay thousands of JSON files. The writer first writes all 500 data files, then publishes them in a single atomic commit of the next version file. A reader picks the latest version it can see and reads exactly the files that version lists. Until the commit lands, the new files exist but belong to no version. The trap: deleting or copying files directly in cloud storage bypasses the log and can corrupt the table.

Learn this in the Data Warehousing and Lakehouses course

Data Layout: Files, Clustering and Change Feeds

Query speed on a lakehouse is mostly decided by how many files a query has to open. Interviewers probe layout because the fixes (compaction, clustering, statistics) are cheap when chosen well and expensive when chosen by habit, such as partitioning a 50 GB table by user.

The idea to anchor on: Databricks skips files using per-file min and max statistics, so a layout is good when rows that are filtered together live in the same files. Liquid clustering is the current default for arranging that, and it replaces both partitioning and Z-ORDER for new tables.

Data Layout: Files, Clustering and Change Feeds

A streaming job has written to a Delta table every minute for six months, and simple queries on it are slow. Diagnose the likely cause and fix it.

EasyData Layout: Files, Clustering and Change Feeds

Sample Answer

Every micro-batch writes at least one file per task, so a minute-level stream produces hundreds of thousands of small files. Each file costs a cloud storage request and a Parquet footer read, and statistics on tiny files skip poorly. DESCRIBE DETAIL shows numFiles and sizeInBytes, which makes the average file size easy to check. Run OPTIMIZE to compact small files into larger ones, and keep it running: on Unity Catalog managed tables, predictive optimization schedules OPTIMIZE and VACUUM automatically and is on by default for accounts created since November 2024. Optimized writes and auto compaction reduce small files at write time. Compaction rewrites data, so it also needs VACUUM later.

Learn this in the Data Warehousing and Lakehouses course

Spark Performance on Databricks

Most Databricks performance interviews are Spark interviews with a platform twist. You get a slow job, a Spark UI screenshot or a cost complaint, and you are expected to locate the shuffle, the skew or the UDF that explains it before suggesting a bigger cluster.

Keep one model in mind. Narrow transformations run within a partition; wide ones (joins, groupBy, distinct) shuffle data across the network and start a new stage. Nearly every tuning answer reduces the data shuffled, spreads it evenly, or keeps work inside the optimized engine instead of Python.

Spark Performance on Databricks

A job reads a table, filters, joins it to another table, aggregates by country and writes the result. The Spark UI shows three stages. Explain where the boundaries come from.

EasySpark Performance on Databricks

Sample Answer

Spark splits a job into stages at shuffle boundaries. Reading and filtering are narrow transformations, so they run inside the scan stage without moving data. A sort-merge join needs rows with the same key on the same executor, so both sides are shuffled by the join key: that creates an exchange and a new stage. The groupBy("country") aggregation shuffles again by country, so the final stage aggregates and writes. Spark also does a partial aggregation before that shuffle, so it moves one partial row per country per task rather than every row. In the plan, Exchange nodes mark each shuffle. A broadcast join would remove the join shuffle entirely, leaving two stages.

Learn this in the Data Warehousing and Lakehouses course

Unity Catalog and Governance

Unity Catalog is where Databricks keeps permissions, lineage and table ownership, and governance questions now show up in data engineer and platform interviews alike. Interviewers want to see that you can grant least-privilege access, protect sensitive columns and trace a bad number back to its source.

The concept everything hangs on is the three-level namespace, catalog.schema.object, with privileges that must be granted at each level. Most access bugs come from a missing USE CATALOG or USE SCHEMA rather than from the table grant itself.

Unity Catalog and Governance

An analyst was granted SELECT on finance.reporting.revenue but still gets a permission error. What is missing, and how would you grant access properly?

EasyUnity Catalog and Governance

Sample Answer

Unity Catalog objects live in a three-level namespace, catalog then schema then table, and reading a table requires privileges at every level: USE CATALOG on finance, USE SCHEMA on finance.reporting and SELECT on the table. Granting only SELECT leaves the analyst unable to reach the table. Grant to account groups, not individual users, so access follows team membership. Privileges are inherited downward, so SELECT granted on a schema covers current and future tables in it, which is convenient for a reporting schema but too broad for one holding raw PII. Check what a principal has with SHOW GRANTS.

SQL
1GRANT USE CATALOG ON CATALOG finance TO `fin-analysts`;
2GRANT USE SCHEMA  ON SCHEMA finance.reporting TO `fin-analysts`;
3GRANT SELECT      ON TABLE finance.reporting.revenue TO `fin-analysts`;
4
5SHOW GRANTS `fin-analysts` ON SCHEMA finance.reporting;
Learn this in the Data Warehousing and Lakehouses course

Ingestion, Pipelines and Compute

The last area covers getting data in, keeping it correct and paying a sensible amount to run it. Interviewers ask about Auto Loader, streaming checkpoints, pipeline expectations and compute choices because these decide whether a pipeline survives restarts, bad input and the monthly bill.

The idea that links them: incremental processing depends on state stored somewhere durable. Auto Loader and Structured Streaming keep it in a checkpoint, Lakeflow pipelines manage it for you, and almost every serious incident in this area starts with someone deleting, sharing or changing that state.

Ingestion, Pipelines and Compute

New JSON files land in an S3 bucket all day. Build ingestion into a bronze Delta table that handles new columns and runs as a scheduled job rather than a 24/7 stream.

MediumIngestion, Pipelines and Compute

Sample Answer

Use Auto Loader (cloudFiles) with a schema location and an availableNow trigger. Auto Loader records every discovered file in a RocksDB store inside the checkpoint, which gives exactly-once ingestion across restarts. With cloudFiles.schemaLocation set, it infers the schema from a sample (up to 50 GB or 1,000 files) and tracks changes. JSON columns are inferred as strings unless cloudFiles.inferColumnTypes is true. The default evolution mode, addNewColumns, stops the stream when a new column appears, updates the schema, and continues after a restart, so run it as a job with retries. Values that don't fit the schema land in _rescued_data instead of being dropped. For discovery, file events beat directory listing on large buckets.

Python
1(spark.readStream.format("cloudFiles")
2   .option("cloudFiles.format", "json")
3   .option("cloudFiles.schemaLocation", "/Volumes/main/raw/meta/events_schema")
4   .option("cloudFiles.inferColumnTypes", "true")
5   .load("s3://acme-landing/events/")
6 .writeStream
7   .option("checkpointLocation", "/Volumes/main/raw/meta/events_checkpoint")
8   .trigger(availableNow=True)
9   .toTable("bronze.events"))
Learn this in the Data Warehousing and Lakehouses course

How to Prepare for Databricks Interviews

Learn the transaction log before the features

Time travel, VACUUM, MERGE conflicts and deletion vectors all follow from how the _delta_log lists files per version. Create a small table, run a few writes and read the JSON commit files yourself. Table formats: Iceberg, Delta and Hudi explains the log, and Small files, compaction and schema evolution covers what OPTIMIZE and schema changes do to it.

Read Spark plans until shuffles jump out

Run explain() on every join and aggregation you practise and point at each Exchange. Then predict which join strategy Spark will pick before checking. Read an execution plan and Shuffles build this habit, and Broadcast joins shows when broadcasting backfires.

Have a skew playbook ready

Skew questions come up in nearly every senior loop. Be able to name the symptom in the Spark UI, what adaptive execution fixes on its own, and when you need salting or separate handling of null keys. Practise with Data skew and Adaptive execution.

Prefer clustering talk over partitioning habits

Interviewers notice when a candidate partitions every table by date out of habit. Know the 1 TB and 1 GB partition guidance, how file statistics drive skipping and why liquid clustering replaced Z-ORDER for new tables. Partitioning and pruning and Clustering and sort keys cover the trade-offs across engines.

Practise streaming recovery out loud

Explain what a checkpoint stores, which query changes it tolerates and how to backfill after a breaking change. Then cover cost: jobs compute versus serverless versus SQL warehouses. Work through Checkpoints and recovery and Watermarks, then Cut a 500K per month warehouse bill for the cost side.

How Ready Are You for Databricks Interviews?

1 / 6
Delta Lake

A table was overwritten by mistake yesterday. What is the best first move?

Frequently Asked Questions

Is Databricks asked in data engineering interviews?

Yes, at companies that run on it, and lakehouse concepts appear even where it isn't used. Expect questions on Delta Lake internals, Spark performance, streaming and governance rather than UI clicks. Interviewers care whether you can explain why a job is slow or a table is wrong. Knowing Databricks-specific features such as Auto Loader, Unity Catalog and Lakeflow pipelines is a strong plus when the job description names the platform.

Do I need a Databricks certification to get hired?

No. Certifications can help a resume pass a keyword screen, especially for consulting roles, but interviews test reasoning that a multiple-choice exam does not. Hiring managers respond better to a project you can explain in depth: a pipeline you built, a job you made faster and how you measured it. If you have time for only one, build the project and talk through its trade-offs.

How long does it take to prepare for a Databricks interview?

With solid Spark and SQL experience, two to three weeks of focused work is usually enough: Delta Lake internals, layout, Unity Catalog and streaming recovery. Coming from pandas or a traditional warehouse, plan on five to eight weeks so you can learn Spark execution first. Databricks Free Edition, the successor to Community Edition, is enough to practise MERGE, time travel and Auto Loader on real tables.

What level of Spark do I need for a Databricks role?

You should read a physical plan, explain shuffles and stages, choose join strategies, diagnose skew from the Spark UI and write clean DataFrame code without UDFs where built-ins exist. Senior roles add streaming state, checkpoint recovery and cost trade-offs. You rarely need RDD APIs or Scala, although some teams still use Scala for older jobs. PySpark and SQL cover most interview prompts.

Are Databricks interview questions different from Snowflake ones?

The warehouse fundamentals overlap: columnar storage, pruning, clustering, cost control and access control. Databricks interviews lean harder on Spark execution, open table formats, streaming and notebooks or code-first pipelines. Snowflake interviews lean toward SQL, virtual warehouse sizing and features such as streams and tasks. If you prepare the shared concepts once, switching between the two mostly means learning each platform's vocabulary.

Which Databricks product names should I use in 2026?

Use the current names but recognise the old ones. Delta Live Tables became Lakeflow pipelines, built on Apache Spark Declarative Pipelines, and the Python API is now from pyspark import pipelines as dp. Workflows became Lakeflow Jobs. Unity Catalog, Auto Loader, Photon and Delta Lake keep their names. Interviewers will not penalise an old name, but mixing in current ones shows you have used the platform recently.

Dan Lee's profile image

Written by

Dan Lee

Data & AI Lead

Dan is a seasoned data scientist and ML coach with 10+ years of experience at Google, PayPal, and startups. He has helped candidates land top-paying roles and offers personalized guidance to accelerate your data career.

Connect on LinkedIn