cert-bank

Guides

이 글을 한국어로 읽기

DBX-DEA Study Guide: Ask "What Happens on the Second Run?" and Half the Exam Answers Itself

Many Databricks Data Engineer Associate questions describe something that already works once (a file load, a cleaning step, a job, a deployment, a grant) and really ask what happens next time. Does the load pick up the same files again? Does a repaired task append its rows twice? Does a second deployment overwrite a manual fix? This guide uses that one question, "what happens on the second run?", as a thread through the exam, because it is the thinking the correct answers share.

Study guide · Databricks Certified Data Engineer Associate (DBX-DEA)

DBX-DEA Study Guide: Ask "What Happens on the Second Run?" and Half the Exam Answers Itself

The exam in numbers

45 scored questions (plus up to 10 unscored ones) in 90 minutes. Databricks does not publish the pass mark, so check the official exam page for current details and aim well above the line. Seven sections, with these weights:

SectionWeight
Data Transformation and Modeling22%
Data Ingestion and Loading21%
Working with Lakeflow Jobs16%
Governance and Security15%
Implementing CI/CD10%
Troubleshooting, Monitoring, and Optimization10%
Databricks Intelligence Platform6%

Ingestion and transformation together are 43% of the exam. Among the 225 DBX-DEA questions we wrote for this question bank, more than half (56%) show code or SQL in the stem or the options, and Unity Catalog is the service named most often (45 questions).

Second run of a file load

Three tools load files, and each one remembers what it has already done in a different place.

  • COPY INTO is a SQL command that records which files it loaded into the target. Run it again and it skips them. That makes it safe to rerun after a timeout: only the missing files load. The catch: tracking is by file name, not by content. A file rewritten in place under the same name is skipped, and reloading it needs the force copy option (which applies to every selected file, so narrow the selection and remove the old rows first).
  • Auto Loader (cloudFiles) keeps the list of discovered files in its checkpoint. Delete the checkpoint or point the stream at a new one, and the next run starts fresh: every historic file is loaded again and the target gets duplicates. Each stream needs its own checkpoint, even when two streams write to one table.
  • A plain batch read (spark.read) remembers nothing. On a growing folder it reads everything every time.

The trigger decides whether a stream keeps running. trigger(availableNow=True) processes everything new and stops, which is the answer when a stem says "hourly, only new files, no cluster running between loads". A processingTime trigger keeps the query alive for minute-level freshness.

Beginner tip: in a free workspace, put two small CSV files in a volume, run a COPY INTO into an empty Delta table, then run the exact same statement again and look at the row count. Add a third file and run it once more. Then do the same with an Auto Loader stream using availableNow, and finally delete its checkpoint folder and run it again. Ten minutes of this turns a whole family of "will this load duplicates?" questions into something you have watched happen.

Second run of a transformation

A cleaning step is correct only if it gives the same result every time it runs. Three traps recur:

  • Deduplication that is not deterministic. dropDuplicates(["customer_id"]) keeps whichever row Spark meets first, and after a shuffle that order can change between runs. To keep the latest row per key, rank with a window: QUALIFY row_number() OVER (PARTITION BY customer_id ORDER BY updated_at DESC) = 1.
  • Appending instead of merging. INSERT INTO run twice doubles the rows. MERGE INTO ... ON key, INSERT OVERWRITE of a partition or replaceWhere reach the same end state however often they run.
  • Strict casts on dirty data. CAST fails the whole query on one bad value; try_cast returns NULL. Which one is right depends on whether bad data should stop the pipeline. When rows must not silently disappear, the expected answer is a quarantine table with a reason column, not a filter.

A streaming table processes each input row once, so later changes in a joined dimension never reach rows already written. A materialized view is correct as of its last refresh, because it recomputes what changed. If a report must show today's customer segment for last year's orders, it wants a materialized view. Inside Lakeflow Spark Declarative Pipelines, expectations decide what bad rows do: warn (default), drop, or fail the update.

Going deeper: why a Delta write is all or nothing

A Delta table is a folder of ordinary Parquet files plus a transaction log in _delta_log. Readers never list the data files; they rebuild the table from the log, which is an ordered list of commits saying "add this file" or "remove that file". A writer first writes its new Parquet files, which no reader can see yet, and then tries to add one commit to the log. Only when that commit succeeds do the files become part of the table. If the job dies before the commit, the files are orphans that a later VACUUM cleans up, and readers never saw a half-written result. That is why a failed write leaves the previous version intact, and why DESCRIBE HISTORY, time travel and RESTORE work at all: each version is a point in the log.

Second run of a job

Lakeflow Jobs (the bank notes its older name, Workflows) has two ways of running a failed task again, and neither makes your code safe to repeat.

FeatureWhat it re-runsExam clue
Retry policyThe failed task, automatically, during the run. Default for most setups: no retry. A timeout applies to each attempt separately."Transient network error", "source briefly unavailable".
Repair runThe unsuccessful tasks of a finished run plus the tasks that depend on them, with the current settings."Without re-running the tasks that succeeded."
Run ifNot a re-run: decides whether a task runs given its dependencies' outcomes (All succeeded, All done, At least one failed ...)."A cleanup task must run even if a step fails."

A repaired task starts from its first line. If it appended a million rows before failing, the repair appends them again. When a question pairs "repair" with duplicates, the fix is an idempotent write, not a different job setting.

Second deployment

Declarative Automation Bundles (the current name; formerly Databricks Asset Bundles) describe jobs and pipelines in YAML, deployed per target with databricks bundle deploy -t prod. Deployment is declarative: the bundle keeps a state, so a resource removed from the YAML is removed from the workspace on the next deploy, and manual edits in the workspace can be overwritten. Variables resolve at deploy time, and a command-line --var beats environment variables, target values and defaults. For production, run_as a service principal, so jobs do not break when the engineer who deployed them leaves.

Second person asking for access

Unity Catalog access is additive and inherited downwards. Reading a table needs USE CATALOG, USE SCHEMA and SELECT. A grant on a catalog covers current and future tables inside it. The classic trap: revoking SELECT on one table changes nothing when the access came from a catalog-level grant. Revoke where the grant was made, then grant again at a narrower level. The SQL DENY statement applies only to the legacy hive_metastore, not to Unity Catalog. For hiding values, use a column mask; for hiding rows, a row filter; for rules across many tables, ABAC policies on governed tags.

Going deeper: how to read skew in the Spark UI

A stage is split into one task per partition, and the stage finishes only when its slowest task finishes. The stage page shows task durations as minimum, 25th percentile, median, 75th percentile and maximum. Databricks gives a rule of thumb: if the maximum is about 50% or more above the 75th percentile, the stage probably has skew, meaning a few key values hold most of the rows and a few tasks do most of the work. Adaptive query execution, on by default, can split skewed join partitions. Spill statistics appear on the stage page only when the stage spilled, so their absence means no spill. Troubleshooting questions in this section usually give you one of these numbers and ask what it means.

What happens on the second run: the thread through DBX-DEA
1
File load
COPY INTO skips files it already loaded; Auto Loader remembers them in its checkpoint
2
Transformation
INSERT INTO run twice doubles the rows; MERGE INTO or replaceWhere reach the same end state
3
Job
A repair run restarts the failed task from its first line, so the write must be idempotent
4
Deployment
A bundle keeps state: removed resources go, and manual workspace edits can be overwritten
5
Access
Unity Catalog grants are inherited downwards: revoke where the grant was made
Each card is one section of this guide, with what a second run does there and the answer the exam expects.

A study order

  1. Ingestion: COPY INTO, Auto Loader (checkpoints, schema evolution, the _rescued_data column), triggers, Lakeflow Connect for databases and SaaS.
  2. Transformation: casting, NULL handling, joins, dedup, streaming tables versus materialized views, expectations.
  3. Lakeflow Jobs: retries, repair, Run if, triggers.
  4. Governance: the privilege hierarchy, managed versus external tables, masks and filters.
  5. Bundles, the Spark UI, then timed mixed sets.

Practise

Practise in the question bank: every question explains each option, right and wrong. Start with the domain you just studied, then take a timed mock exam.

Section weights as listed in the Databricks exam guide for the Data Engineer Associate. Databricks renames products often (Workflows became Lakeflow Jobs, Delta Live Tables became Lakeflow Spark Declarative Pipelines); check the current exam guide and documentation.