cert-bank

Guides

이 글을 한국어로 읽기

DBX-ASD Study Guide: Ask Five Questions of Every Code Snippet Before You Read the Choices

The Databricks Certified Associate Developer for Apache Spark exam (DBX-ASD) is a code-reading exam. In the 225 questions we wrote for this question bank, 177 (79%) put a code fragment, an API call or a configuration setting right in the question. You win by predicting what the code does before you look at the options, because the wrong options are written to match what careless readers expect. This guide gives you five questions to ask of every snippet, in the order that catches the most traps.

Study guide · Databricks Certified Associate Developer for Apache Spark (DBX-ASD)

DBX-ASD Study Guide: Ask Five Questions of Every Code Snippet Before You Read the Choices

The exam

45 questions in 90 minutes, so about two minutes each. Databricks does not publish the pass mark. There are seven domains: Apache Spark Architecture and Components (20%), Using Spark SQL (20%), Developing Apache Spark DataFrame/DataSet API Applications (30%), Troubleshooting and Tuning (10%), Structured Streaming (10%), Using Spark Connect to Deploy Applications (5%) and Using Pandas API on Spark (5%). The first three domains carry 70% of the exam.

Our practice sets follow Spark 4 behaviour, for example ANSI mode on by default. Check which Spark version the current exam guide names, because defaults have changed between major versions.

Question 1: does anything actually run?

Transformations (select, filter, withColumn, join, groupBy().agg(), orderBy) only add steps to a plan. Actions (count, collect, show, take, writes) start a job. Each action runs its own job (sometimes several), so two actions on the same DataFrame read the source twice unless it is cached. cache() is itself lazy: the first action fills the cache, and only later actions benefit. printSchema() and explain() read the plan, not the data, so they are not actions.

A consequence that the exam likes: a bad column name fails at analysis time in classic Spark (Spark Connect defers analysis until execution or schema access), but a data-dependent error (an invalid cast under ANSI mode, for example) appears only when an action runs, often many lines after the cause.

Beginner tip: install PySpark on your laptop, start it with local[*] and keep the Spark UI open at port 4040. Write five transformations and one count(), then add a second count(), then put cache() before them. Watch the Jobs tab after each run. Ten minutes of this settles most "how many jobs" and "when is the cache filled" questions.

Question 2: where does it run?

The driver plans and schedules. Executors run tasks and hold cached data. The cluster manager (standalone, YARN or Kubernetes) hands out executors. From that split:

  • Top-level Python runs on the driver. A lambda or UDF passed to a DataFrame operation runs on executors, so its print output lands in executor logs, not in your console.
  • collect() and toPandas() pull everything into the one driver. Above spark.driver.maxResultSize (1g by default) the job aborts. Aggregate on the cluster first, or limit the result.
  • A lost executor costs a recomputation of its partitions from lineage. Losing the driver ends the application.
  • Deploy mode decides where the driver lives. The default is client, and standalone mode has no cluster mode for Python applications.
  • With Spark Connect, your code is a thin client sending plans over gRPC. There is no SparkContext, no RDD API and no access to private JVM fields such as df._jdf.

Question 3: how many pieces, and does data move?

One task processes one partition on one core, so partitions set parallelism. File scans make partitions of up to 128 MB (spark.sql.files.maxPartitionBytes). Wide operations (groupBy, most joins, distinct, orderBy) shuffle into spark.sql.shuffle.partitions partitions, 200 by default, which AQE can then coalesce at run time. A shuffle also ends a stage, so groupBy('k').count().collect() runs as two stages (with AQE on, the Spark UI can list them under separate jobs). In explain() output, every Exchange node is a shuffle.

CallWhat it really does
repartition(n)Always a full shuffle; can raise or lower the count; balances sizes
coalesce(n)Merges partitions without a shuffle; can only lower the count. Asking for more changes nothing and raises no error. coalesce(1) can pull upstream work onto one task
broadcast(small) in a joinShips the small side to every executor, so the large side is not shuffled

Adaptive Query Execution is on by default (since Spark 3.2). At shuffle boundaries it merges small post-shuffle partitions, switches a sort-merge join to a broadcast join when one side turns out to be small, and splits skewed join partitions. A query with no shuffle gains nothing from it.

Going deeper: when AQE calls a partition "skewed"

A shuffle stage that has finished knows the real size of each of its output partitions, which is what lets AQE re-plan the stages that have not started yet. For a sort-merge join, it treats a partition as skewed only if two conditions both hold: it is larger than spark.sql.adaptive.skewJoin.skewedPartitionFactor (5) times the median partition size, and larger than skewedPartitionThresholdInBytes (256 MB). With a median of 100 MB, a 400 MB partition passes the byte threshold but not the factor (500 MB), so it is left alone. A 620 MB partition passes both and is split into even tasks. Skew in a plain groupBy is a different case: one key still hashes to one partition, so the remedies there are salting the key or pre-aggregating, not more partitions.

Question 4: what exactly comes back?

The DataFrame domain (30%) is mostly about exact results. The details that decide answers:

  • Save modes: the default is errorifexists. The alternatives are append, overwrite and ignore.
  • Views: a temporary view lives in one session. A global temporary view lives in global_temp and is shared by the sessions of one application.
  • ANSI mode (default in Spark 4): invalid casts and parses raise errors, while the try_* functions return null instead.
  • Joins: a left semi join returns left rows that have a match, with only left columns. A left anti join returns left rows with no match. A cross join needs no special setting.
  • Counting: approx_count_distinct trades exactness for speed. describe() and summary() give summary statistics.
  • Shared values: a broadcast variable ships read-only data once per executor. Accumulator updates are applied exactly once only inside actions. Inside transformations, a retried task can add twice.

Question 5: in a stream, when is a result final?

Structured Streaming treats a stream as a table that keeps growing. Three output modes decide what is written: append (new rows only, the default), update (changed rows) and complete (the whole result). An aggregation without a watermark cannot use append, because its counts can still change, and the file sink supports only append.

A watermark (withWatermark("ts", "10 minutes")) declares when a window is final, which lets Spark drop its state. It must be on the same event-time column as the aggregation, and it must come before the aggregation. In append mode, a window is emitted once, only after the watermark passes its end. For deduplication, dropDuplicates(["guid"]) without a watermark keeps every key forever. With a watermark, put the event time in the key, or use dropDuplicatesWithinWatermark (Spark 3.5 and later).

For scheduled incremental runs, use trigger(availableNow=True). The one-time trigger is deprecated, and the continuous trigger is experimental and gives only at-least-once guarantees.

Going deeper: the three parts of exactly-once

Exactly-once results need all three parts at once. The source must be replayable, with offsets (Kafka or files), so a range can be read again. The engine writes the offset range of each micro-batch, and the running state, to write-ahead logs in the checkpoint location before committing. The sink must be idempotent, so writing the same batch twice leaves the same result. After a crash, restarting with the same checkpoint re-runs the uncommitted batch. That is why a checkpoint alone does not help with the socket source (it cannot be replayed), and why a new, empty checkpoint directory starts a brand-new query rather than resuming.

DBX-ASD: five questions to ask of every code snippet
1
Does anything actually run?
Transformations only add to the plan; each action starts a job; cache() is lazy
2
Where does it run?
Driver plans and schedules; executors run tasks; collect() pulls all to the driver
3
How many pieces, does data move?
One task per partition; wide operations shuffle; repartition vs coalesce; AQE
4
What exactly comes back?
Save modes, views, ANSI mode and try_* functions, join types
5
In a stream, when is it final?
Output modes; a watermark on the event-time column, before the aggregation
Ask the five questions in this order before reading the options: it catches the most traps first.

The two small domains

Pandas API on Spark (import pyspark.pandas as ps) runs pandas-style code distributed. to_pandas() collects to the driver. Avoid the sequence default index type on large data, because it can force everything onto one partition. For pandas UDFs, the type decides the shape: Series to Series keeps the length, Series to Scalar acts as an aggregate, applyInPandas gets a whole group, and mapInPandas gets batches and may change the row count.

A study order

  1. DataFrame API: columns, rows, joins, aggregations, dates, schemas, UDFs.
  2. Architecture and Spark SQL: driver and executors, jobs, stages and tasks, caching, reading, writing and views.
  3. Tuning and streaming: partitions, AQE, out-of-memory diagnosis, output modes, watermarks.
  4. Spark Connect and pandas API on Spark. Then timed mixed sets, predicting each output before reading the options.

Practise

Practise in the question bank: every question explains each option, right and wrong. Start with the domain you just studied, then take a timed mock exam.

Domains and weights as listed in the Databricks exam guide for the Associate Developer for Apache Spark certification. Configuration defaults quoted here are from the Apache Spark documentation and differ between Spark versions; check the version the current exam guide uses.