Declarative Pipelines vs Lakeflow Jobs vs Auto Loader vs COPY INTO: Which One a Databricks Question Wants
Four names show up as options in the same Databricks data engineering questions: declarative pipelines, Lakeflow Jobs running notebooks, Auto Loader and COPY INTO. They look like four competing ways to load data, which makes the questions feel like guesswork. They are not four of the same thing, though. Two of them load files, one declares tables and keeps them up to date, and one runs and schedules work. Once you see which layer each one sits in, the stem nearly always contains the detail that decides.
Concept comparison · DBX-DEA, DBX-DEP and other Databricks data exams

First, the names
Databricks renames products often, and exam material mixes old and new names. In the questions in this question bank:
- Lakeflow Spark Declarative Pipelines, also called Lakeflow Declarative Pipelines or simply Lakeflow pipelines, were formerly Delta Live Tables. They are built on the open Spark Declarative Pipelines.
- Lakeflow Jobs was formerly Workflows.
- Auto Loader (the
cloudFilessource) and COPY INTO kept their names.
If an option uses an older name, it is not wrong for that reason alone. Judge the mechanism.
Three layers, and how they nest
| Layer | Tool | What it is |
|---|---|---|
| Load files | COPY INTO | One SQL statement that loads new files into a Delta table and skips files it already loaded. |
| Load files | Auto Loader | A Structured Streaming source that discovers new files incrementally and records them in a checkpoint. |
| Declare tables | Declarative pipelines | You declare streaming tables, materialized views and views; the engine works out the order, manages checkpoints and retries, and applies expectations. |
| Run and schedule | Lakeflow Jobs | A graph of tasks (notebook, SQL, pipeline, dashboard ...) with triggers, retries, repair runs and notifications. |
They nest. A streaming table that ingests files reads them with Auto Loader underneath; in SQL that is CREATE OR REFRESH STREAMING TABLE bronze_orders AS SELECT * FROM STREAM read_files(...). A job can start that pipeline with a pipeline task, then run a notebook task after it. So "pipeline or job?" is often a false choice: many correct answers use both, and the question is which one owns which part.
The clues in the stem
| If the stem says ... | it usually wants |
|---|---|
| SQL only, files in the thousands rather than millions, safe to rerun after a failure | COPY INTO |
| Millions of files, files arriving continuously, schema that changes, new columns | Auto Loader |
| SQL users who have outgrown COPY INTO | A streaming table (Auto Loader underneath) |
| Bronze, silver and gold tables with dependencies, data quality rules, CDC with SCD type 1 or 2 | A declarative pipeline (expectations, AUTO CDC) |
| Call a REST API, write to several systems per batch, arbitrary Python side effects | A notebook or Python task in Lakeflow Jobs (Structured Streaming with foreachBatch for streams) |
| Run after something else, on a schedule, on file arrival, with retries, notifications or a dashboard refresh | Lakeflow Jobs, around whichever loader or pipeline does the work |
COPY INTO versus Auto Loader
This pair belongs mostly to the DBX-DEA Data Ingestion and Loading section: 25 of the 27 DBX-DEA questions in this bank that name either tool sit there. Both are incremental and both avoid loading a file twice. The differences:
- Scale. The Databricks documentation suggests COPY INTO for files in the order of thousands over time and Auto Loader for millions or more, because Auto Loader needs fewer operations to discover files and can split the work into batches. In file notification mode with file events, Auto Loader lists the directory once on the first run and then finds new files in the file events cache instead of listing again.
- Interface. COPY INTO is one SQL statement you run, typically from a SQL task or a scheduled query. Auto Loader is a stream you start in Python or, through a streaming table, in SQL.
- Schema. Auto Loader infers and evolves the schema (with
cloudFiles.schemaLocation) and puts values that do not fit into_rescued_data. COPY INTO can evolve the target schema when you setmergeSchema, and itsVALIDATEoption checks files without writing. - Reloading a file. COPY INTO skips a file rewritten under the same name;
forcereloads it (and every other selected file). Auto Loader reloads everything if its checkpoint is deleted.
Going deeper: where each one remembers the files it has seen
COPY INTO records which files it has loaded for its target table, so the same statement against the same target skips what is already there. That is what makes it idempotent and safe to rerun after a timeout. Auto Loader keeps its record in the stream's checkpoint location, in a RocksDB store of discovered files, which gives exactly-once file processing and lets a failed stream resume where it stopped. The consequence decides several questions: for COPY INTO the history belongs to the table, for Auto Loader it belongs to the checkpoint. Point an Auto Loader stream at a new checkpoint and, because cloudFiles.includeExistingFiles defaults to true on the first start, every historic file loads again. Two streams sharing one checkpoint mix their offsets, so each stream needs its own.
Declarative pipeline versus a job running notebooks
When both could do the work, choose by what you want to own. A pipeline takes over ordering, checkpoints, retries and data quality, and in exchange restricts what you can write. The engine plans the whole graph before reading any data, so:
pivot()is rejected in a pipeline, because it must read the data to learn the columns; write conditional aggregation instead.- Each dataset can be defined only once. Two sources feeding one streaming table need append flows, not two pipelines.
- Expectations (warn, drop, fail) decide which rows reach a table; they are not orchestration. In a triggered pipeline a failed expectation rolls back only that flow's update while the other flows keep updating, and a table that reads a validation table does not wait for it. To block downstream work, put validation and the downstream work in separate pipelines and chain them as pipeline tasks in a job.
- Side effects such as calling a REST service or running a
MERGEover JDBC are not a supported design inside dataset definitions. That is the moment to use Structured Streaming withforeachBatchas a job task, or, in a Python pipeline, a ForEachBatch sink as the target of an append flow.
Beginner tip: build the same tiny bronze-to-silver flow twice. First as two notebook tasks in a job: an Auto Loader stream with availableNow, then a cleaning query. Then as a pipeline with one streaming table and one materialized view plus an expectation. Look at what you had to write in the first version (checkpoint paths, task order, retry settings) that the pipeline did for you, and at the pipeline's graph and data quality tab. After that, "which one fits?" questions read like a checklist.
Who decides when it runs
A pipeline has a mode: triggered (process what is there, then stop) or continuous (keep running). When a job starts the pipeline, the job wins: a scheduled job runs one update and stops even if the pipeline is set to continuous. Databricks recommends leaving the pipeline triggered and using a continuous job when you want it always on. Plain streaming code follows a similar rule on serverless jobs, which accept only triggers that end on their own such as availableNow; an always-on stream with a short processingTime interval belongs in a continuous serverless pipeline instead.
Going deeper: why a full refresh can lose data a streaming table already had
A streaming table reads each input record once and keeps it, even if the source later deletes it. A full refresh clears the table and its checkpoints and reads again from whatever the source still holds. If a storage lifecycle rule deletes files after thirty days, or a Kafka topic keeps seven days, the rebuilt table holds only that window. This is why the narrower recovery options (refresh failed tables, reset one flow's checkpoint) are usually the right answers at professional level, and why the table property pipelines.reset.allowed = false exists to block full refreshes on tables whose history cannot be replayed.
Practise
Practise in the question bank: every question explains each option, right and wrong. Start with the domain you just studied, then take a timed mock exam.
- DBX-DEA exam page · All 285 DBX-DEA questions · Timed mock exam · 20-day study plan
- By domain (DBX-DEA): Databricks Intelligence Platform · Data Ingestion and Loading · Data Transformation and Modeling · Working with Lakeflow Jobs · Implementing CI/CD · Troubleshooting, Monitoring, and Optimization · Governance and Security
- DBX-DEP exam page · All 360 DBX-DEP questions · Timed mock exam · 25-day study plan
- By domain (DBX-DEP): Developing Code for Data Processing using Python and SQL · Data Ingestion & Acquisition · Data Manipulation · Monitoring and Alerting · Cost & Performance Optimization · Ensuring Data Security and Compliance · Data Governance · Debugging and Deploying · Data Modeling
- DBX-DEA: Data Ingestion and Loading · DBX-DEA: Working with Lakeflow Jobs
- DBX-DEP: Data Ingestion & Acquisition · DBX-DEP: Developing Code for Data Processing
Product names as used in the current Databricks documentation and exam guides at the time of writing; they change often. Check the official exam guides for DBX-DEA and DBX-DEP and the current documentation.