cert-bank

Guides

이 글을 한국어로 읽기

GCP-PDE Study Guide: Five Phrases That Decide Most Data Engineer Questions, and BigQuery Under All of Them

Professional Data Engineer questions usually offer four pipelines that would all move the data. What separates them is one phrase in the stem: "near real time", "must not lose data", "without changing the code", "lowest operational overhead", "the numbers are wrong". Each phrase points to a small set of Google Cloud features, and learning those links is most of the work. This guide is organised around those five phrases, with BigQuery underneath them, because it is underneath most of the exam.

Study guide · Google Cloud Certified Professional Data Engineer (GCP-PDE)

GCP-PDE Study Guide: Five Phrases That Decide Most Data Engineer Questions, and BigQuery Under All of Them

The exam

40 to 50 multiple choice and multiple select questions in 2 hours (this site's mock exam uses 45). Google does not publish a pass mark; check the official exam page for current details. Five sections, with approximate weights from the exam guide: Designing data processing systems (~22%), Ingesting and processing the data (~25%), Storing the data (~20%), Preparing and using data for analysis (~15%) and Maintaining and automating data workloads (~18%).

BigQuery dominates. It is named in the question or its options in 132 of the 225 GCP-PDE questions in this question bank (59%), far ahead of Dataflow (43) and Pub/Sub (27). If you study only one product deeply, study BigQuery: partitioning, clustering, slots and editions, security on columns and rows, sharing, BigQuery ML.

Phrase 1: "near real time" and "late data"

Pub/Sub decouples producers from consumers; Dataflow (Apache Beam) processes the stream. The exam goes beyond "use Dataflow" into how a stream is cut into results:

  • Windows group an unbounded stream by event time: fixed (tumbling), sliding (hopping) or session windows.
  • The watermark is Dataflow's estimate that all data for a window has arrived. By default a window's result is emitted when the watermark passes the window's end, and data arriving after that is dropped.
  • Allowed lateness plus a late trigger keep the window open for stragglers and emit refined results. Data later than the allowed lateness is still dropped, and the downstream table must accept updates for a window it has already received.
Going deeper: event time, processing time and why the watermark exists

A mobile app may record an order at 10:05 and deliver it at 10:40 after reconnecting. Grouped by processing time (when Dataflow sees it), the order lands in the wrong hour. Grouped by event time (when it happened), it lands in the right hour, but then the pipeline has to decide when an hour is "finished", since another 10:05 order could still be on its way. The watermark is that decision: the source estimates how far event time has progressed, and Dataflow emits a window once the watermark passes its end. The trade-off is completeness against latency. Waiting longer catches more stragglers but delays every result; emitting early and then refining with late triggers gives fast answers that change. A question that says "orders arrive up to 30 minutes late and every order must be counted" wants allowed lateness of at least 30 minutes and a late trigger, not a bigger window.

Phrase 2: "must not lose or duplicate data"

Streaming Dataflow jobs run exactly-once by default. At-least-once mode loses nothing but may duplicate, and it is cheaper and faster; it suits map-only work such as log parsing, or pipelines that deduplicate downstream. Counts, sums and fraud detection stay exactly-once. The sink must tell the same story: the BigQuery Storage Write API gives exactly-once writes, while the at-least-once method and the legacy streaming inserts can duplicate.

In Pub/Sub, a message that keeps failing goes to a dead-letter topic after roughly the configured number of delivery attempts (5 to 100), but only if Pub/Sub's service account has the right roles on the topic and the subscription. Exactly-once delivery works only on pull subscriptions, within a region. And to stop a streaming job without losing what it is holding, drain it; cancel stops at once and gives up buffered data.

Phrase 3: "without changing the code"

This phrase protects existing investments. Existing Spark or Hadoop jobs: Dataproc (now shown in the documentation as Managed Service for Apache Spark), ideally as ephemeral clusters created for a job and deleted after, with data in Cloud Storage or BigQuery rather than on HDFS. Existing Airflow DAGs: Cloud Composer (now Managed Service for Apache Airflow). A database that must keep running while you copy it: Database Migration Service for a migration with a short cutover, Datastream for ongoing change data capture into BigQuery while the source stays the system of record. Too little bandwidth for a large file transfer: Transfer Appliance.

Phrase 4: "lowest operational overhead"

When nothing forbids it, the managed or serverless option wins, and the smallest tool that does the job beats the larger one. Match the orchestrator to the complexity:

NeedSmallest fitting tool
One SQL statement on a scheduleBigQuery scheduled query
Start something at a time, no stepsCloud Scheduler
A short chain of service and API calls with retriesWorkflows
SQL transformations in BigQuery with dependenciesDataform
Many dependent tasks, sensors, backfills, Python DAGsCloud Composer (managed Apache Airflow)
Integration without codeCloud Data Fusion

The same idea applies to storage: pick from the access pattern, not the product name. Large analytical scans go to BigQuery; very high-throughput key lookups and time series to Bigtable; relational transactions to Cloud SQL or AlloyDB, and to Spanner only when writes must scale out or stay consistent across regions. A modest regional application with neither need is usually better served by Cloud SQL or AlloyDB.

Phrase 5: "the numbers are wrong" or "the query is too slow and too expensive"

Failure scope decides recovery. A zone failure is handled by high availability, a region failure by a cross-region replica you can promote, and wrong data by going back in time: BigQuery time travel or table snapshots, point-in-time recovery for databases, Pub/Sub seek to replay a stream into an idempotent sink. Replicas do not help with wrong data, because they copy it.

For cost and speed in BigQuery, partition on the column people filter by (often the event date, not ingestion time when data arrives late), cluster on other frequent filters, and set require partition filter so nobody scans a whole table by accident. Partition expiration removes old partitions and keeps the table; table expiration deletes the table. For dashboards, materialised views precompute results of repeated, predictable queries, and BI Engine keeps the most frequently used data in memory.

Beginner tip: copy one of BigQuery's public sample tables into your own dataset twice: once as is, once partitioned by a date column and clustered by a second column. Write the same query with a date filter against both, and before running it read the "This query will process ..." estimate in the editor. Partition pruning shows up in that estimate; clustering does not, because the blocks a clustered query skips are known only when it runs, so compare the bytes billed afterwards too. Finally, set "require partition filter" on the partitioned copy and run a query without the date filter: BigQuery refuses it. Ten minutes of this makes the storing and cost questions concrete.

The phrase in the stem that picks the pipeline
"Near real time" or "late data"?
YES →
Dataflow windows, watermark, allowed lateness
↓ no
"Must not lose or duplicate data"?
YES →
Exactly-once Dataflow and the Storage Write API
↓ no
"Without changing the code"?
YES →
Dataproc, Cloud Composer, DMS or Datastream
↓ no
"The numbers are wrong"?
YES →
Go back in time: time travel, snapshots, PITR
↓ no
"The query is too slow and too expensive"?
YES →
Partition, cluster, require partition filter
↓ no
OTHERWISE →
Lowest operational overhead: the smallest managed tool that does the job
Find the phrase first: each one points to a small set of Google Cloud features.

A study order

  1. BigQuery in depth: storage design, partitioning and clustering, security, sharing, slots and editions.
  2. Streaming: Pub/Sub delivery features, Dataflow windows, watermarks, triggers, drain and update.
  3. Batch and migration: Dataproc, Data Fusion, Database Migration Service, Datastream, transfer services.
  4. Other stores (Bigtable row keys, Spanner, Cloud SQL, Firestore, Memorystore), governance with Dataplex (now Knowledge Catalog), and orchestration.
  5. Operations: monitoring, failures and recovery, then mixed sets.

Practise

Practise in the question bank: every question explains each option, right and wrong. Start with the domain you just studied, then take a timed mock exam.

Section weights as listed in Google's Professional Data Engineer exam guide. Several data products are being renamed; check the current official exam page and documentation for the names used.