Databricks Certified Data Engineer Professional Exam Questions Instant Access

Question 1

A Data engineer wants to run unit's tests using common Python testing frameworks on python functions defined across several Databricks notebooks currently used in production.

How can the data engineer run unit tests against function that work with data in production?

ARun unit tests against non-production data that closely mirrors production

BDefine and unit test functions using Files in Repos

CDefine units test and functions within the same notebook

DDefine and import unit test functions from a separate Databricks notebook

Answer : A

The best practice for running unit tests on functions that interact with data is to use a dataset that closely mirrors the production data. This approach allows data engineers to validate the logic of their functions without the risk of affecting the actual production data. It's important to have a representative sample of production data to catch edge cases and ensure the functions will work correctly when used in a production environment.

Databricks Documentation on Testing: Testing and Validation of Data and Notebooks

Question 2

A data engineer deploys a multi-task Databricks job that orchestrates three notebooks. One task intermittently fails with Exit Code 1 but succeeds on retry. The engineer needs to collect detailed logs for the failing attempts, including stdout/stderr and cluster lifecycle context, and share them with the platform team.

What steps the data engineer needs to follow using built-in tools?

AUse the notebook interactive debugger to re-run the entire multi-task job, and capture step-through traces for the failing task.

BDownload worker logs directly from the Spark UI and ignore driver logs, as worker logs contain stdout/stderr for all tasks and cluster events.

CExport the notebook run results to HTML; this bundle includes complete stdout, stderr, and cluster event history across all tasks.

DFrom the job run details page, export the job's logs or configure log delivery; then retrieve the compute driver logs and event logs from the compute details page to correlate stdout/stderr with cluster events.

Answer : D

The recommended way to troubleshoot and collect detailed job logs is through the Job Run Details page in Databricks. From there, engineers can export run logs or configure automatic log delivery to a storage destination. The driver and event logs available under compute details provide stdout, stderr, and cluster lifecycle context required for root-cause analysis.

Reference Source: Databricks Jobs Monitoring and Logging Documentation -- ''Access driver logs and configure log delivery.''

Question 3

The Databricks CLI is used to trigger a run of an existing job by passing the job_id parameter. The response indicating the job run request was submitted successfully includes a field run_id. Which statement describes what the number alongside this field represents?

AThe job_id and number of times the job has been run are concatenated and returned.

BThe globally unique ID of the newly triggered run.

CThe job_id is returned in this field.

DThe number of times the job definition has been run in this workspace.

Answer : B

Exact extract: ''run_id: The canonical identifier of a run.''

===========

Question 4

The data governance team has instituted a requirement that all tables containing Personal Identifiable Information (PH) must be clearly annotated. This includes adding column comments, table comments, and setting the custom table property "contains_pii" = true.

The following SQL DDL statement is executed to create a new table:

Which command allows manual confirmation that these three requirements have been met?

ADESCRIBE EXTENDED dev.pii test

BDESCRIBE DETAIL dev.pii test

CSHOW TBLPROPERTIES dev.pii test

DDESCRIBE HISTORY dev.pii test

ESHOW TABLES dev

Answer : A

This is the correct answer because it allows manual confirmation that these three requirements have been met. The requirements are that all tables containing Personal Identifiable Information (PII) must be clearly annotated, which includes adding column comments, table comments, and setting the custom table property ''contains_pii'' = true. The DESCRIBE EXTENDED command is used to display detailed information about a table, such as its schema, location, properties, and comments. By using this command on the dev.pii_test table, one can verify that the table has been created with the correct column comments, table comment, and custom table property as specified in the SQL DDL statement. Verified Reference: [Databricks Certified Data Engineer Professional], under ''Lakehouse'' section; Databricks Documentation, under ''DESCRIBE EXTENDED'' section.

Question 5

A junior data engineer has manually configured a series of jobs using the Databricks Jobs UI. Upon reviewing their work, the engineer realizes that they are listed as the "Owner" for each job. They attempt to transfer "Owner" privileges to the "DevOps" group, but cannot successfully accomplish this task.

Which statement explains what is preventing this privilege transfer?

ADatabricks jobs must have exactly one owner; 'Owner' privileges cannot be assigned to a group.

BThe creator of a Databricks job will always have 'Owner' privileges; this configuration cannot be changed.

COther than the default 'admins' group, only individual users can be granted privileges on jobs.

DA user can only transfer job ownership to a group if they are also a member of that group.

EOnly workspace administrators can grant 'Owner' privileges to a group.

Answer : A

The reason why the junior data engineer cannot transfer ''Owner'' privileges to the ''DevOps'' group is that Databricks jobs must have exactly one owner, and the owner must be an individual user, not a group. A job cannot have more than one owner, and a job cannot have a group as an owner. The owner of a job is the user who created the job, or the user who was assigned the ownership by another user. The owner of a job has the highest level of permission on the job, and can grant or revoke permissions to other users or groups. However, the owner cannot transfer the ownership to a group, only to another user. Therefore, the junior data engineer's attempt to transfer ''Owner'' privileges to the ''DevOps'' group is not possible.Reference:

Jobs access control: https://docs.databricks.com/security/access-control/table-acls/index.html

Job permissions: https://docs.databricks.com/security/access-control/table-acls/privileges.html#job-permissions

Question 6

A data engineer is attempting to execute the following PySpark code:

df = spark.read.table("sales")

result = df.groupBy("region").agg(sum("revenue"))

However, upon inspecting the execution plan and profiling the Spark job, they observe excessive data shuffling during the aggregation phase.

Which technique should be applied to reduce shuffling during the groupBy aggregation operation?

ACaching the DataFrame df.

BRepartition by region before aggregation.

CUse coalesce() after the aggregation.

DUse broadcast join.

Answer : B

Databricks documents that shuffle occurs when Spark redistributes data across partitions for grouping or joining. To optimize aggregation performance, repartitioning by the grouping key (region) ensures rows with the same key are co-located in the same partition, thus minimizing shuffle movement. Caching improves reuse of DataFrames but does not reduce shuffle volume. coalesce() reduces the number of partitions after computation and cannot prevent shuffle. Broadcast joins are unrelated to single-table aggregations. The recommended practice for reducing shuffle in aggregation is explicit repartitioning by the grouping column.

Question 7

A data engineer is creating a daily reporting job. There are two reporting notebooks---one for weekdays and one for weekends. An ''if/else condition'' task is configured as {{job.start_time.is_weekday}} == true to route the job to either the weekday or weekend notebook tasks. The same job would be used across multiple time zones.

Which action should a senior data engineer take upon reviewing the job to merge or reject the pull request?

AReject, as the {{job.start_time.is_weekday}} is for the UTC timezone.

BReject, as the {{job.start_time.is_weekday}} is not a valid value reference.

CMerge, as the job configuration looks good.

DReject, as they should use {{job.trigger_time.is_weekday}} instead.

Answer : A

Databricks parameter templates like {{job.start_time.is_weekday}} evaluate in UTC time by default, not in local workspace or regional time zones. Therefore, when jobs are configured to run across different time zones, relying on is_weekday using UTC may cause scheduling and task routing mismatches (for example, triggering the weekday notebook in one region while it's still the weekend locally).

Databricks recommends adjusting conditional logic or pipeline parameters explicitly to handle time zone conversions if business requirements depend on local times. Because the engineer's configuration does not account for this behavior, a senior data engineer should reject the pull request and suggest time-zone-aware logic before merging.