Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Databricks Certified Associate Developer for Apache Spark 3.0 Exam Practice Test

Page: 1 / 14
Total 180 questions

Want more questions? Get Premium Access.
()

Question 1

Which of the following code blocks reads in the parquet file stored at location filePath, given that all columns in the parquet file contain only whole numbers and are stored in the most appropriate

format for this kind of data?

A1. spark.read.schema(
2. StructType(
3. StructField('transactionId', IntegerType(), True),
4. StructField('predError', IntegerType(), True)
5. )).load(filePath)

B1. spark.read.schema([
2. StructField('transactionId', NumberType(), True),
3. StructField('predError', IntegerType(), True)
4. ]).load(filePath)

C1. spark.read.schema(
2. StructType([
3. StructField('transactionId', StringType(), True),
4. StructField('predError', IntegerType(), True)]
5. )).parquet(filePath)

D1. spark.read.schema(
2. StructType([
3. StructField('transactionId', IntegerType(), True),
4. StructField('predError', IntegerType(), True)]
5. )).format('parquet').load(filePath)

E1. spark.read.schema([
2. StructField('transactionId', IntegerType(), True),
3. StructField('predError', IntegerType(), True)
4. ]).load(filePath, format='parquet')

Answer : D

The schema passed into schema should be of type StructType or a string, so all entries in which a list is passed are incorrect.

In addition, since all numbers are whole numbers, the IntegerType() data type is the correct option here. NumberType() is not a valid data type and StringType() would fail, since the parquet file is

stored in the 'most appropriate format for this kind of data', meaning that it is most likely an IntegerType, and Spark does not convert data types if a schema is provided.

Also note that StructType accepts only a single argument (a list of StructFields). So, passing multiple arguments is invalid.

Finally, Spark needs to know which format the file is in. However, all of the options listed are valid here, since Spark assumes parquet as a default when no file format is specifically passed.

More info: pyspark.sql.DataFrameReader.schema --- PySpark 3.1.2 documentation and StructType --- PySpark 3.1.2 documentation

Question 2

Which of the following code blocks returns a copy of DataFrame itemsDf where the column supplier has been renamed to manufacturer?

AitemsDf.withColumn(['supplier', 'manufacturer'])

BitemsDf.withColumn('supplier').alias('manufacturer')

CitemsDf.withColumnRenamed('supplier', 'manufacturer')

DitemsDf.withColumnRenamed(col('manufacturer'), col('supplier'))

EitemsDf.withColumnsRenamed('supplier', 'manufacturer')

Answer : C

itemsDf.withColumnRenamed('supplier', 'manufacturer')

Correct! This uses the relatively trivial DataFrame method withColumnRenamed for renaming column supplier to column manufacturer.

Note that the Question: asks for 'a copy of DataFrame itemsDf'. This may be confusing if you are not familiar with Spark yet. RDDs (Resilient Distributed Datasets) are the foundation of

Spark DataFrames and are immutable. As such, DataFrames are immutable, too. Any command that changes anything in the DataFrame therefore necessarily returns a copy, or a new version, of it

that has the changes applied.

itemsDf.withColumnsRenamed('supplier', 'manufacturer')

Incorrect. Spark's DataFrame API does not have a withColumnsRenamed() method.

itemsDf.withColumnRenamed(col('manufacturer'), col('supplier'))

No. Watch out -- although the col() method works for many methods of the DataFrame API, withColumnRenamed is not one of them. As outlined in the documentation linked below,

withColumnRenamed expects strings.

itemsDf.withColumn(['supplier', 'manufacturer'])

Wrong. While DataFrame.withColumn() exists in Spark, it has a different purpose than renaming columns. withColumn is typically used to add columns to DataFrames, taking the name of the new

column as a first, and a Column as a second argument. Learn more via the documentation that is linked below.

itemsDf.withColumn('supplier').alias('manufacturer')

No. While DataFrame.withColumn() exists, it requires 2 arguments. Furthermore, the alias() method on DataFrames would not help the cause of renaming a column much. DataFrame.alias() can be

useful in addressing the input of join statements. However, this is far outside of the scope of this question. If you are curious nevertheless, check out the link below.

More info: pyspark.sql.DataFrame.withColumnRenamed --- PySpark 3.1.1 documentation, pyspark.sql.DataFrame.withColumn --- PySpark 3.1.1 documentation, and pyspark.sql.DataFrame.alias ---

PySpark 3.1.2 documentation (https://bit.ly/3aSB5tm , https://bit.ly/2Tv4rbE , https://bit.ly/2RbhBd2)

Static notebook | Dynamic notebook: See test 1, Question: 31 (Databricks import instructions) (https://flrs.github.io/spark_practice_tests_code/#1/31.html ,

https://bit.ly/sparkpracticeexams_import_instructions)

Question 3

Which of the following are valid execution modes?

AKubernetes, Local, Client

BClient, Cluster, Local

CServer, Standalone, Client

DCluster, Server, Local

EStandalone, Client, Cluster

Answer : B

This is a tricky Question: to get right, since it is easy to confuse execution modes and deployment modes. Even in literature, both terms are sometimes used interchangeably.

There are only 3 valid execution modes in Spark: Client, cluster, and local execution modes. Execution modes do not refer to specific frameworks, but to where infrastructure is located with respect

to each other.

In client mode, the driver sits on a machine outside the cluster. In cluster mode, the driver sits on a machine inside the cluster. Finally, in local mode, all Spark infrastructure is started in a single JVM

(Java Virtual Machine) in a single computer which then also includes the driver.

Deployment modes often refer to ways that Spark can be deployed in cluster mode and how it uses specific frameworks outside Spark. Valid deployment modes are standalone, Apache YARN,

Apache Mesos and Kubernetes.

Client, Cluster, Local

Correct, all of these are the valid execution modes in Spark.

Standalone, Client, Cluster

No, standalone is not a valid execution mode. It is a valid deployment mode, though.

Kubernetes, Local, Client

No, Kubernetes is a deployment mode, but not an execution mode.

Cluster, Server, Local

No, Server is not an execution mode.

Server, Standalone, Client

No, standalone and server are not execution modes.

More info: Apache Spark Internals - Learning Journal

Question 4

The code block displayed below contains an error. The code block should return a copy of DataFrame transactionsDf where the name of column transactionId has been changed to

transactionNumber. Find the error.

Code block:

transactionsDf.withColumn("transactionNumber", "transactionId")

AThe arguments to the withColumn method need to be reordered.

BThe arguments to the withColumn method need to be reordered and the copy() operator should be appended to the code block to ensure a copy is returned.

CThe copy() operator should be appended to the code block to ensure a copy is returned.

DEach column name needs to be wrapped in the col() method and method withColumn should be replaced by method withColumnRenamed.

EThe method withColumn should be replaced by method withColumnRenamed and the arguments to the method need to be reordered.

Answer : E

Correct code block:

transactionsDf.withColumnRenamed('transactionId', 'transactionNumber')

Note that in Spark, a copy is returned by default. So, there is no need to append copy() to the code block.

More info: pyspark.sql.DataFrame.withColumnRenamed --- PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 2, Question: 26 (Databricks import instructions)

Question 5

The code block shown below should return all rows of DataFrame itemsDf that have at least 3 items in column itemNameElements. Choose the answer that correctly fills the blanks in the code block

to accomplish this.

Example of DataFrame itemsDf:

1. +------+----------------------------------+-------------------+------------------------------------------+

3. +------+----------------------------------+-------------------+------------------------------------------+

7. +------+----------------------------------+-------------------+------------------------------------------+

Code block:

itemsDf.__1__(__2__(__3__)__4__)

A1. select
2. count
3. col('itemNameElements')
4. >3

B1. filter
2. count
3. itemNameElements
4. >=3

C1. select
2. count
3. 'itemNameElements'
4. >3

D1. filter
2. size
3. 'itemNameElements'
4. >=3
(Correct)

E1. select
2. size
3. 'itemNameElements'
4. >3

Answer : D

Correct code block:

itemsDf.filter(size('itemNameElements')>3)

Output of code block:

+------+----------------------------------+-------------------+------------------------------------------+

+------+----------------------------------+-------------------+------------------------------------------+

+------+----------------------------------+-------------------+------------------------------------------+

The big difficulty with this Question: is in knowing the difference between count and size (refer to documentation below). size is the correct function to choose here since it returns the number

of elements in an array on a per-row basis.

The other consideration for solving this Question: is the difference between select and filter. Since we want to return the rows in the original DataFrame, filter is the right choice. If we would

use select, we would simply get a single-column DataFrame showing which rows match the criteria, like so:

+----------------------------+

|(size(itemNameElements) > 3)|

+----------------------------+

|true |

|false |

+----------------------------+

More info:

Count documentation: pyspark.sql.functions.count --- PySpark 3.1.1 documentation

Size documentation: pyspark.sql.functions.size --- PySpark 3.1.1 documentation

Static notebook | Dynamic notebook: See test 1, Question: 47 (Databricks import instructions)

Question 6

Which of the following describes a valid concern about partitioning?

AA shuffle operation returns 200 partitions if not explicitly set.

BDecreasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.

CNo data is exchanged between executors when coalesce() is run.

DShort partition processing times are indicative of low skew.

EThe coalesce() method should be used to increase the number of partitions.

Answer : A

A shuffle operation returns 200 partitions if not explicitly set.

Correct. 200 is the default value for the Spark property spark.sql.shuffle.partitions. This property determines how many partitions Spark uses when shuffling data for joins or aggregations.

The coalesce() method should be used to increase the number of partitions.

Incorrect. The coalesce() method can only be used to decrease the number of partitions.

Decreasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.

No. For narrow transformations, fewer partitions usually result in a longer overall runtime, if more executors are available than partitions.

A narrow transformation does not include a shuffle, so no data need to be exchanged between executors. Shuffles are expensive and can be a bottleneck for executing Spark workloads.

Narrow transformations, however, are executed on a per-partition basis, blocking one executor per partition. So, it matters how many executors are available to perform work in parallel relative to the

number of partitions. If the number of executors is greater than the number of partitions, then some executors are idle while other process the partitions. On the flip side, if the number of executors is

smaller than the number of partitions, the entire operation can only be finished after some executors have processed multiple partitions, one after the other. To minimize the overall runtime, one

would want to have the number of partitions equal to the number of executors (but not more).

So, for the scenario at hand, increasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.

No data is exchanged between executors when coalesce() is run.

No. While coalesce() avoids a full shuffle, it may still cause a partial shuffle, resulting in data exchange between executors.

Short partition processing times are indicative of low skew.

Incorrect. Data skew means that data is distributed unevenly over the partitions of a dataset. Low skew therefore means that data is distributed evenly.

Partition processing time, the time that executors take to process partitions, can be indicative of skew if some executors take a long time to process a partition, but others do not. However, a short

processing time is not per se indicative a low skew: It may simply be short because the partition is small.

A situation indicative of low skew may be when all executors finish processing their partitions in the same timeframe. High skew may be indicated by some executors taking much longer to finish their

partitions than others. But the answer does not make any comparison -- so by itself it does not provide enough information to make any assessment about skew.

More info: Spark Repartition & Coalesce - Explained and Performance Tuning - Spark 3.1.2 Documentation

Question 7

Which of the following describes Spark's Adaptive Query Execution?

AAdaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.

BAdaptive Query Execution is enabled in Spark by default.

CAdaptive Query Execution reoptimizes queries at execution points.

DAdaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.

EAdaptive Query Execution applies to all kinds of queries.

Answer : D

Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.

This is almost correct. All of these features, except for dynamically injecting scan filters, are part of Adaptive Query Execution. Dynamically injecting scan filters for join operations to limit the amount

of data to be considered in a query is part of Dynamic Partition Pruning and not of Adaptive Query Execution.

Adaptive Query Execution reoptimizes queries at execution points.

No, Adaptive Query Execution reoptimizes queries at materialization points.

Adaptive Query Execution is enabled in Spark by default.

No, Adaptive Query Execution is disabled in Spark needs to be enabled through the spark.sql.adaptive.enabled property.

Adaptive Query Execution applies to all kinds of queries.

No, Adaptive Query Execution applies only to queries that are not streaming queries and that contain at least one exchange (typically expressed through a join, aggregate, or window operator) or

one subquery.

More info: How to Speed up SQL Queries with Adaptive Query Execution, Learning Spark, 2nd Edition, Chapter 12 (https://bit.ly/3tOh8M1)