Databricks-Machine-Learning-Associate Exam Practice Test Instant Access

Question 1

A data scientist has a Spark DataFrame spark_df. They want to create a new Spark DataFrame that contains only the rows from spark_df where the value in column price is greater than 0.

Which of the following code blocks will accomplish this task?

Aspark_df[spark_df['price'] > 0]

Bspark_df.filter(col('price') > 0)

CSELECT * FROM spark_df WHERE price > 0

Dspark_df.loc[spark_df['price'] > 0,:]

Espark_df.loc[:,spark_df['price'] > 0]

Answer : B

To filter rows in a Spark DataFrame based on a condition, you use the filter method along with a column condition. The correct syntax in PySpark to accomplish this task is spark_df.filter(col('price') > 0), which filters the DataFrame to include only those rows where the value in the 'price' column is greater than 0. The col function is used to specify column-based operations. The other options provided either do not use correct Spark DataFrame syntax or are intended for different types of data manipulation frameworks like pandas. Reference:

PySpark DataFrame API documentation (Filtering DataFrames).

Question 2

A machine learning engineer has created a Feature Table new_table using Feature Store Client fs. When creating the table, they specified a metadata description with key information about the Feature Table. They now want to retrieve that metadata programmatically.

Which of the following lines of code will return the metadata description?

AThere is no way to return the metadata description programmatically.

Bfs.create_training_set('new_table')

Cfs.get_table('new_table').description

Dfs.get_table('new_table').load_df()

Efs.get_table('new_table')

Answer : C

To retrieve the metadata description of a feature table created using the Feature Store Client (referred here as fs), the correct method involves calling get_table on the fs client with the table name as an argument, followed by accessing the description attribute of the returned object. The code snippet fs.get_table('new_table').description correctly achieves this by fetching the table object for 'new_table' and then accessing its description attribute, where the metadata is stored. The other options do not correctly focus on retrieving the metadata description. Reference:

Databricks Feature Store documentation (Accessing Feature Table Metadata).

Question 3

A machine learning engineer is trying to scale a machine learning pipeline pipeline that contains multiple feature engineering stages and a modeling stage. As part of the cross-validation process, they are using the following code block:

A colleague suggests that the code block can be changed to speed up the tuning process by passing the model object to the estimator parameter and then placing the updated cv object as the final stage of the pipeline in place of the original model.

Which of the following is a negative consequence of the approach suggested by the colleague?

AThe model will take longer to train for each unique combination of hvperparameter values

BThe feature engineering stages will be computed using validation data

CThe cross-validation process will no longer be

DThe cross-validation process will no longer be reproducible

EThe model will be refit one more per cross-validation fold

Answer : B

If the model object is passed to the estimator parameter of CrossValidator and the cross-validation object itself is placed as a stage in the pipeline, the feature engineering stages within the pipeline would be applied separately to each training and validation fold during cross-validation. This leads to a significant issue: the feature engineering stages would be computed using validation data, thereby leaking information from the validation set into the training process. This would potentially invalidate the cross-validation results by giving an overly optimistic performance estimate. Reference:

Cross-validation and Pipeline Integration in MLlib (Avoiding Data Leakage in Pipelines).

Question 4

A data scientist is wanting to explore summary statistics for Spark DataFrame spark_df. The data scientist wants to see the count, mean, standard deviation, minimum, maximum, and interquartile range (IQR) for each numerical feature.

Which of the following lines of code can the data scientist run to accomplish the task?

Aspark_df.summary ()

Bspark_df.stats()

Cspark_df.describe().head()

Dspark_df.printSchema()

Espark_df.toPandas()

Answer : A

The summary() function in PySpark's DataFrame API provides descriptive statistics which include count, mean, standard deviation, min, max, and quantiles for numeric columns. Here are the steps on how it can be used:

Import PySpark: Ensure PySpark is installed and correctly configured in the Databricks environment.

Load Data: Load the data into a Spark DataFrame.

Apply Summary: Use spark_df.summary() to generate summary statistics.

View Results: The output from the summary() function includes the statistics specified in the query (count, mean, standard deviation, min, max, and potentially quartiles which approximate the interquartile range).

Reference

PySpark Documentation: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.summary.html

Question 5

A data scientist wants to efficiently tune the hyperparameters of a scikit-learn model. They elect to use the Hyperopt library's fmin operation to facilitate this process. Unfortunately, the final model is not very accurate. The data scientist suspects that there is an issue with the objective_function being passed as an argument to fmin.

They use the following code block to create the objective_function:

Which of the following changes does the data scientist need to make to their objective_function in order to produce a more accurate model?

AAdd test set validation process

BAdd a random_state argument to the RandomForestRegressor operation

CRemove the mean operation that is wrapping the cross_val_score operation

DReplace the r2 return value with -r2

EReplace the fmin operation with the fmax operation

Answer : D

When using the Hyperopt library with fmin, the goal is to find the minimum of the objective function. Since you are using cross_val_score to calculate the R2 score which is a measure of the proportion of the variance for a dependent variable that's explained by an independent variable(s) in a regression model, higher values are better. However, fmin seeks to minimize the objective function, so to align with fmin's goal, you should return the negative of the R2 score (-r2). This way, by minimizing the negative R2, fmin is effectively maximizing the R2 score, which can lead to a more accurate model.

Reference

Hyperopt Documentation: http://hyperopt.github.io/hyperopt/

Scikit-Learn documentation on model evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

Question 6

In which of the following situations is it preferable to impute missing feature values with their median value over the mean value?

AWhen the features are of the categorical type

BWhen the features are of the boolean type

CWhen the features contain a lot of extreme outliers

DWhen the features contain no outliers

EWhen the features contain no missing no values

Answer : C

Imputing missing values with the median is often preferred over the mean in scenarios where the data contains a lot of extreme outliers. The median is a more robust measure of central tendency in such cases, as it is not as heavily influenced by outliers as the mean. Using the median ensures that the imputed values are more representative of the typical data point, thus preserving the integrity of the dataset's distribution. The other options are not specifically relevant to the question of handling outliers in numerical data. Reference:

Data Imputation Techniques (Dealing with Outliers).

Question 7

Which of the following tools can be used to distribute large-scale feature engineering without the use of a UDF or pandas Function API for machine learning pipelines?

AKeras

Bpandas

CPvTorch

DSpark ML

EScikit-learn

Answer : D

Spark ML (Machine Learning Library) is designed specifically for handling large-scale data processing and machine learning tasks directly within Apache Spark. It provides tools and APIs for large-scale feature engineering without the need to rely on user-defined functions (UDFs) or pandas Function API, allowing for more scalable and efficient data transformations directly distributed across a Spark cluster. Unlike Keras, pandas, PyTorch, and scikit-learn, Spark ML operates natively in a distributed environment suitable for big data scenarios. Reference:

Spark MLlib documentation (Feature Engineering with Spark ML).

Databricks-Machine-Learning-Associate Databricks Certified Machine Learning Associate Exam Practice Test