You are storing data in Cloud Storage for a machine learning project. The data is frequently accessed during the model training phase, minimally accessed after 30 days, and unlikely to be accessed after 90 days. You need to choose the appropriate storage class for the different stages of the project to minimize cost. What should you do?
Answer : B
Comprehensive and Detailed In-Depth
Cost minimization requires matching storage classes to access patterns using lifecycle rules. Let's assess:
Option A: Nearline during training (frequent access) incurs high retrieval costs and latency, unsuitable for ML workloads. Coldline after 30 days and Archive after 90 days are reasonable but misaligned initially.
Option B: Standard storage (no retrieval fees, low latency) is ideal for frequent access during training. Transitioning to Nearline (30-day minimum, low access) after 30 days and Coldline (90-day minimum, rare access) after 90 days matches the pattern and minimizes costs effectively.
Option C: Nearline during training is costly for frequent access, and Archive to Coldline is illogical (Archive is cheaper than Coldline).
Option D: DRA storage doesn't exist in Google Cloud (legacy AWS term); the progression should be Standard -> Nearline -> Coldline. Why B is Best: Standard ensures training efficiency, while Nearline and Coldline reduce costs as access drops, all manageable via lifecycle rules (e.g., SetStorageClass actions). This is Google's recommended tiering strategy. Extract from Google Documentation: From 'Cloud Storage Classes' (https://cloud.google.com/storage/docs/storage-classes): 'Use Standard storage for frequently accessed data, such as during active ML training. Transition to Nearline after 30 days for infrequent access, and Coldline after 90 days for rare access, optimizing costs with lifecycle management.' Reference: Google Cloud Documentation - 'Storage Classes' (https://cloud.google.com/storage/docs/storage-classes).
Why B is Best: Standard ensures training efficiency, while Nearline and Coldline reduce costs as access drops, all manageable via lifecycle rules (e.g., SetStorageClass actions). This is Google's recommended tiering strategy.
Extract from Google Documentation: From 'Cloud Storage Classes' (https://cloud.google.com/storage/docs/storage-classes): 'Use Standard storage for frequently accessed data, such as during active ML training. Transition to Nearline after 30 days for infrequent access, and Coldline after 90 days for rare access, optimizing costs with lifecycle management.'
Option D: DRA storage doesn't exist in Google Cloud (legacy AWS term); the progression should be Standard -> Nearline -> Coldline. Why B is Best: Standard ensures training efficiency, while Nearline and Coldline reduce costs as access drops, all manageable via lifecycle rules (e.g., SetStorageClass actions). This is Google's recommended tiering strategy. Extract from Google Documentation: From 'Cloud Storage Classes' (https://cloud.google.com/storage/docs/storage-classes): 'Use Standard storage for frequently accessed data, such as during active ML training. Transition to Nearline after 30 days for infrequent access, and Coldline after 90 days for rare access, optimizing costs with lifecycle management.' Reference: Google Cloud Documentation - 'Storage Classes' (https://cloud.google.com/storage/docs/storage-classes).
You have a BigQuery dataset containing sales dat
a. This data is actively queried for the first 6 months. After that, the data is not queried but needs to be retained for 3 years for compliance reasons. You need to implement a data management strategy that meets access and compliance requirements, while keeping cost and administrative overhead to a minimum. What should you do?
Answer : B
Partitioning the BigQuery table by month allows efficient querying of recent data for the first 6 months, reducing query costs. After 6 months, exporting the data to Coldline storage minimizes storage costs for data that is rarely accessed but needs to be retained for compliance. Implementing a lifecycle policy in Cloud Storage automates the deletion of the data after 3 years, ensuring compliance while reducing administrative overhead. This approach balances cost efficiency and compliance requirements effectively.
Your team needs to analyze large datasets stored in BigQuery to identify trends in user behavior. The analysis will involve complex statistical calculations, Python packages, and visualizations. You need to recommend a managed collaborative environment to develop and share the analysis. What should you recommend?
Answer : A
Using a Colab Enterprise notebook connected to BigQuery provides a managed, collaborative environment ideal for complex statistical calculations, Python packages, and visualizations. Colab Enterprise supports Python libraries for advanced analytics and offers seamless integration with BigQuery for querying large datasets. It allows teams to collaboratively develop and share analyses while taking advantage of its visualization capabilities. This approach is particularly suitable for tasks involving sophisticated computations and custom visualizations.
Your company stores historical data in Cloud Storage. You need to ensure that all data is saved in a bucket for at least three years. What should you do?
Answer : C
Comprehensive and Detailed in Depth
Why C is correct:Bucket retention policies are specifically designed to enforce a minimum retention period for objects within a Cloud Storage bucket. This ensures that data cannot be deleted or overwritten before the specified period.
Why other options are incorrect:A: Object versioning allows you to keep multiple versions of an object, but it doesn't guarantee a minimum retention period.
B: Changing the storage class to Archive is for cost optimization, not data retention enforcement.
D: Object holds are for legal holds, not general retention.
Cloud Storage Retention Policies: https://cloud.google.com/storage/docs/bucket-lock
Cloud Storage Object Versioning: https://cloud.google.com/storage/docs/object-versioning
Cloud Storage Storage Classes: https://cloud.google.com/storage/docs/storage-classes
Cloud Storage Object Holds: https://cloud.google.com/storage/docs/legal-holds
You work for a retail company that collects customer data from various sources:
Online transactions: Stored in a MySQL database
Customer feedback: Stored as text files on a company server
Social media activity: Streamed in real-time from social media platforms
You need to design a data pipeline to extract and load the data into the appropriate Google Cloud storage system(s) for further analysis and ML model training. What should you do?
Answer : B
Comprehensive and Detailed In-Depth
The pipeline must extract diverse data types and load them into systems optimized for analysis and ML. Let's assess:
Option A: Cloud SQL for transactions keeps data relational but isn't ideal for analysis/ML (less scalable than BigQuery). BigQuery for feedback is fine but skips staging. Cloud Storage for streaming social media loses real-time context and requires extra steps for analysis.
Option B: BigQuery for transactions (via export from MySQL) supports analysis/ML with SQL. Cloud Storage stages feedback text files for preprocessing, then BigQuery ingestion. Pub/Sub and Dataflow stream social media into BigQuery, enabling real-time analysis---optimal for all sources.
Option C: Cloud Storage for all data is a staging step, not a final solution for analysis/ML, requiring additional pipelines.
Option D: Bigtable for transactions is for NoSQL workloads, not analytics. Cloud Storage for feedback is staging-only. Cloud SQL for streaming social media is impractical (not real-time optimized). Why B is Best: BigQuery is Google's analytics/ML hub (e.g., BigQuery ML). Staging feedback in Cloud Storage allows preprocessing, and streaming via Pub/Sub/Dataflow ensures real-time data in BigQuery. For example, MySQL data exports to BigQuery via bq load, feedback uploads to GCS, and Dataflow processes social media into BigQuery tables. Extract from Google Documentation: From 'Data Analytics on Google Cloud' (https://cloud.google.com/architecture/data-analytics): 'Load structured data (e.g., MySQL) and unstructured data (e.g., text) into BigQuery for analysis and ML, using Cloud Storage for staging and Pub/Sub with Dataflow for streaming real-time data like social media activity.' Reference: Google Cloud Documentation - 'BigQuery Data Ingestion' (https://cloud.google.com/bigquery/docs/loading-data).
Why B is Best: BigQuery is Google's analytics/ML hub (e.g., BigQuery ML). Staging feedback in Cloud Storage allows preprocessing, and streaming via Pub/Sub/Dataflow ensures real-time data in BigQuery. For example, MySQL data exports to BigQuery via bq load, feedback uploads to GCS, and Dataflow processes social media into BigQuery tables.
Extract from Google Documentation: From 'Data Analytics on Google Cloud' (https://cloud.google.com/architecture/data-analytics): 'Load structured data (e.g., MySQL) and unstructured data (e.g., text) into BigQuery for analysis and ML, using Cloud Storage for staging and Pub/Sub with Dataflow for streaming real-time data like social media activity.'
Option D: Bigtable for transactions is for NoSQL workloads, not analytics. Cloud Storage for feedback is staging-only. Cloud SQL for streaming social media is impractical (not real-time optimized). Why B is Best: BigQuery is Google's analytics/ML hub (e.g., BigQuery ML). Staging feedback in Cloud Storage allows preprocessing, and streaming via Pub/Sub/Dataflow ensures real-time data in BigQuery. For example, MySQL data exports to BigQuery via bq load, feedback uploads to GCS, and Dataflow processes social media into BigQuery tables. Extract from Google Documentation: From 'Data Analytics on Google Cloud' (https://cloud.google.com/architecture/data-analytics): 'Load structured data (e.g., MySQL) and unstructured data (e.g., text) into BigQuery for analysis and ML, using Cloud Storage for staging and Pub/Sub with Dataflow for streaming real-time data like social media activity.' Reference: Google Cloud Documentation - 'BigQuery Data Ingestion' (https://cloud.google.com/bigquery/docs/loading-data).
Your organization has decided to move their on-premises Apache Spark-based workload to Google Cloud. You want to be able to manage the code without needing to provision and manage your own cluster. What should you do?
Answer : A
Migrating the Spark jobs to Dataproc Serverless is the best approach because it allows you to run Spark workloads without the need to provision or manage clusters. Dataproc Serverless automatically scales resources based on workload requirements, simplifying operations and reducing administrative overhead. This solution is ideal for organizations that want to focus on managing their Spark code without worrying about the underlying infrastructure. It is cost-effective and fully managed, aligning well with the goal of minimizing cluster management.
You are designing a BigQuery data warehouse with a team of experienced SQL developers. You need to recommend a cost-effective, fully-managed, serverless solution to build ELT processes with SQL pipelines. Your solution must include source code control, environment parameterization, and data quality checks. What should you do?
Answer : B
Comprehensive and Detailed In-Depth
The solution must support SQL-based ELT, be serverless and cost-effective, and include advanced features like version control and quality checks. Let's dive in:
Option A: Cloud Data Fusion is a visual ETL tool, not SQL-centric (uses plugins), and isn't fully serverless (requires instance management). It lacks native source code control and parameterization.
Option B: Dataform is a serverless, SQL-based ELT platform for BigQuery. It uses SQLX scripts, integrates with Git for version control, supports environment variables (parameterization), and offers assertions for data quality---all meeting the requirements cost-effectively.
Option C: Dataproc is for Spark/MapReduce, not SQL ELT, and requires cluster management, contradicting serverless and cost goals.
Option D: Cloud Composer orchestrates workflows (Python DAGs), not SQL pipelines natively. It's managed but not optimized for ELT within BigQuery alone. Why B is Best: Dataform leverages your team's SQL skills, runs in BigQuery (no extra infrastructure), and provides Git integration (e.g., GitHub), parameterization (e.g., DECLARE env STRING DEFAULT 'prod';), and quality checks (e.g., assert 'no_nulls' AS SELECT COUNT(*) FROM table WHERE col IS NULL). It's the perfect fit. Extract from Google Documentation: From 'Dataform Overview' (https://cloud.google.com/dataform/docs): 'Dataform is a fully managed, serverless solution for building SQL-based ELT pipelines in BigQuery, with built-in Git version control, environment parameterization, and data quality assertions for robust data warehouse management.' Reference: Google Cloud Documentation - 'Dataform' (https://cloud.google.com/dataform).
Why B is Best: Dataform leverages your team's SQL skills, runs in BigQuery (no extra infrastructure), and provides Git integration (e.g., GitHub), parameterization (e.g., DECLARE env STRING DEFAULT 'prod';), and quality checks (e.g., assert 'no_nulls' AS SELECT COUNT(*) FROM table WHERE col IS NULL). It's the perfect fit.
Extract from Google Documentation: From 'Dataform Overview' (https://cloud.google.com/dataform/docs): 'Dataform is a fully managed, serverless solution for building SQL-based ELT pipelines in BigQuery, with built-in Git version control, environment parameterization, and data quality assertions for robust data warehouse management.'
Option D: Cloud Composer orchestrates workflows (Python DAGs), not SQL pipelines natively. It's managed but not optimized for ELT within BigQuery alone. Why B is Best: Dataform leverages your team's SQL skills, runs in BigQuery (no extra infrastructure), and provides Git integration (e.g., GitHub), parameterization (e.g., DECLARE env STRING DEFAULT 'prod';), and quality checks (e.g., assert 'no_nulls' AS SELECT COUNT(*) FROM table WHERE col IS NULL). It's the perfect fit. Extract from Google Documentation: From 'Dataform Overview' (https://cloud.google.com/dataform/docs): 'Dataform is a fully managed, serverless solution for building SQL-based ELT pipelines in BigQuery, with built-in Git version control, environment parameterization, and data quality assertions for robust data warehouse management.' Reference: Google Cloud Documentation - 'Dataform' (https://cloud.google.com/dataform).