Build a Data Mesh with Dataplex: Challenge Lab Reviews

I have a rule named 'NOT NULL' , but task 5 does not recognize.

Allan S. · Reviewed מעל שנה אחת ago

H C. · Reviewed מעל שנה אחת ago

I have a rule named 'NOT NULL' , but task 5 does not recognize.

Allan S. · Reviewed מעל שנה אחת ago

Mirosław G. · Reviewed מעל שנה אחת ago

Mirosław G. · Reviewed מעל שנה אחת ago

Vaishnavi V. · Reviewed מעל שנה אחת ago

The last task is impossible to finish. metadata_registry_defaults: dataplex: projects: qwiklabs-gcp-01-0110787d189f locations: us-west1 lakes: sales-lake zones: curated-customer-zone row_filters: NONE: filter_sql_expr: |- True rule_dimensions: - consistency - correctness - duplication - completeness - conformance - integrity rules: NOT NULL: rule_type: NOT_NULL dimension: completeness rule_bindings: VALID_USER: entity_uri: bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items column_id: user_id row_filter_id: NONE rule_ids: - NOT NULL VALID_ORDER: entity_uri: bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items column_id: order_id row_filter_id: NONE rule_ids: - NOT NULL # Rule named 'NOT NULL' not found for the user_id column in the data quality task. # Please re-create the task including this rule as mentioned in the instructions. # Please create a data quality job named 'customer-orders-data-quality-job' as mentioned in the task. # If already done, wait until the 'results' table is created in 'orders_dq_dataset' dataset along with # data quality results. # Log Using the default container image Waiting for container log creation PYSPARK_PYTHON=/opt/dataproc/conda/bin/python JAVA_HOME=/usr/lib/jvm/temurin-11-jdk-amd64 SPARK_EXTRA_CLASSPATH= :: loading settings :: file = /etc/spark/conf/ivysettings.xml OS runtime details: PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" Python executable path: /opt/dataproc/conda/bin/python Python Version: sys.version_info(major=3, minor=9, micro=15, releaselevel='final', serial=0) PySpark working directory: PosixPath('/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491') PySpark directory content: ['.clouddq-executable.zip.hashsum.crc', 'clouddq-executable.zip', 'clouddq_pyspark_driver.py', '.dq-customer-orders.yaml.crc', '.clouddq_pyspark_driver.py.crc', 'clouddq-executable.zip.hashsum', 'dq-customer-orders.yaml', '.clouddq-executable.zip.crc'] Input PySpark arguments: ['/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/clouddq_pyspark_driver.py', 'clouddq-executable.zip', 'ALL', 'gs://qwiklabs-gcp-01-0110787d189f-dq-config/dq-customer-orders.yaml', '--gcp_project_id="qwiklabs-gcp-01-0110787d189f"', '--gcp_region_id="us-west1"', '--gcp_bq_dataset_id="orders_dq_dataset"', '--target_bigquery_summary_table="qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results"'] User-specified CloudDQ YAML configs: gs://qwiklabs-gcp-01-0110787d189f-dq-config/dq-customer-orders.yaml Creating configs directory at: `/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs` Copying YAML file dq-customer-orders.yaml to configs directory `configs`... Configs directory contents is: [PosixPath('configs/dq-customer-orders.yaml')] CloudDQ executable expected hexdigest: a14971737fa18e443390047a3d8b04a5378768bea0fed8a5bc34adabdba131c1 Executing commands: /opt/dataproc/conda/bin/python clouddq-executable.zip ALL /tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs --gcp_project_id="qwiklabs-gcp-01-0110787d189f" --gcp_region_id="us-west1" --gcp_bq_dataset_id="orders_dq_dataset" --target_bigquery_summary_table="qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results" 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.gcp_credentials[77] INFO Successfully created GCP Client. 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Starting CloudDQ run with configs: {"clouddq_run_configs": {"rule_binding_ids": "ALL", "rule_binding_config_path": "/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs", "environment_target": "dev", "gcp_project_id": "qwiklabs-gcp-01-0110787d189f", "gcp_region_id": "us-west1", "gcp_bq_dataset_id": "orders_dq_dataset", "gcp_service_account_key_path": null, "gcp_impersonation_credentials": null, "metadata": "{}", "dry_run": false, "progress_watermark": true, "target_bigquery_summary_table": "qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results", "intermediate_table_expiration_hours": 24, "num_threads": 8, "debug": false, "print_sql_queries": false, "skip_sql_validation": false, "summary_to_stdout": false, "enable_experimental_bigquery_entity_uris": true, "enable_experimental_dataplex_gcs_validation": true, "bigquery_client": null, "gcp_credentials": {"credentials": "<google.auth.compute_engine.credentials.Credentials object at 0x7f370cf1e280>", "project_id": "qwiklabs-gcp-01-0110787d189f", "user_id": "984150749013-compute@developer.gserviceaccount.com"}}} 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_connection_configs[77] INFO Using Application-Default Credentials (ADC) to authenticate to GCP... 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_connection_configs[77] INFO Using Application-Default Credentials (ADC) to authenticate to GCP... 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Writing rule_binding views and intermediate summary results to BigQuery dq_summary_table_name: `qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.dq_summary`. 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Using dq_summary_dataset: qwiklabs-gcp-01-0110787d189f.orders_dq_dataset 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Using target_bigquery_summary_table: `qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results`. 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Preparing SQL for rule bindings: ['VALID_USER', 'VALID_ORDER'] 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.classes.dq_configs_cache[77] INFO Calling Dataplex Metadata API to retrieve schema for entity_uri: {'entity_uri': 'bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items', 'rule_binding_ids_list': 'VALID_ORDER,VALID_USER'} 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.classes.dq_configs_cache[77] INFO Retrieved Dataplex Entities: [DataplexEntity(name='projects/qwiklabs-gcp-01-0110787d189f/locations/us-west1/lakes/sales-lake/zones/curated-customer-zone/entities/ordered_items', createTime='2024-01-17T18:51:46.713702Z', updateTime='2024-01-17T18:51:46.713702Z', id='ordered_items', type='TABLE', asset='customer-orders', dataPath='projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items', system='BIGQUERY', format={'format': 'OTHER'}, schema=DataplexEntitySchema(fields=[{'name': 'delivered_at', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'shipped_at', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'created_at', 'type': 'TIMESTAMP', 'mode': 'NULLABLE'}, {'name': 'status', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'inventory_item_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'sale_price', 'type': 'DOUBLE', 'mode': 'NULLABLE'}, {'name': 'user_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'order_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'returned_at', 'type': 'TIMESTAMP', 'mode': 'NULLABLE'}, {'name': 'id', 'type': 'INT64', 'mode': 'NULLABLE'}], partitionFields=None, partitionStyle=None))] 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO target_entity_summary_configs: {'qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1': {'rule_binding_ids_list': ['VALID_ORDER']}, 'qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1': {'rule_binding_ids_list': ['VALID_USER']}} 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_utils[77] INFO Executing dbt command: ['run', '--profiles-dir', '/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/dbt', '--vars', '{"entity_dq_statistics_models": ["qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1", "qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1"]}', '--target', 'dev'] 19:56:24 Running with dbt=1.1.5 19:56:24 Partial parse save file not found. Starting full parse. 19:56:25 Found 6 models, 0 tests, 0 snapshots, 0 analyses, 191 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics 19:56:25 19:56:25 Concurrency: 8 threads (target='dev') 19:56:25 19:56:25 1 of 5 START view model orders_dq_dataset.VALID_ORDER .......................... [RUN] 19:56:25 2 of 5 START view model orders_dq_dataset.VALID_USER ........................... [RUN] 19:56:26 2 of 5 OK created view model orders_dq_dataset.VALID_USER ...................... [OK in 0.75s] 19:56:26 1 of 5 OK created view model orders_dq_dataset.VALID_ORDER ..................... [OK in 0.76s] 19:56:26 3 of 5 START table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1 [RUN] 19:56:26 4 of 5 START table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1 [RUN] 19:56:29 3 of 5 OK created table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1 [CREATE TABLE (1.0 rows, 7.8 KB processed) in 2.57s] 19:56:29 4 of 5 OK created table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1 [CREATE TABLE (1.0 rows, 7.8 KB processed) in 2.80s] 19:56:29 5 of 5 START incremental model orders_dq_dataset.dq_summary .................... [RUN] 19:56:32 5 of 5 OK created incremental model orders_dq_dataset.dq_summary ............... [MERGE (2.0 rows, 5.0 KB processed) in 2.60s] 19:56:32 19:56:32 Finished running 2 view models, 2 table models, 1 incremental model in 6.39s. 19:56:32 19:56:32 Completed successfully 19:56:32 19:56:32 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 19:56:42 Error sending message, disabling tracking 2024-01-17 19:56:42 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO dbt invocation id for current execution is 2550959a-af66-4b73-974d-2018cd72494c 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.bigquery.dq_target_table_utils[77] INFO Table qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results already exists and query results are appended to the table. 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.bigquery.dq_target_table_utils[77] INFO Loaded 2 rows to qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results. {"clouddq_job_completion_config": {"invocation_id": "2550959a-af66-4b73-974d-2018cd72494c", "target_bigquery_summary_table": "qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results", "summary_to_stdout": false, "target_rule_binding_ids": ["VALID_USER", "VALID_ORDER"], "partition_date": "2024-01-17", "num_rows_loaded_to_target_table": 2}} 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Job completed successfully. Waiting up to 5 seconds. Sent all pending logs.

Damir H. · Reviewed מעל שנה אחת ago

Pawel J. · Reviewed מעל שנה אחת ago

Mufti Alie S. · Reviewed מעל שנה אחת ago

Fernando F. · Reviewed מעל שנה אחת ago

George C. · Reviewed מעל שנה אחת ago

Jitendra S. · Reviewed מעל שנה אחת ago

My job succeeded but still i am getting below error message as "Rule named 'NOT NULL' not found for the user_id column in the data quality task. Please re-create the task including this rule as mentioned in the instructions."

Sivakumar P. · Reviewed מעל שנה אחת ago

shawn r. · Reviewed מעל שנה אחת ago

good

Sivakumar P. · Reviewed מעל שנה אחת ago

shawn r. · Reviewed מעל שנה אחת ago

good

Sivakumar P. · Reviewed מעל שנה אחת ago

good

Sivakumar P. · Reviewed מעל שנה אחת ago

Marcin K. · Reviewed מעל שנה אחת ago

The Data Quality exercise does not work. It is outdated (legacy) and should not be in here.

Peter H. · Reviewed מעל שנה אחת ago

Abey G. · Reviewed מעל שנה אחת ago

There seems to be a bug in the last part (Task 5). Data quality job succeeds, but the progress is not counted.

Julia W. · Reviewed מעל שנה אחת ago

Prakash V. · Reviewed מעל שנה אחת ago

The last exercise should give some kind of YAML file to start with or use AutoDQ, since the older version is mentioned as legacy.

Pia F. · Reviewed מעל שנה אחת ago

Kris H. · Reviewed מעל שנה אחת ago