Build a Data Mesh with Dataplex: Challenge Lab Reviews

Sahu S. · Reviewed أكثر من سنة ago

I have a rule named 'NOT NULL' , but task 5 does not recognize.

Allan S. · Reviewed أكثر من سنة ago

H C. · Reviewed أكثر من سنة ago

I have a rule named 'NOT NULL' , but task 5 does not recognize.

Allan S. · Reviewed أكثر من سنة ago

Mirosław G. · Reviewed أكثر من سنة ago

Mirosław G. · Reviewed أكثر من سنة ago

Vaishnavi V. · Reviewed أكثر من سنة ago

The last task is impossible to finish. metadata_registry_defaults: dataplex: projects: qwiklabs-gcp-01-0110787d189f locations: us-west1 lakes: sales-lake zones: curated-customer-zone row_filters: NONE: filter_sql_expr: |- True rule_dimensions: - consistency - correctness - duplication - completeness - conformance - integrity rules: NOT NULL: rule_type: NOT_NULL dimension: completeness rule_bindings: VALID_USER: entity_uri: bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items column_id: user_id row_filter_id: NONE rule_ids: - NOT NULL VALID_ORDER: entity_uri: bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items column_id: order_id row_filter_id: NONE rule_ids: - NOT NULL # Rule named 'NOT NULL' not found for the user_id column in the data quality task. # Please re-create the task including this rule as mentioned in the instructions. # Please create a data quality job named 'customer-orders-data-quality-job' as mentioned in the task. # If already done, wait until the 'results' table is created in 'orders_dq_dataset' dataset along with # data quality results. # Log Using the default container image Waiting for container log creation PYSPARK_PYTHON=/opt/dataproc/conda/bin/python JAVA_HOME=/usr/lib/jvm/temurin-11-jdk-amd64 SPARK_EXTRA_CLASSPATH= :: loading settings :: file = /etc/spark/conf/ivysettings.xml OS runtime details: PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" Python executable path: /opt/dataproc/conda/bin/python Python Version: sys.version_info(major=3, minor=9, micro=15, releaselevel='final', serial=0) PySpark working directory: PosixPath('/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491') PySpark directory content: ['.clouddq-executable.zip.hashsum.crc', 'clouddq-executable.zip', 'clouddq_pyspark_driver.py', '.dq-customer-orders.yaml.crc', '.clouddq_pyspark_driver.py.crc', 'clouddq-executable.zip.hashsum', 'dq-customer-orders.yaml', '.clouddq-executable.zip.crc'] Input PySpark arguments: ['/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/clouddq_pyspark_driver.py', 'clouddq-executable.zip', 'ALL', 'gs://qwiklabs-gcp-01-0110787d189f-dq-config/dq-customer-orders.yaml', '--gcp_project_id="qwiklabs-gcp-01-0110787d189f"', '--gcp_region_id="us-west1"', '--gcp_bq_dataset_id="orders_dq_dataset"', '--target_bigquery_summary_table="qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results"'] User-specified CloudDQ YAML configs: gs://qwiklabs-gcp-01-0110787d189f-dq-config/dq-customer-orders.yaml Creating configs directory at: `/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs` Copying YAML file dq-customer-orders.yaml to configs directory `configs`... Configs directory contents is: [PosixPath('configs/dq-customer-orders.yaml')] CloudDQ executable expected hexdigest: a14971737fa18e443390047a3d8b04a5378768bea0fed8a5bc34adabdba131c1 Executing commands: /opt/dataproc/conda/bin/python clouddq-executable.zip ALL /tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs --gcp_project_id="qwiklabs-gcp-01-0110787d189f" --gcp_region_id="us-west1" --gcp_bq_dataset_id="orders_dq_dataset" --target_bigquery_summary_table="qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results" 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.gcp_credentials[77] INFO Successfully created GCP Client. 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Starting CloudDQ run with configs: {"clouddq_run_configs": {"rule_binding_ids": "ALL", "rule_binding_config_path": "/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs", "environment_target": "dev", "gcp_project_id": "qwiklabs-gcp-01-0110787d189f", "gcp_region_id": "us-west1", "gcp_bq_dataset_id": "orders_dq_dataset", "gcp_service_account_key_path": null, "gcp_impersonation_credentials": null, "metadata": "{}", "dry_run": false, "progress_watermark": true, "target_bigquery_summary_table": "qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results", "intermediate_table_expiration_hours": 24, "num_threads": 8, "debug": false, "print_sql_queries": false, "skip_sql_validation": false, "summary_to_stdout": false, "enable_experimental_bigquery_entity_uris": true, "enable_experimental_dataplex_gcs_validation": true, "bigquery_client": null, "gcp_credentials": {"credentials": "<google.auth.compute_engine.credentials.Credentials object at 0x7f370cf1e280>", "project_id": "qwiklabs-gcp-01-0110787d189f", "user_id": "984150749013-compute@developer.gserviceaccount.com"}}} 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_connection_configs[77] INFO Using Application-Default Credentials (ADC) to authenticate to GCP... 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_connection_configs[77] INFO Using Application-Default Credentials (ADC) to authenticate to GCP... 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Writing rule_binding views and intermediate summary results to BigQuery dq_summary_table_name: `qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.dq_summary`. 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Using dq_summary_dataset: qwiklabs-gcp-01-0110787d189f.orders_dq_dataset 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Using target_bigquery_summary_table: `qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results`. 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Preparing SQL for rule bindings: ['VALID_USER', 'VALID_ORDER'] 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.classes.dq_configs_cache[77] INFO Calling Dataplex Metadata API to retrieve schema for entity_uri: {'entity_uri': 'bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items', 'rule_binding_ids_list': 'VALID_ORDER,VALID_USER'} 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.classes.dq_configs_cache[77] INFO Retrieved Dataplex Entities: [DataplexEntity(name='projects/qwiklabs-gcp-01-0110787d189f/locations/us-west1/lakes/sales-lake/zones/curated-customer-zone/entities/ordered_items', createTime='2024-01-17T18:51:46.713702Z', updateTime='2024-01-17T18:51:46.713702Z', id='ordered_items', type='TABLE', asset='customer-orders', dataPath='projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items', system='BIGQUERY', format={'format': 'OTHER'}, schema=DataplexEntitySchema(fields=[{'name': 'delivered_at', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'shipped_at', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'created_at', 'type': 'TIMESTAMP', 'mode': 'NULLABLE'}, {'name': 'status', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'inventory_item_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'sale_price', 'type': 'DOUBLE', 'mode': 'NULLABLE'}, {'name': 'user_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'order_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'returned_at', 'type': 'TIMESTAMP', 'mode': 'NULLABLE'}, {'name': 'id', 'type': 'INT64', 'mode': 'NULLABLE'}], partitionFields=None, partitionStyle=None))] 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO target_entity_summary_configs: {'qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1': {'rule_binding_ids_list': ['VALID_ORDER']}, 'qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1': {'rule_binding_ids_list': ['VALID_USER']}} 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_utils[77] INFO Executing dbt command: ['run', '--profiles-dir', '/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/dbt', '--vars', '{"entity_dq_statistics_models": ["qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1", "qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1"]}', '--target', 'dev'] 19:56:24 Running with dbt=1.1.5 19:56:24 Partial parse save file not found. Starting full parse. 19:56:25 Found 6 models, 0 tests, 0 snapshots, 0 analyses, 191 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics 19:56:25 19:56:25 Concurrency: 8 threads (target='dev') 19:56:25 19:56:25 1 of 5 START view model orders_dq_dataset.VALID_ORDER .......................... [RUN] 19:56:25 2 of 5 START view model orders_dq_dataset.VALID_USER ........................... [RUN] 19:56:26 2 of 5 OK created view model orders_dq_dataset.VALID_USER ...................... [OK in 0.75s] 19:56:26 1 of 5 OK created view model orders_dq_dataset.VALID_ORDER ..................... [OK in 0.76s] 19:56:26 3 of 5 START table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1 [RUN] 19:56:26 4 of 5 START table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1 [RUN] 19:56:29 3 of 5 OK created table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1 [CREATE TABLE (1.0 rows, 7.8 KB processed) in 2.57s] 19:56:29 4 of 5 OK created table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1 [CREATE TABLE (1.0 rows, 7.8 KB processed) in 2.80s] 19:56:29 5 of 5 START incremental model orders_dq_dataset.dq_summary .................... [RUN] 19:56:32 5 of 5 OK created incremental model orders_dq_dataset.dq_summary ............... [MERGE (2.0 rows, 5.0 KB processed) in 2.60s] 19:56:32 19:56:32 Finished running 2 view models, 2 table models, 1 incremental model in 6.39s. 19:56:32 19:56:32 Completed successfully 19:56:32 19:56:32 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 19:56:42 Error sending message, disabling tracking 2024-01-17 19:56:42 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO dbt invocation id for current execution is 2550959a-af66-4b73-974d-2018cd72494c 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.bigquery.dq_target_table_utils[77] INFO Table qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results already exists and query results are appended to the table. 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.bigquery.dq_target_table_utils[77] INFO Loaded 2 rows to qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results. {"clouddq_job_completion_config": {"invocation_id": "2550959a-af66-4b73-974d-2018cd72494c", "target_bigquery_summary_table": "qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results", "summary_to_stdout": false, "target_rule_binding_ids": ["VALID_USER", "VALID_ORDER"], "partition_date": "2024-01-17", "num_rows_loaded_to_target_table": 2}} 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Job completed successfully. Waiting up to 5 seconds. Sent all pending logs.

Damir H. · Reviewed أكثر من سنة ago

Pawel J. · Reviewed أكثر من سنة ago

Mufti Alie S. · Reviewed أكثر من سنة ago

Fernando F. · Reviewed أكثر من سنة ago

George C. · Reviewed أكثر من سنة ago

Jitendra S. · Reviewed أكثر من سنة ago

My job succeeded but still i am getting below error message as "Rule named 'NOT NULL' not found for the user_id column in the data quality task. Please re-create the task including this rule as mentioned in the instructions."

Sivakumar P. · Reviewed أكثر من سنة ago

shawn r. · Reviewed أكثر من سنة ago

good

Sivakumar P. · Reviewed أكثر من سنة ago

shawn r. · Reviewed أكثر من سنة ago

good

Sivakumar P. · Reviewed أكثر من سنة ago

good

Sivakumar P. · Reviewed أكثر من سنة ago

Marcin K. · Reviewed أكثر من سنة ago

The Data Quality exercise does not work. It is outdated (legacy) and should not be in here.

Peter H. · Reviewed أكثر من سنة ago

Abey G. · Reviewed أكثر من سنة ago

There seems to be a bug in the last part (Task 5). Data quality job succeeds, but the progress is not counted.

Julia W. · Reviewed أكثر من سنة ago

Prakash V. · Reviewed أكثر من سنة ago

The last exercise should give some kind of YAML file to start with or use AutoDQ, since the older version is mentioned as legacy.

Pia F. · Reviewed أكثر من سنة ago