[Deprecated] TFX Metadata Reviews

Prasuna M. · Reviewed over 1 year ago

Roman T. · Reviewed over 1 year ago

J H. · Reviewed over 1 year ago

Hadrián P. · Reviewed over 1 year ago

Scot J. · Reviewed over 1 year ago

Ada A. · Reviewed over 1 year ago

Yomna M. · Reviewed over 1 year ago

Ezra J. · Reviewed over 1 year ago

Vlad L. · Reviewed over 1 year ago

2024-02-27 11:25:35.208831: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib 2024-02-27 11:25:35.208907: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. INFO:absl:tensorflow_ranking is not available: No module named 'tensorflow_ranking' INFO:absl:tensorflow_text is not available: No module named 'tensorflow_text' INFO:absl:Running driver for Tuner INFO:absl:MetadataStore with gRPC connection initialized INFO:absl:Adding KFP pod name tfx-covertype-lab-04-nxkkw-2692208594 to execution INFO:absl:Running executor for Tuner INFO:absl:Attempting to infer TFX Python dependency for beam INFO:absl:Copying all content from install dir /tfx-src/tfx to temp dir /tmp/tmp4c8gwt61/build/tfx INFO:absl:Generating a temp setup file at /tmp/tmp4c8gwt61/build/tfx/setup.py INFO:absl:Creating temporary sdist package, logs available at /tmp/tmp4c8gwt61/build/tfx/setup.log INFO:absl:Added --extra_package=/tmp/tmp4c8gwt61/build/tfx/dist/tfx_ephemeral-0.25.0.tar.gz to beam args WARNING:absl:workerCount is overridden with 2 INFO:absl:json_inputs='{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\"train\", \"eval\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}'. INFO:absl:json_outputs='{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}'. INFO:absl:json_exec_properties='{"custom_config": "{\"ai_platform_training_args\": {\"masterConfig\": {\"imageUri\": \"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\"}, \"project\": \"qwiklabs-gcp-01-272c395e4a02\", \"region\": \"us-central1\", \"serviceAccount\": \"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\"}}", "eval_args": "{\"num_steps\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\"num_steps\": 5000}", "tune_args": "{\n \"num_parallel_trials\": 3\n}", "tuner_fn": null}'. WARNING:googleapiclient.discovery_cache:file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect from google.appengine.api import memcache ModuleNotFoundError: No module named 'google.appengine' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in from oauth2client.contrib.locked_file import LockedFile ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in from oauth2client.locked_file import LockedFile ModuleNotFoundError: No module named 'oauth2client.locked_file' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/__init__.py", line 42, in autodetect from . import file_cache File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth" ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/ml/v1/rest INFO:absl:TrainingInput={'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\\"}, \\"project\\": \\"qwiklabs-gcp-01-272c395e4a02\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'region': 'us-central1', 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com', 'workerCount': 2, 'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard'} INFO:absl:Submitting job='tfx_tuner_20240227112545', project='qwiklabs-gcp-01-272c395e4a02' to AI Platform. INFO:googleapiclient.discovery:URL being requested: POST https://ml.googleapis.com/v1/projects/qwiklabs-gcp-01-272c395e4a02/jobs?alt=json INFO:googleapiclient.discovery:URL being requested: GET https://ml.googleapis.com/v1/projects/qwiklabs-gcp-01-272c395e4a02/jobs/tfx_tuner_20240227112545?alt=json ERROR:absl:Job 'projects/qwiklabs-gcp-01-272c395e4a02/jobs/tfx_tuner_20240227112545' did not succeed. Detailed response {'jobId': 'tfx_tuner_20240227112545', 'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard', 'workerCount': '2', 'region': 'us-central1', 'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\\"}, \\"project\\": \\"qwiklabs-gcp-01-272c395e4a02\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com'}, 'createTime': '2024-02-27T11:25:45Z', 'startTime': '2024-02-27T11:26:00Z', 'endTime': '2024-02-27T11:26:32Z', 'state': 'FAILED', 'errorMessage': 'The replica master 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 1 exited with a non-zero status of 1. Termination reason: Error. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=999903194638&resource=ml_job%2Fjob_id%2Ftfx_tuner_20240227112545&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22tfx_tuner_20240227112545%22', 'trainingOutput': {'consumedMLUnits': 0.01}, 'labels': {'tfx_executor': 'ensions-google_cloud_ai_platform-tuner-executor-_workerexecutor', 'tfx_py_version': '3-7', 'tfx_runner': 'kfp', 'tfx_version': '0-25-0'}, 'etag': 'e43TZkKQZ4E=', 'jobPosition': '0'}. Traceback (most recent call last): File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in main() File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main execution_info = launcher.launch() File "/tfx-src/tfx/orchestration/launcher/base_component_launcher.py", line 209, in launch copy.deepcopy(execution_decision.exec_properties)) File "/tfx-src/tfx/orchestration/launcher/in_process_component_launcher.py", line 72, in _run_executor copy.deepcopy(input_dict), output_dict, copy.deepcopy(exec_properties)) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/tuner/executor.py", line 121, in Do job_id) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/runner.py", line 305, in start_aip_training job_labels=job_labels) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/runner.py", line 194, in _launch_aip_training raise RuntimeError(err_msg) RuntimeError: Job 'projects/qwiklabs-gcp-01-272c395e4a02/jobs/tfx_tuner_20240227112545' did not succeed. Detailed response {'jobId': 'tfx_tuner_20240227112545', 'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard', 'workerCount': '2', 'region': 'us-central1', 'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\\"}, \\"project\\": \\"qwiklabs-gcp-01-272c395e4a02\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com'}, 'createTime': '2024-02-27T11:25:45Z', 'startTime': '2024-02-27T11:26:00Z', 'endTime': '2024-02-27T11:26:32Z', 'state': 'FAILED', 'errorMessage': 'The replica master 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 1 exited with a non-zero status of 1. Termination reason: Error. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=999903194638&resource=ml_job%2Fjob_id%2Ftfx_tuner_20240227112545&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22tfx_tuner_20240227112545%22', 'trainingOutput': {'consumedMLUnits': 0.01}, 'labels': {'tfx_executor': 'ensions-google_cloud_ai_platform-tuner-executor-_workerexecutor', 'tfx_py_version': '3-7', 'tfx_runner': 'kfp', 'tfx_version': '0-25-0'}, 'etag': 'e43TZkKQZ4E=', 'jobPosition': '0'}.

Dmytro Y. · Reviewed over 1 year ago

siong n. · Reviewed over 1 year ago

VU T. · Reviewed over 1 year ago

Szymon T. · Reviewed over 1 year ago

The AI PLatform no longer exists and the setting up KubeFlow pipeline instance is not an option anymore. This causes confusion while doing the labs in this course, they should be updated to reflect the new flow. It is demotivating and makes the learning experience difficult.

Vipul R. · Reviewed over 1 year ago

Rafee M. · Reviewed over 1 year ago

雄介 藤. · Reviewed over 1 year ago

Job failed - but didnt return an error message

Cameron G. · Reviewed over 1 year ago

Shubham A. · Reviewed almost 2 years ago

Mohammad S. · Reviewed almost 2 years ago

CHRISTIAN ALEJANDRO S. · Reviewed almost 2 years ago

Francisco V. · Reviewed almost 2 years ago

Binaya Kumar P. · Reviewed almost 2 years ago

Alexander B. · Reviewed almost 2 years ago

CHRISTIAN ALEJANDRO S. · Reviewed almost 2 years ago

Christoph C. · Reviewed almost 2 years ago