[Deprecated] TFX Metadata Reviews

Prasuna M. · Reviewed أكثر من سنة ago

Roman T. · Reviewed أكثر من سنة ago

J H. · Reviewed أكثر من سنة ago

Hadrián P. · Reviewed أكثر من سنة ago

Scot J. · Reviewed أكثر من سنة ago

Ada A. · Reviewed أكثر من سنة ago

Yomna M. · Reviewed أكثر من سنة ago

Ezra J. · Reviewed أكثر من سنة ago

Vlad L. · Reviewed أكثر من سنة ago

2024-02-27 11:25:35.208831: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib 2024-02-27 11:25:35.208907: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. INFO:absl:tensorflow_ranking is not available: No module named 'tensorflow_ranking' INFO:absl:tensorflow_text is not available: No module named 'tensorflow_text' INFO:absl:Running driver for Tuner INFO:absl:MetadataStore with gRPC connection initialized INFO:absl:Adding KFP pod name tfx-covertype-lab-04-nxkkw-2692208594 to execution INFO:absl:Running executor for Tuner INFO:absl:Attempting to infer TFX Python dependency for beam INFO:absl:Copying all content from install dir /tfx-src/tfx to temp dir /tmp/tmp4c8gwt61/build/tfx INFO:absl:Generating a temp setup file at /tmp/tmp4c8gwt61/build/tfx/setup.py INFO:absl:Creating temporary sdist package, logs available at /tmp/tmp4c8gwt61/build/tfx/setup.log INFO:absl:Added --extra_package=/tmp/tmp4c8gwt61/build/tfx/dist/tfx_ephemeral-0.25.0.tar.gz to beam args WARNING:absl:workerCount is overridden with 2 INFO:absl:json_inputs='{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\"train\", \"eval\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}'. INFO:absl:json_outputs='{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}'. INFO:absl:json_exec_properties='{"custom_config": "{\"ai_platform_training_args\": {\"masterConfig\": {\"imageUri\": \"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\"}, \"project\": \"qwiklabs-gcp-01-272c395e4a02\", \"region\": \"us-central1\", \"serviceAccount\": \"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\"}}", "eval_args": "{\"num_steps\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\"num_steps\": 5000}", "tune_args": "{\n \"num_parallel_trials\": 3\n}", "tuner_fn": null}'. WARNING:googleapiclient.discovery_cache:file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect from google.appengine.api import memcache ModuleNotFoundError: No module named 'google.appengine' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in from oauth2client.contrib.locked_file import LockedFile ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in from oauth2client.locked_file import LockedFile ModuleNotFoundError: No module named 'oauth2client.locked_file' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/__init__.py", line 42, in autodetect from . import file_cache File "/usr/local/lib/python3.7/dist-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth" ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/ml/v1/rest INFO:absl:TrainingInput={'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\\"}, \\"project\\": \\"qwiklabs-gcp-01-272c395e4a02\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'region': 'us-central1', 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com', 'workerCount': 2, 'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard'} INFO:absl:Submitting job='tfx_tuner_20240227112545', project='qwiklabs-gcp-01-272c395e4a02' to AI Platform. INFO:googleapiclient.discovery:URL being requested: POST https://ml.googleapis.com/v1/projects/qwiklabs-gcp-01-272c395e4a02/jobs?alt=json INFO:googleapiclient.discovery:URL being requested: GET https://ml.googleapis.com/v1/projects/qwiklabs-gcp-01-272c395e4a02/jobs/tfx_tuner_20240227112545?alt=json ERROR:absl:Job 'projects/qwiklabs-gcp-01-272c395e4a02/jobs/tfx_tuner_20240227112545' did not succeed. Detailed response {'jobId': 'tfx_tuner_20240227112545', 'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard', 'workerCount': '2', 'region': 'us-central1', 'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\\"}, \\"project\\": \\"qwiklabs-gcp-01-272c395e4a02\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com'}, 'createTime': '2024-02-27T11:25:45Z', 'startTime': '2024-02-27T11:26:00Z', 'endTime': '2024-02-27T11:26:32Z', 'state': 'FAILED', 'errorMessage': 'The replica master 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 1 exited with a non-zero status of 1. Termination reason: Error. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=999903194638&resource=ml_job%2Fjob_id%2Ftfx_tuner_20240227112545&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22tfx_tuner_20240227112545%22', 'trainingOutput': {'consumedMLUnits': 0.01}, 'labels': {'tfx_executor': 'ensions-google_cloud_ai_platform-tuner-executor-_workerexecutor', 'tfx_py_version': '3-7', 'tfx_runner': 'kfp', 'tfx_version': '0-25-0'}, 'etag': 'e43TZkKQZ4E=', 'jobPosition': '0'}. Traceback (most recent call last): File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in main() File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main execution_info = launcher.launch() File "/tfx-src/tfx/orchestration/launcher/base_component_launcher.py", line 209, in launch copy.deepcopy(execution_decision.exec_properties)) File "/tfx-src/tfx/orchestration/launcher/in_process_component_launcher.py", line 72, in _run_executor copy.deepcopy(input_dict), output_dict, copy.deepcopy(exec_properties)) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/tuner/executor.py", line 121, in Do job_id) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/runner.py", line 305, in start_aip_training job_labels=job_labels) File "/tfx-src/tfx/extensions/google_cloud_ai_platform/runner.py", line 194, in _launch_aip_training raise RuntimeError(err_msg) RuntimeError: Job 'projects/qwiklabs-gcp-01-272c395e4a02/jobs/tfx_tuner_20240227112545' did not succeed. Detailed response {'jobId': 'tfx_tuner_20240227112545', 'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard', 'workerType': 'standard', 'workerCount': '2', 'region': 'us-central1', 'masterConfig': {'imageUri': 'gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647', 'containerCommand': ['python', '-m', 'tfx.scripts.run_executor', '--executor_class_path', 'tfx.extensions.google_cloud_ai_platform.tuner.executor._WorkerExecutor', '--inputs', '{"examples": [{"artifact": {"id": "21", "type_id": "18", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transformed_examples/21", "properties": {"split_names": {"string_value": "[\\"train\\", \\"eval\\"]"}}, "custom_properties": {"name": {"string_value": "transformed_examples"}, "producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}}, "state": "LIVE", "create_time_since_epoch": "1709033025920", "last_update_time_since_epoch": "1709033119881"}, "artifact_type": {"id": "18", "name": "Examples", "properties": {"split_names": "STRING", "span": "INT", "version": "INT"}}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "Examples"}], "transform_graph": [{"artifact": {"id": "20", "type_id": "22", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Transform/transform_graph/21", "custom_properties": {"producer_component": {"string_value": "Transform"}, "state": {"string_value": "published"}, "name": {"string_value": "transform_graph"}}, "state": "LIVE", "create_time_since_epoch": "1709033025917", "last_update_time_since_epoch": "1709033119878"}, "artifact_type": {"id": "22", "name": "TransformGraph"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "TransformGraph"}]}', '--outputs', '{"best_hyperparameters": [{"artifact": {"id": "25", "type_id": "28", "uri": "gs://qwiklabs-gcp-01-272c395e4a02-kubeflowpipelines-default/tfx_covertype_lab_04/b2da8988-ae20-4336-b3c8-6b91ae1315af/Tuner/best_hyperparameters/24", "custom_properties": {"name": {"string_value": "best_hyperparameters"}, "producer_component": {"string_value": "Tuner"}}}, "artifact_type": {"id": "28", "name": "HyperParameters"}, "__artifact_class_module__": "tfx.types.standard_artifacts", "__artifact_class_name__": "HyperParameters"}]}', '--exec-properties', '{"custom_config": "{\\"ai_platform_training_args\\": {\\"masterConfig\\": {\\"imageUri\\": \\"gcr.io/qwiklabs-gcp-01-272c395e4a02/tfx_covertype_lab_04@sha256:276d6670faaf7a4939cc0b991101cad0e92bb3ad870973a3018474337aaa1647\\"}, \\"project\\": \\"qwiklabs-gcp-01-272c395e4a02\\", \\"region\\": \\"us-central1\\", \\"serviceAccount\\": \\"tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com\\"}}", "eval_args": "{\\"num_steps\\": 500}", "kfp_pod_name": "tfx-covertype-lab-04-nxkkw-2692208594", "module_file": "model.py", "train_args": "{\\"num_steps\\": 5000}", "tune_args": "{\\n \\"num_parallel_trials\\": 3\\n}", "tuner_fn": null}']}, 'serviceAccount': 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-272c395e4a02.iam.gserviceaccount.com'}, 'createTime': '2024-02-27T11:25:45Z', 'startTime': '2024-02-27T11:26:00Z', 'endTime': '2024-02-27T11:26:32Z', 'state': 'FAILED', 'errorMessage': 'The replica master 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 0 exited with a non-zero status of 1. Termination reason: Error. The replica worker 1 exited with a non-zero status of 1. Termination reason: Error. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=999903194638&resource=ml_job%2Fjob_id%2Ftfx_tuner_20240227112545&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22tfx_tuner_20240227112545%22', 'trainingOutput': {'consumedMLUnits': 0.01}, 'labels': {'tfx_executor': 'ensions-google_cloud_ai_platform-tuner-executor-_workerexecutor', 'tfx_py_version': '3-7', 'tfx_runner': 'kfp', 'tfx_version': '0-25-0'}, 'etag': 'e43TZkKQZ4E=', 'jobPosition': '0'}.

Dmytro Y. · Reviewed أكثر من سنة ago

siong n. · Reviewed أكثر من سنة ago

VU T. · Reviewed أكثر من سنة ago

Szymon T. · Reviewed أكثر من سنة ago

The AI PLatform no longer exists and the setting up KubeFlow pipeline instance is not an option anymore. This causes confusion while doing the labs in this course, they should be updated to reflect the new flow. It is demotivating and makes the learning experience difficult.

Vipul R. · Reviewed ما يقرب من سنتين ago

Rafee M. · Reviewed ما يقرب من سنتين ago

雄介 藤. · Reviewed ما يقرب من سنتين ago

Job failed - but didnt return an error message

Cameron G. · Reviewed ما يقرب من سنتين ago

Shubham A. · Reviewed ما يقرب من سنتين ago

Mohammad S. · Reviewed ما يقرب من سنتين ago

CHRISTIAN ALEJANDRO S. · Reviewed ما يقرب من سنتين ago

Francisco V. · Reviewed ما يقرب من سنتين ago

Binaya Kumar P. · Reviewed ما يقرب من سنتين ago

Alexander B. · Reviewed ما يقرب من سنتين ago

CHRISTIAN ALEJANDRO S. · Reviewed ما يقرب من سنتين ago

Christoph C. · Reviewed ما يقرب من سنتين ago