[DEPRECATED] TFX on Cloud AI Platform Pipelines Reviews

André F. · Reviewed about 5 years ago

anirban d. · Reviewed about 5 years ago

Encountered an error at the "csvexamplegen" step during the experiment runs (both with and without hyperparameter tuning). Here is the error message: 2021-03-26 02:27:50.327318: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib 2021-03-26 02:27:50.327398: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. WARNING: You are using pip version 20.2; however, version 21.0.1 is available. You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command. WARNING: You are using pip version 20.2; however, version 21.0.1 is available. You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command. INFO:absl:tensorflow_ranking is not available: No module named 'tensorflow_ranking' INFO:absl:tensorflow_text is not available: No module named 'tensorflow_text' INFO:absl:Running driver for CsvExampleGen INFO:absl:MetadataStore with gRPC connection initialized INFO:absl:select span and version = (0, None) INFO:absl:latest span and version = (0, None) INFO:absl:Adding KFP pod name tfx-covertype-continuous-training-5mfgd-3614928810 to execution INFO:absl:Running executor for CsvExampleGen INFO:absl:Attempting to infer TFX Python dependency for beam INFO:absl:Copying all content from install dir /tfx-src/tfx to temp dir /tmp/tmpac0a6x09/build/tfx INFO:absl:Generating a temp setup file at /tmp/tmpac0a6x09/build/tfx/setup.py INFO:absl:Creating temporary sdist package, logs available at /tmp/tmpac0a6x09/build/tfx/setup.log INFO:absl:Added --extra_package=/tmp/tmpac0a6x09/build/tfx/dist/tfx_ephemeral-0.25.0.tar.gz to beam args INFO:absl:Generating examples. INFO:absl:Processing input csv data gs://workshop-datasets/covertype/small/* to TFExample. INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds. INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds. INFO:apache_beam.io.gcp.gcsio:Starting the size estimation of the input INFO:oauth2client.transport:Attempting refresh to obtain initial access_token INFO:apache_beam.io.gcp.gcsio:Finished listing 1 files in 0.25779294967651367 seconds. INFO:apache_beam.runners.portability.stager:Downloading source distribution of the SDK from PyPi INFO:apache_beam.runners.portability.stager:Executing command: ['/usr/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/tmpl3a3dtn5', 'apache-beam==2.25.0', '--no-deps', '--no-binary', ':all:'] INFO:apache_beam.runners.portability.stager:Staging SDK sources from PyPI: dataflow_python_sdk.tar INFO:apache_beam.runners.portability.stager:Downloading binary distribution of the SDK from PyPi INFO:apache_beam.runners.portability.stager:Executing command: ['/usr/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/tmpl3a3dtn5', 'apache-beam==2.25.0', '--no-deps', '--only-binary', ':all:', '--python-version', '37', '--implementation', 'cp', '--abi', 'cp37m', '--platform', 'manylinux1_x86_64'] INFO:apache_beam.runners.portability.stager:Staging binary distribution of the SDK from PyPI: apache_beam-2.25.0-cp37-cp37m-manylinux1_x86_64.whl WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0. If the image is not available at local, we will try to pull from hub.docker.com INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp INFO:apache_beam.io.gcp.gcsio:Starting the size estimation of the input INFO:apache_beam.io.gcp.gcsio:Finished listing 1 files in 0.05882406234741211 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/pipeline.pb... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/pipeline.pb in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/tfx_ephemeral-0.25.0.tar.gz... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/tfx_ephemeral-0.25.0.tar.gz in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/extra_packages.txt... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/extra_packages.txt in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/dataflow_python_sdk.tar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/dataflow_python_sdk.tar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/apache_beam-2.25.0-cp37-cp37m-manylinux1_x86_64.whl... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://qwiklabs-gcp-02-88b39fa5888f-kubeflowpipelines-default//beam/tmp/beamapp-root-0326022819-066746.1616725699.067150/apache_beam-2.25.0-cp37-cp37m-manylinux1_x86_64.whl in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job createTime: '2021-03-26T02:28:24.597191Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2021-03-25_19_28_21-4707912746545961513' location: 'us-central1' name: 'beamapp-root-0326022819-066746' projectId: 'qwiklabs-gcp-02-88b39fa5888f' stageStates: [] startTime: '2021-03-26T02:28:24.597191Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-03-25_19_28_21-4707912746545961513] INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-03-25_19_28_21-4707912746545961513 INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-03-25_19_28_21-4707912746545961513?project=qwiklabs-gcp-02-88b39fa5888f INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-03-25_19_28_21-4707912746545961513 is in state JOB_STATE_PENDING INFO:apache_beam.runners.dataflow.dataflow_runner:2021-03-26T02:28:26.863Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2021-03-25_19_28_21-4707912746545961513. The number of workers will be between 1 and 1000. INFO:apache_beam.runners.dataflow.dataflow_runner:2021-03-26T02:28:26.994Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2021-03-25_19_28_21-4707912746545961513. INFO:apache_beam.runners.dataflow.dataflow_runner:2021-03-26T02:28:27.948Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: There was a problem refreshing your credentials. Please check: 1. Dataflow API is enabled for your project. 2. Make sure both the Dataflow service account and the controller service account have sufficient permissions. If you are not specifying a controller service account, ensure the default Compute Engine service account [PROJECT_NUMBER]-compute@developer.gserviceaccount.com exists and has sufficient permissions. If you have deleted the default Compute Engine service account, you must specify a controller service account. For more information, see: https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#security_and_permissions_for_pipelines_on_google_cloud_platform. , There is no cloudservices robot account for your project. Please ensure that the Dataflow API is enabled for your project. INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-03-25_19_28_21-4707912746545961513 is in state JOB_STATE_FAILED Traceback (most recent call last): File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in <module> main() File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main execution_info = launcher.launch() File "/tfx-src/tfx/orchestration/launcher/base_component_launcher.py", line 209, in launch copy.deepcopy(execution_decision.exec_properties)) File "/tfx-src/tfx/orchestration/launcher/in_process_component_launcher.py", line 72, in _run_executor copy.deepcopy(input_dict), output_dict, copy.deepcopy(exec_properties)) File "/tfx-src/tfx/components/example_gen/base_example_gen_executor.py", line 300, in Do split_name))) File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 569, in __exit__ self.result.wait_until_finish() File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1650, in wait_until_finish self) apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: Workflow failed. Causes: There was a problem refreshing your credentials. Please check: 1. Dataflow API is enabled for your project. 2. Make sure both the Dataflow service account and the controller service account have sufficient permissions. If you are not specifying a controller service account, ensure the default Compute Engine service account [PROJECT_NUMBER]-compute@developer.gserviceaccount.com exists and has sufficient permissions. If you have deleted the default Compute Engine service account, you must specify a controller service account. For more information, see: https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#security_and_permissions_for_pipelines_on_google_cloud_platform. , There is no cloudservices robot account for your project. Please ensure that the Dataflow API is enabled for your project.

Maosi C. · Reviewed about 5 years ago

Nels L. · Reviewed about 5 years ago

Robin A. · Reviewed about 5 years ago

Rakesh J. · Reviewed about 5 years ago

gianpiero p. · Reviewed about 5 years ago

gianpiero p. · Reviewed about 5 years ago

Ana D. · Reviewed about 5 years ago

ERROR when trying to deploy the pipeline !!! kfp_server_api.exceptions.ApiException: (401) Reason: Unauthorized

COSTA C. · Reviewed about 5 years ago

Great

Suresh K. · Reviewed about 5 years ago

Claudio I. · Reviewed about 5 years ago

Dani R. · Reviewed about 5 years ago

Dani R. · Reviewed about 5 years ago

Bill K. · Reviewed about 5 years ago

Henrik W. · Reviewed about 5 years ago

Yeseul Y. · Reviewed about 5 years ago

Stuart M. · Reviewed about 5 years ago

When I want to create a new instance in the pipeline, the system said "You do not have permission to view this project "

Ying X. · Reviewed about 5 years ago

Ivan N. · Reviewed about 5 years ago

Johan B. · Reviewed about 5 years ago

Oussama B. · Reviewed about 5 years ago

Esra D. · Reviewed about 5 years ago

Médéric H. · Reviewed about 5 years ago

Oussama B. · Reviewed about 5 years ago