Dataflow を使用したサーバーレスのデータ処理 - カスタム Dataflow Flex テンプレート（Python） のレビュー

Ramshetti Pratibha · 4ヶ月前にレビュー済み

Gradinar Gabriel · 5ヶ月前にレビュー済み

Bennett Stuart · 5ヶ月前にレビュー済み

Gradinar Gabriel · 5ヶ月前にレビュー済み

Comotti Marina Esther · 5ヶ月前にレビュー済み

Fernandez Duque Angel · 5ヶ月前にレビュー済み

Task 2 step 3 fails (df-env) root@4e25fad68d16:~/training-data-analyst/quests/dataflow_python/2_Branching_Pipelines/solution# gcloud builds submit --tag $TEMPLATE_IMAGE . Creating temporary archive of 19068 file(s) totalling 606.0 MiB before compression. Uploading tarball of [.] to [gs://qwiklabs-gcp-03-4a4cb84fea93_cloudbuild/source/1738589652.547012-c3eeaf9f4d1f48eb944ed9108b1b147d.tgz] Created [https://cloudbuild.googleapis.com/v1/projects/qwiklabs-gcp-03-4a4cb84fea93/locations/global/builds/92bf7730-fc25-4092-b84f-363bd12b3b8e]. Logs are available at [ https://console.cloud.google.com/cloud-build/builds/92bf7730-fc25-4092-b84f-363bd12b3b8e?project=892080016347 ]. Waiting for build to complete. Polling interval: 1 second(s). ----------------------------------------------------------------------------------------------------- REMOTE BUILD OUTPUT ----------------------------------------------------------------------------------------------------- starting build "92bf7730-fc25-4092-b84f-363bd12b3b8e" FETCHSOURCE Fetching storage object: gs://qwiklabs-gcp-03-4a4cb84fea93_cloudbuild/source/1738589652.547012-c3eeaf9f4d1f48eb944ed9108b1b147d.tgz#1738589852677131 Copying gs://qwiklabs-gcp-03-4a4cb84fea93_cloudbuild/source/1738589652.547012-c3eeaf9f4d1f48eb944ed9108b1b147d.tgz#1738589852677131... \ [1 files][139.6 MiB/139.6 MiB] Operation completed over 1 objects/139.6 MiB. BUILD Pulling image: gcr.io/kaniko-project/executor:latest latest: Pulling from kaniko-project/executor Digest: sha256:9e69fd4330ec887829c780f5126dd80edc663df6def362cd22e79bcdf00ac53f Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest gcr.io/kaniko-project/executor:latest INFO[0010] Retrieving image manifest gcr.io/dataflow-templates-base/python3-template-launcher-base INFO[0010] Retrieving image gcr.io/dataflow-templates-base/python3-template-launcher-base from registry gcr.io INFO[0011] Built cross stage deps: map[] INFO[0011] Retrieving image manifest gcr.io/dataflow-templates-base/python3-template-launcher-base INFO[0011] Returning cached image manifest INFO[0011] Executing 0 build triggers INFO[0011] Building stage 'gcr.io/dataflow-templates-base/python3-template-launcher-base' [idx: '0', base-idx: '-1'] INFO[0011] Checking for cached layer gcr.io/qwiklabs-gcp-03-4a4cb84fea93/dataflow/my_pipeline/cache:bf47e0ff7f937494af455dd44bf310fe52ac6353a968476e51615c4fe4b0bb53... INFO[0011] No cached layer found for cmd RUN mkdir -p ${WORKDIR} INFO[0011] Unpacking rootfs as cmd RUN mkdir -p ${WORKDIR} requires it. INFO[0076] ARG WORKDIR=/dataflow/template INFO[0076] No files changed in this command, skipping snapshotting. INFO[0076] RUN mkdir -p ${WORKDIR} INFO[0076] Initializing snapshotter ... INFO[0076] Taking snapshot of full filesystem... INFO[0094] Cmd: /bin/sh INFO[0094] Args: [-c mkdir -p ${WORKDIR}] INFO[0094] Running: [/bin/sh -c mkdir -p ${WORKDIR}] INFO[0094] Taking snapshot of full filesystem... INFO[0101] WORKDIR ${WORKDIR} INFO[0101] Cmd: workdir INFO[0101] Changed working directory to /dataflow/template INFO[0101] Pushing layer gcr.io/qwiklabs-gcp-03-4a4cb84fea93/dataflow/my_pipeline/cache:bf47e0ff7f937494af455dd44bf310fe52ac6353a968476e51615c4fe4b0bb53 to cache now INFO[0101] Pushing image to gcr.io/qwiklabs-gcp-03-4a4cb84fea93/dataflow/my_pipeline/cache:bf47e0ff7f937494af455dd44bf310fe52ac6353a968476e51615c4fe4b0bb53 INFO[0101] No files changed in this command, skipping snapshotting. INFO[0101] RUN apt-get update && apt-get install -y libffi-dev && rm -rf /var/lib/apt/lists/* INFO[0101] Cmd: /bin/sh INFO[0101] Args: [-c apt-get update && apt-get install -y libffi-dev && rm -rf /var/lib/apt/lists/*] INFO[0101] Running: [/bin/sh -c apt-get update && apt-get install -y libffi-dev && rm -rf /var/lib/apt/lists/*] Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB] Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB] Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB] Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8792 kB] Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.5 kB] Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [243 kB] INFO[0103] Pushed gcr.io/qwiklabs-gcp-03-4a4cb84fea93/dataflow/my_pipeline/cache@sha256:bfc985d61e5c5f2ff34f51e295f6a58b5e94f4e710e9606113c77c168b285ce0 Fetched 9303 kB in 2s (5537 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... libffi-dev is already the newest version (3.4.4-1). 0 upgraded, 0 newly installed, 0 to remove and 33 not upgraded. INFO[0105] Taking snapshot of full filesystem... INFO[0111] COPY my_pipeline.py . INFO[0111] Pushing layer gcr.io/qwiklabs-gcp-03-4a4cb84fea93/dataflow/my_pipeline/cache:75c5483ec971adf327e4ff143a1a50490922398eb12ede4a2947816cd2e6ac16 to cache now INFO[0111] Pushing image to gcr.io/qwiklabs-gcp-03-4a4cb84fea93/dataflow/my_pipeline/cache:75c5483ec971adf327e4ff143a1a50490922398eb12ede4a2947816cd2e6ac16 INFO[0112] Taking snapshot of files... INFO[0112] ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/my_pipeline.py" INFO[0112] No files changed in this command, skipping snapshotting. INFO[0112] RUN python3 -m pip install apache-beam[gcp]==2.25.0 INFO[0112] Cmd: /bin/sh INFO[0112] Args: [-c python3 -m pip install apache-beam[gcp]==2.25.0] INFO[0112] Running: [/bin/sh -c python3 -m pip install apache-beam[gcp]==2.25.0] Collecting apache-beam==2.25.0 (from apache-beam[gcp]==2.25.0) Downloading apache-beam-2.25.0.zip (2.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 91.5 MB/s eta 0:00:00 Preparing metadata (setup.py): started INFO[0113] Pushed gcr.io/qwiklabs-gcp-03-4a4cb84fea93/dataflow/my_pipeline/cache@sha256:ef69116b07a20a0dc10a11c70a265a6c27ac4cf508eb91c8b9a18a5ac0f4bfac Preparing metadata (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [60 lines of output] /tmp/pip-install-4k1o28ff/apache-beam_54f377bb4c4a468fa2109fa577cd29ea/setup.py:32: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html from pkg_resources import DistributionNotFound /tmp/pip-install-4k1o28ff/apache-beam_54f377bb4c4a468fa2109fa577cd29ea/setup.py:40: SetuptoolsDeprecationWarning: The test command is disabled and references to it are deprecated. !! ******************************************************************************** Please remove any references to `setuptools.command.test` in all supported versions of the affected package. This deprecation is overdue, please update your project and remove deprecated calls to avoid build errors in the future. ******************************************************************************** !! from setuptools.command.test import test /tmp/pip-install-4k1o28ff/apache-beam_54f377bb4c4a468fa2109fa577cd29ea/setup.py:101: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if StrictVersion(_PIP_VERSION) < StrictVersion(REQUIRED_PIP_VERSION): /tmp/pip-install-4k1o28ff/apache-beam_54f377bb4c4a468fa2109fa577cd29ea/setup.py:113: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if StrictVersion(_CYTHON_VERSION) < StrictVersion(REQUIRED_CYTHON_VERSION): /tmp/pip-install-4k1o28ff/apache-beam_54f377bb4c4a468fa2109fa577cd29ea/setup.py:273: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.9. You may encounter bugs or missing features. warnings.warn( /usr/local/lib/python3.9/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-4k1o28ff/apache-beam_54f377bb4c4a468fa2109fa577cd29ea/apache_beam/coders/coder_impl.pxd tree = Parsing.p_module(s, pxd, full_module_name) Error compiling Cython file: ------------------------------------------------------------ ... if False: # pylint: disable=using-constant-test # This clause is interpreted by the compiler. from cython import compiled as is_compiled else: is_compiled = False ^ ------------------------------------------------------------ apache_beam/coders/coder_impl.py:86:4: Cannot assign to or delete this Traceback (most recent call last): File "<string>", line 2, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "/tmp/pip-install-4k1o28ff/apache-beam_54f377bb4c4a468fa2109fa577cd29ea/setup.py", line 291, in <module> ext_modules=cythonize([ File "/usr/local/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize cythonize_one(*args) File "/usr/local/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one raise CompileError(None, pyx_file) Cython.Compiler.Errors.CompileError: apache_beam/coders/coder_impl.py Compiling apache_beam/coders/stream.pyx because it changed. Compiling apache_beam/runners/worker/statesampler_fast.pyx because it changed. Compiling apache_beam/transforms/cy_dataflow_distribution_counter.pyx because it changed. Compiling apache_beam/coders/coder_impl.py because it changed. Compiling apache_beam/metrics/cells.py because it changed. Compiling apache_beam/metrics/execution.py because it changed. Compiling apache_beam/runners/common.py because it changed. Compiling apache_beam/runners/worker/logger.py because it changed. Compiling apache_beam/runners/worker/opcounters.py because it changed. Compiling apache_beam/runners/worker/operations.py because it changed. Compiling apache_beam/transforms/cy_combiners.py because it changed. Compiling apache_beam/utils/counters.py because it changed. Compiling apache_beam/utils/windowed_value.py because it changed. [ 1/13] Cythonizing apache_beam/coders/coder_impl.py [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. [notice] A new release of pip is available: 24.3.1 -> 25.0 [notice] To update, run: pip install --upgrade pip error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1 ERROR ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- BUILD FAILURE: Build step failure: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 1 ERROR: (gcloud.builds.submit) build 92bf7730-fc25-4092-b84f-363bd12b3b8e completed with status "FAILURE"

Bennett Stuart · 5ヶ月前にレビュー済み

Richard Jayaseelan George · 5ヶ月前にレビュー済み

good

Kale Sameer · 5ヶ月前にレビュー済み

Gupta mamta · 5ヶ月前にレビュー済み

there are some errors which prohibit the completion of this lab

Irvin Steven · 5ヶ月前にレビュー済み

Overall good lab, but initially I could not create the BigQuery table from the Dataflow template. It turns out that there was an IAM issue, as after I gave my developer account the BigQuery admin role was I able to create the table.

mohanty pranav · 5ヶ月前にレビュー済み

R Karthick · 5ヶ月前にレビュー済み

Joshi Bhavya · 5ヶ月前にレビュー済み

Jonnalagedda Rajeev Gautham · 5ヶ月前にレビュー済み

Konaje Prashant · 5ヶ月前にレビュー済み

De la Torre Juan · 5ヶ月前にレビュー済み

Sounderraj N Vishnu · 5ヶ月前にレビュー済み

Kottapalli Bhavya Sai Sree · 5ヶ月前にレビュー済み

Rajendran Ranjithkumar · 5ヶ月前にレビュー済み

Pooja Gurugubelli · 5ヶ月前にレビュー済み

Rachamalla Tharun Reddy · 5ヶ月前にレビュー済み

Anquetil Thomas · 5ヶ月前にレビュー済み

Tripathi Shikha · 5ヶ月前にレビュー済み

Ennouri Hind · 5ヶ月前にレビュー済み