关于“Serverless Data Processing with Dataflow - Writing an ETL Pipeline using Apache Beam and Dataflow (Python)”的评价

Akash R. · 评论over 3 years之前

Abhishek P. · 评论over 3 years之前

Runtime error using the provided solution, wasnt able to advance the lab (base) jupyter@theia-20221017-211349:~/project/training-data-analyst/quests/dataflow_python/1_Basic_ETL/lab$ python3 my_pipeline.py --project=${PROJECT_ID} --region=us-central1 --stagingLocation=gs://$PROJECT_ID/staging/ --tempLocation=gs://$PROJECT_ID/temp/ --runner=DataflowRunner /opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py:2285: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported is_streaming_pipeline = p.options.view_as(StandardOptions).streaming /opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_file_loads.py:1129: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location INFO:root:Building pipeline ... INFO:apache_beam.runners.portability.stager:Downloading source distribution of the SDK from PyPi INFO:apache_beam.runners.portability.stager:Executing command: ['/opt/conda/bin/python3', '-m', 'pip', 'download', '--dest', '/tmp/tmpw1vdo9yu', 'apache-beam==2.42.0', '--no-deps', '--no-binary', ':all:'] ERROR: Ignored the following versions that require a different python version: 2.10.0 Requires-Python >=2.7,<3.0; 2.3.0 Requires-Python >=2.7,<3.0; 2.4.0 Requires-Python >=2.7,<3.0; 2.5.0 Requires-Python >=2.7,<3.0; 2.6.0 Requires-Python >=2.7,<3.0; 2.7.0 Requires-Python >=2.7,<3.0; 2.8.0 Requires-Python >=2.7,<3.0; 2.9.0 Requires-Python >=2.7,<3.0 ERROR: Could not find a version that satisfies the requirement apache-beam==2.42.0 (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0, 2.11.0, 2.12.0, 2.13.0, 2.14.0, 2.15.0, 2.16.0, 2.17.0, 2.18.0, 2.19.0, 2.20.0, 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.26.0, 2.27.0, 2.28.0, 2.29.0rc1, 2.29.0, 2.30.0rc1, 2.30.0, 2.31.0rc1, 2.31.0, 2.32.0rc1, 2.32.0, 2.33.0rc1, 2.33.0rc2, 2.33.0, 2.34.0rc1, 2.34.0rc2, 2.34.0, 2.35.0rc2, 2.35.0rc3, 2.35.0rc5, 2.35.0rc6, 2.35.0rc8, 2.35.0, 2.36.0rc1, 2.36.0rc2, 2.36.0rc3, 2.36.0, 2.37.0rc1, 2.37.0rc2, 2.37.0rc3, 2.37.0, 2.38.0rc1, 2.38.0, 2.39.0rc1, 2.39.0rc2, 2.39.0, 2.40.0rc1, 2.40.0rc2, 2.40.0, 2.41.0rc1, 2.41.0rc2, 2.41.0, 2.42.0rc0, 2.42.0rc1, 2.42.0rc2) ERROR: No matching distribution found for apache-beam==2.42.0 Traceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 89, in check_output out = subprocess.check_output(*args, **kwargs) File "/opt/conda/lib/python3.7/subprocess.py", line 411, in check_output **kwargs).stdout File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'pip', 'download', '--dest', '/tmp/tmpw1vdo9yu', 'apache-beam==2.42.0', '--no-deps', '--no-binary', ':all:']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "my_pipeline.py", line 106, in <module> run() File "my_pipeline.py", line 103, in run p.run() File "/opt/conda/lib/python3.7/site-packages/apache_beam/pipeline.py", line 550, in run self._options).run(False) File "/opt/conda/lib/python3.7/site-packages/apache_beam/pipeline.py", line 574, in run return self.runner.run_pipeline(self, self._options) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 475, in run_pipeline artifacts = environments.python_sdk_dependencies(options) File "/opt/conda/lib/python3.7/site-packages/apache_beam/transforms/environments.py", line 806, in python_sdk_dependencies skip_prestaged_dependencies=skip_prestaged_dependencies) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 310, in create_job_resources Stager._create_beam_sdk(sdk_remote_location, temp_dir)) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 822, in _create_beam_sdk sdk_local_file = Stager._download_pypi_sdk_package(temp_dir) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 936, in _download_pypi_sdk_package processes.check_output(cmd_args) File "/opt/conda/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 97, in check_output .format(traceback.format_exc(), args[0][6], error.output)) RuntimeError: Full traceback: Traceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 89, in check_output out = subprocess.check_output(*args, **kwargs) File "/opt/conda/lib/python3.7/subprocess.py", line 411, in check_output **kwargs).stdout File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'pip', 'download', '--dest', '/tmp/tmpw1vdo9yu', 'apache-beam==2.42.0', '--no-deps', '--no-binary', ':all:']' returned non-zero exit status 1. Pip install failed for package: apache-beam==2.42.0 Output from execution of subprocess: b''

Juan M. · 评论over 3 years之前

John D. · 评论over 3 years之前

Ricardo Q. · 评论over 3 years之前

Eric T. · 评论over 3 years之前

Marcos T. · 评论over 3 years之前

Mark C. · 评论over 3 years之前

Andrei B. · 评论over 3 years之前

Antonio C. · 评论over 3 years之前

Couganadane V. · 评论over 3 years之前

Gadamsetty S. · 评论over 3 years之前

Nikolaus M. · 评论over 3 years之前

Constructive lab

Ali M. · 评论over 3 years之前

Tamer K. · 评论over 3 years之前

Prajjwal S. · 评论over 3 years之前

Until now, this is the best explain lab I ever did in cloudskillboots. Have time tags before tasks, have at least double of required time if you only follow step by step the lab. Give you the opportunity to try yourself an the solution code is an extra non a step...

Ricardo Miguel N. · 评论over 3 years之前

Prashant K. · 评论over 3 years之前

jairo m. · 评论over 3 years之前

khadija b. · 评论over 3 years之前

Prajjwal S. · 评论over 3 years之前

Noé G. · 评论over 3 years之前

pratik B. · 评论over 3 years之前

Deshen C. · 评论over 3 years之前

Kyrylo K. · 评论over 3 years之前