Serverless Data Processing with Dataflow - Writing an ETL Pipeline using Apache Beam and Dataflow (Python) Reviews

Akash R. · Reviewed больше 3 лет ago

Abhishek P. · Reviewed больше 3 лет ago

Runtime error using the provided solution, wasnt able to advance the lab (base) jupyter@theia-20221017-211349:~/project/training-data-analyst/quests/dataflow_python/1_Basic_ETL/lab$ python3 my_pipeline.py --project=${PROJECT_ID} --region=us-central1 --stagingLocation=gs://$PROJECT_ID/staging/ --tempLocation=gs://$PROJECT_ID/temp/ --runner=DataflowRunner /opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py:2285: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported is_streaming_pipeline = p.options.view_as(StandardOptions).streaming /opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_file_loads.py:1129: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location INFO:root:Building pipeline ... INFO:apache_beam.runners.portability.stager:Downloading source distribution of the SDK from PyPi INFO:apache_beam.runners.portability.stager:Executing command: ['/opt/conda/bin/python3', '-m', 'pip', 'download', '--dest', '/tmp/tmpw1vdo9yu', 'apache-beam==2.42.0', '--no-deps', '--no-binary', ':all:'] ERROR: Ignored the following versions that require a different python version: 2.10.0 Requires-Python >=2.7,<3.0; 2.3.0 Requires-Python >=2.7,<3.0; 2.4.0 Requires-Python >=2.7,<3.0; 2.5.0 Requires-Python >=2.7,<3.0; 2.6.0 Requires-Python >=2.7,<3.0; 2.7.0 Requires-Python >=2.7,<3.0; 2.8.0 Requires-Python >=2.7,<3.0; 2.9.0 Requires-Python >=2.7,<3.0 ERROR: Could not find a version that satisfies the requirement apache-beam==2.42.0 (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0, 2.11.0, 2.12.0, 2.13.0, 2.14.0, 2.15.0, 2.16.0, 2.17.0, 2.18.0, 2.19.0, 2.20.0, 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.26.0, 2.27.0, 2.28.0, 2.29.0rc1, 2.29.0, 2.30.0rc1, 2.30.0, 2.31.0rc1, 2.31.0, 2.32.0rc1, 2.32.0, 2.33.0rc1, 2.33.0rc2, 2.33.0, 2.34.0rc1, 2.34.0rc2, 2.34.0, 2.35.0rc2, 2.35.0rc3, 2.35.0rc5, 2.35.0rc6, 2.35.0rc8, 2.35.0, 2.36.0rc1, 2.36.0rc2, 2.36.0rc3, 2.36.0, 2.37.0rc1, 2.37.0rc2, 2.37.0rc3, 2.37.0, 2.38.0rc1, 2.38.0, 2.39.0rc1, 2.39.0rc2, 2.39.0, 2.40.0rc1, 2.40.0rc2, 2.40.0, 2.41.0rc1, 2.41.0rc2, 2.41.0, 2.42.0rc0, 2.42.0rc1, 2.42.0rc2) ERROR: No matching distribution found for apache-beam==2.42.0 Traceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 89, in check_output out = subprocess.check_output(*args, **kwargs) File "/opt/conda/lib/python3.7/subprocess.py", line 411, in check_output **kwargs).stdout File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'pip', 'download', '--dest', '/tmp/tmpw1vdo9yu', 'apache-beam==2.42.0', '--no-deps', '--no-binary', ':all:']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "my_pipeline.py", line 106, in <module> run() File "my_pipeline.py", line 103, in run p.run() File "/opt/conda/lib/python3.7/site-packages/apache_beam/pipeline.py", line 550, in run self._options).run(False) File "/opt/conda/lib/python3.7/site-packages/apache_beam/pipeline.py", line 574, in run return self.runner.run_pipeline(self, self._options) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 475, in run_pipeline artifacts = environments.python_sdk_dependencies(options) File "/opt/conda/lib/python3.7/site-packages/apache_beam/transforms/environments.py", line 806, in python_sdk_dependencies skip_prestaged_dependencies=skip_prestaged_dependencies) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 310, in create_job_resources Stager._create_beam_sdk(sdk_remote_location, temp_dir)) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 822, in _create_beam_sdk sdk_local_file = Stager._download_pypi_sdk_package(temp_dir) File "/opt/conda/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 936, in _download_pypi_sdk_package processes.check_output(cmd_args) File "/opt/conda/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 97, in check_output .format(traceback.format_exc(), args[0][6], error.output)) RuntimeError: Full traceback: Traceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 89, in check_output out = subprocess.check_output(*args, **kwargs) File "/opt/conda/lib/python3.7/subprocess.py", line 411, in check_output **kwargs).stdout File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'pip', 'download', '--dest', '/tmp/tmpw1vdo9yu', 'apache-beam==2.42.0', '--no-deps', '--no-binary', ':all:']' returned non-zero exit status 1. Pip install failed for package: apache-beam==2.42.0 Output from execution of subprocess: b''

Juan M. · Reviewed больше 3 лет ago

John D. · Reviewed больше 3 лет ago

Ricardo Q. · Reviewed больше 3 лет ago

Eric T. · Reviewed больше 3 лет ago

Marcos T. · Reviewed больше 3 лет ago

Mark C. · Reviewed больше 3 лет ago

Andrei B. · Reviewed больше 3 лет ago

Antonio C. · Reviewed больше 3 лет ago

Couganadane V. · Reviewed больше 3 лет ago

Gadamsetty S. · Reviewed больше 3 лет ago

Nikolaus M. · Reviewed больше 3 лет ago

Constructive lab

Ali M. · Reviewed больше 3 лет ago

Tamer K. · Reviewed больше 3 лет ago

Prajjwal S. · Reviewed больше 3 лет ago

Until now, this is the best explain lab I ever did in cloudskillboots. Have time tags before tasks, have at least double of required time if you only follow step by step the lab. Give you the opportunity to try yourself an the solution code is an extra non a step...

Ricardo Miguel N. · Reviewed больше 3 лет ago

Prashant K. · Reviewed больше 3 лет ago

jairo m. · Reviewed больше 3 лет ago

khadija b. · Reviewed больше 3 лет ago

Prajjwal S. · Reviewed больше 3 лет ago

Noé G. · Reviewed больше 3 лет ago

pratik B. · Reviewed больше 3 лет ago

Deshen C. · Reviewed больше 3 лет ago

Kyrylo K. · Reviewed больше 3 лет ago