在 Google Cloud 上使用 Dataflow 和 BigQuery 處理 ETL 作業 (Python) Reviews

Favin F. · Reviewed almost 6 years ago

HIROYUKI T. · Reviewed almost 6 years ago

Keerthana B. · Reviewed almost 6 years ago

K. R. · Reviewed almost 6 years ago

Sameer Anand J. · Reviewed almost 6 years ago

Did not work

Rishi A. · Reviewed almost 6 years ago

Urja J. · Reviewed almost 6 years ago

Hi team. The final task seems to require an older Beam SDK==2.23.0 to work. Error message using Beam==2.25.0: ValueError: Initializing SDFBoundedSourceRestrictionTracker requires a _SDFBoundedSourceRestriction [while running 'Read Account Details from BigQuery /ReadFromBigQuery/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)']

Bing Yuan L. · Reviewed almost 6 years ago

The last one did not finish (tried a few times): INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-10-23_18_48_55-6903357488097196600 is in state JOB_STATE_FAILED Traceback (most recent call last): File "dataflow_python_examples/data_lake_to_mart.py", line 306, in <module> run() File "dataflow_python_examples/data_lake_to_mart.py", line 301, in run p.run().wait_until_finish() File "/home/student_02_aec7c56c687d/dataflow-python-examples/venv/lib/python3.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1650, in wait_until_finish self) apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent call last): File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 720, in apache_beam.runners.common.PerWindowInvoker.invoke_process File "apache_beam/runners/common.py", line 752, in apache_beam.runners.common.PerWindowInvoker._should_process_window_for_sdf File "apache_beam/runners/common.py", line 547, in apache_beam.runners.common.DoFnInvoker.invoke_create_tracker File "/usr/local/lib/python3.7/site-packages/apache_beam/io/iobase.py", line 1492, in create_tracker restriction) File "/usr/local/lib/python3.7/site-packages/apache_beam/io/iobase.py", line 1443, in __init__ 'Initializing SDFBoundedSourceRestrictionTracker' ValueError: Initializing SDFBoundedSourceRestrictionTracker requires a _SDFBoundedSourceRestriction During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 649, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 179, in execute op.start() File "dataflow_worker/native_operations.py", line 38, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 39, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 44, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 54, in dataflow_worker.native_operations.NativeReadOperation.start File "apache_beam/runners/worker/operations.py", line 356, in apache_beam.runners.worker.operations.Operation.output File "apache_beam/runners/worker/operations.py", line 218, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 703, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/worker/operations.py", line 704, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/common.py", line 1215, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented

Lee Z. · Reviewed almost 6 years ago

Haardik G. · Reviewed almost 6 years ago

Shubham S. · Reviewed almost 6 years ago

Kumar Adarsh P. · Reviewed almost 6 years ago

Hernán M. · Reviewed almost 6 years ago

Visvanathan A. · Reviewed almost 6 years ago

saaya d. · Reviewed almost 6 years ago

MOHIT KUMAR M. · Reviewed almost 6 years ago

Mayank G. · Reviewed almost 6 years ago

Tarunika R. · Reviewed almost 6 years ago

Sumana B. · Reviewed almost 6 years ago

Divya M. · Reviewed almost 6 years ago

Diana E. · Reviewed almost 6 years ago

Jainam S. · Reviewed almost 6 years ago

very lengthy lab

Abhijeet p. · Reviewed almost 6 years ago

Thavapalan A. · Reviewed almost 6 years ago

nice.

jigar d. · Reviewed almost 6 years ago