Dataflow を使用したサーバーレスのデータ処理 - Apache Beam と Dataflow を使用して ETL パイプラインを作成する（Python） のレビュー

proper folders were not cloned for all the tasks

Shafi Shaik · 6日前にレビュー済み

Shaw Sriyansh · 6日前にレビュー済み

Shaw Sriyansh · 6日前にレビュー済み

Cezar Junior Luis Antonio · 6日前にレビュー済み

Sharma Jyoti · 6日前にレビュー済み

Cezar Junior Luis Antonio · 6日前にレビュー済み

Cezar Junior Luis Antonio · 6日前にレビュー済み

Could not get the pipeline to run due to multiple errors. Even the solution would not run. Very disappointing to be kicked out of the lab and lose progress before being able to resolve / troubleshoot. Errors were around: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)'))': /computeMetadata/v1/instance/service-accounts/default/?recursive=true and Compute Engine Medadata server unavailable.

De Meillon Mariette · 7日前にレビュー済み

too complex, needs more direction on what to do

Orccotoma Oscar · 8日前にレビュー済み

Longas Daniela · 8日前にレビュー済み

problema con la infrectuctura (falta de espacio)

Chiguay Gabriela · 8日前にレビュー済み

Cezar Junior Luis Antonio · 9日前にレビュー済み

Could not complete Part1 Task 6 run the pipeline (using the provided Solution code) due to the following error: raise BeamIOError("Match operation failed", exceptions) apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://qwiklabs-gcp-00-f5855126f119/events.json': RefreshError(TransportError("Failed to retrieve https://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Compute Engine Metadata server unavailable. Last exception: HTTPSConnectionPool(host='metadata.google.internal', port=443): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)')))"))}

Meng Lingmin · 9日前にレビュー済み

Qwiklabs Dataflow Lab Fix Summary Problems SSL/Metadata auth failure — google-auth 2.44.0+ broke the default HTTP transport, causing CERTIFICATE_VERIFY_FAILED errors when the notebook tried to reach GCP's metadata server Zone resource exhaustion — n1-standard-1 (Dataflow's default) was completely unavailable across all us-central1 zones for Qwiklabs accounts Pipeline exits early — p.run() doesn't wait for completion, causing silent failures argparse rejects extra flags — parse_args() blocks passing extra Beam arguments like --worker_machine_type Fixes 1. Fix SSL auth error bashexport GCE_METADATA_MTLS_MODE=none 2. Use e2-standard-2 machine type Add to your run command: bash--worker_machine_type=e2-standard-2 3. Fix pipeline code In my_pipeline.py: python# Change this: opts = parser.parse_args() options = PipelineOptions() p.run() # To this: opts, pipeline_args = parser.parse_known_args() options = PipelineOptions(pipeline_args) p.run().wait_until_finish() 4. Full working command bashcd $BASE_DIR export PROJECT_ID=$(gcloud config get-value project) export GCE_METADATA_MTLS_MODE=none python3 my_pipeline.py \ --project=${PROJECT_ID} \ --region=us-central1 \ --stagingLocation=gs://$PROJECT_ID/staging/ \ --tempLocation=gs://$PROJECT_ID/temp/ \ --runner=DataflowRunner \ --worker_machine_type=e2-standard-2 5. For the Dataflow Template UI Under Optional Parameters → uncheck "Use default machine type" → Series: E2 → Machine type: e2-standard-2

Olson David · 9日前にレビュー済み

console did not open

Muhammad Nihal Hussain · 9日前にレビュー済み

Fechete Mara Malina · 9日前にレビュー済み

VERY BAD>> I am unable to run my jobs are with DataflowRunner as I am always getting resources contraints errors.. job is not able to spn up us-cerntral1 region.. I am getting same error in all labs which requires to submit jobs on dataflow. I am able to run with DirectRunner. Please help in this as I have spent too many hours but end up getting same error again and again

Gurram Mallikarjunarao · 10日前にレビュー済み

couldnt finisih it lots of errors in the step by step or resources available

Petersen Sebastián · 10日前にレビュー済み

couldn't finish because dataflow job could get the resources to actually run. Stupid and a waste of my time. still can't run due to limited resources

Wilson Tyler · 10日前にレビュー済み

couldn't finish because dataflow job could get the resources to actually run. Stupid and a waste of my time.

Wilson Tyler · 10日前にレビュー済み

1. There are 2 issues running the lab: - lab 1: apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://qwiklabs-gcp-04-9d1f7241cd59/events.json': RefreshError(TransportError("Failed to retrieve https://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Compute Engine Metadata server unavailable. Last exception: HTTPSConnectionPool(host='metadata.google.internal', port=443): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate - lab 2 Startup of the worker pool in us-central1 failed to bring up any of the desired 1 workers P.S. Not working http.client transport only supports the http scheme, httpswas specified P.S.S. export GCE_METADATA_MTLS_MODE=none resolves an issue with certificate. But still insufficient resources to run the job.

Prudyvus Igor · 11日前にレビュー済み

Cezar Junior Luis Antonio · 11日前にレビュー済み

Meng Lingmin · 11日前にレビュー済み

Meng Lingmin · 11日前にレビュー済み

Poor instructions - terrible!

Coombs Leighton · 12日前にレビュー済み