Serverless Data Processing with Dataflow - CI/CD with Dataflow Reviews

Vishnu M. · Reviewed 12 شهر ago

Riccardo M. · Reviewed حوالي سنة ago

too many errors making it impossible to complete in time. for instance, the image in the code does not exist. the DAG keeps failing, and more. I've now tried it two times, and it is impossible to complete.

Trym S. · Reviewed حوالي سنة ago

NA

Nenavath R. · Reviewed حوالي سنة ago

wordcount-run fails with error: [2024-12-15, 21:15:32 UTC] {beam.py:119} WARNING - WARNING: Request failed with code 400, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://dataflow.googleapis.com/v1b3/projects/qwiklabs-gcp-01-caa5705744db/locations/us-central1/jobs. [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - Exception in thread "main" java.lang.RuntimeException: Failed to create a workflow job: (3c8e5c18af78a2a): 'us-central1' violates constraint 'constraints/gcp.resourceLocations' on the resource 'projects/qwiklabs-gcp-01-caa5705744db'. [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:972) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:186) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:187) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.examples.WordCount.main(WordCount.java:193) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - { [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "code" : 400, [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "errors" : [ { [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "domain" : "global", [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "message" : "(3c8e5c18af78a2a): 'us-central1' violates constraint 'constraints/gcp.resourceLocations' on the resource 'projects/qwiklabs-gcp-01-caa5705744db'.", [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "reason" : "failedPrecondition" [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - } ], [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "message" : "(3c8e5c18af78a2a): 'us-central1' violates constraint 'constraints/gcp.resourceLocations' on the resource 'projects/qwiklabs-gcp-01-caa5705744db'.", [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "status" : "FAILED_PRECONDITION" [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - } [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:417) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1132) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:515) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:448) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowClient.createJob(DataflowClient.java:61) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:958)

Marcelo B. · Reviewed حوالي سنة ago

Riccardo M. · Reviewed حوالي سنة ago

Sana K. · Reviewed حوالي سنة ago

NA

Nenavath R. · Reviewed حوالي سنة ago

Steven B. · Reviewed حوالي سنة ago

This had 2 blockers. The 2nd killed my 1st attempt. 1. Airflow version non-existent anymore, had to pick one from https://cloud.google.com/composer/docs/concepts/versioning/composer-versions instead. 2. (stage: watching Airflow DAGs) The default regions setup blows up during the first task. Says constraints/gcp.resourceLocations is violated, because f--k us-central1. The student can't adjust those permissions because they have no editor rights.

Anna E. · Reviewed حوالي سنة ago

Kamala Krishnan S. · Reviewed حوالي سنة ago

I've tried more than a dozen times. It always fails on the DAG run.

Flavio B. · Reviewed حوالي سنة ago

Luke M. · Reviewed حوالي سنة ago

Monica C. · Reviewed حوالي سنة ago

Simone G. · Reviewed حوالي سنة ago

Cecilia Belen B. · Reviewed حوالي سنة ago

Simone G. · Reviewed حوالي سنة ago

Lots of commands not working on this lab! test_work_count DAG keeps failing. I'm assuming because of incorrect region which should be asia-east1 instead of us-central1. I made sure I run all the commands to set up the region and zone variables to asia-east1 but not sure why region is incorrect in the jar file. Running command: java -jar /tmp/tmpjnvtv59_dataflow_deployment_106c05a3-bb57-41d1-b2ee-386581a6109b.jar --runner=DataflowRunner --project=qwiklabs-gcp-00-436813f80158 --zone=asia-east1-a --region=us-central1 --stagingLocation=gs://qwiklabs-gcp-00-436813f80158-dataflow-staging-test/staging --jobName=wordcount-run-081da930 --autoscalingAlgorithm=THROUGHPUT_BASED --maxNumWorkers=3 --inputFile=gs://qwiklabs-gcp-00-436813f80158-composer-input-test/input.txt --output=gs://qwiklabs-gcp-00-436813f80158-composer-result-test/output '--labels={"airflow-version":"v2-10-2-composer"}'

Oscar F. · Reviewed حوالي سنة ago

Puneet P. · Reviewed حوالي سنة ago

end

Karthik C. · Reviewed حوالي سنة ago

Kevin A. · Reviewed حوالي سنة ago

Ashwani Kumar S. · Reviewed حوالي سنة ago

Jesus C. · Reviewed حوالي سنة ago

Karthik C. · Reviewed حوالي سنة ago

Alok P. · Reviewed حوالي سنة ago