Serverless Data Processing with Dataflow - CI/CD with Dataflow Ulasan

needs time to study it but interesting complete CICD

Andreas K. · Diulas 6 bulan lalu

Vishnu M. · Diulas 6 bulan lalu

Riccardo M. · Diulas 6 bulan lalu

too many errors making it impossible to complete in time. for instance, the image in the code does not exist. the DAG keeps failing, and more. I've now tried it two times, and it is impossible to complete.

Trym S. · Diulas 6 bulan lalu

NA

Nenavath R. · Diulas 6 bulan lalu

wordcount-run fails with error: [2024-12-15, 21:15:32 UTC] {beam.py:119} WARNING - WARNING: Request failed with code 400, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://dataflow.googleapis.com/v1b3/projects/qwiklabs-gcp-01-caa5705744db/locations/us-central1/jobs. [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - Exception in thread "main" java.lang.RuntimeException: Failed to create a workflow job: (3c8e5c18af78a2a): 'us-central1' violates constraint 'constraints/gcp.resourceLocations' on the resource 'projects/qwiklabs-gcp-01-caa5705744db'. [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:972) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:186) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:187) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.examples.WordCount.main(WordCount.java:193) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - { [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "code" : 400, [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "errors" : [ { [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "domain" : "global", [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "message" : "(3c8e5c18af78a2a): 'us-central1' violates constraint 'constraints/gcp.resourceLocations' on the resource 'projects/qwiklabs-gcp-01-caa5705744db'.", [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "reason" : "failedPrecondition" [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - } ], [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "message" : "(3c8e5c18af78a2a): 'us-central1' violates constraint 'constraints/gcp.resourceLocations' on the resource 'projects/qwiklabs-gcp-01-caa5705744db'.", [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - "status" : "FAILED_PRECONDITION" [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - } [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:417) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1132) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:515) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:448) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowClient.createJob(DataflowClient.java:61) [2024-12-15, 21:15:33 UTC] {beam.py:119} WARNING - at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:958)

Marcelo B. · Diulas 6 bulan lalu

Riccardo M. · Diulas 6 bulan lalu

Sana K. · Diulas 6 bulan lalu

NA

Nenavath R. · Diulas 6 bulan lalu

Steven B. · Diulas 6 bulan lalu

This had 2 blockers. The 2nd killed my 1st attempt. 1. Airflow version non-existent anymore, had to pick one from https://cloud.google.com/composer/docs/concepts/versioning/composer-versions instead. 2. (stage: watching Airflow DAGs) The default regions setup blows up during the first task. Says constraints/gcp.resourceLocations is violated, because f--k us-central1. The student can't adjust those permissions because they have no editor rights.

Anna E. · Diulas 6 bulan lalu

Kamala Krishnan S. · Diulas 6 bulan lalu

I've tried more than a dozen times. It always fails on the DAG run.

Flavio B. · Diulas 6 bulan lalu

Luke M. · Diulas 6 bulan lalu

Monica C. · Diulas 6 bulan lalu

Simone G. · Diulas 6 bulan lalu

Cecilia Belen B. · Diulas 6 bulan lalu

Simone G. · Diulas 6 bulan lalu

Lots of commands not working on this lab! test_work_count DAG keeps failing. I'm assuming because of incorrect region which should be asia-east1 instead of us-central1. I made sure I run all the commands to set up the region and zone variables to asia-east1 but not sure why region is incorrect in the jar file. Running command: java -jar /tmp/tmpjnvtv59_dataflow_deployment_106c05a3-bb57-41d1-b2ee-386581a6109b.jar --runner=DataflowRunner --project=qwiklabs-gcp-00-436813f80158 --zone=asia-east1-a --region=us-central1 --stagingLocation=gs://qwiklabs-gcp-00-436813f80158-dataflow-staging-test/staging --jobName=wordcount-run-081da930 --autoscalingAlgorithm=THROUGHPUT_BASED --maxNumWorkers=3 --inputFile=gs://qwiklabs-gcp-00-436813f80158-composer-input-test/input.txt --output=gs://qwiklabs-gcp-00-436813f80158-composer-result-test/output '--labels={"airflow-version":"v2-10-2-composer"}'

Oscar F. · Diulas 6 bulan lalu

Puneet P. · Diulas 6 bulan lalu

end

Karthik C. · Diulas 6 bulan lalu

Kevin A. · Diulas 6 bulan lalu

Ashwani Kumar S. · Diulas 6 bulan lalu

Jesus C. · Diulas 6 bulan lalu

Karthik C. · Diulas 6 bulan lalu