Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow Reviews

Laksh J. · Reviewed מעל 2 שנים ago

The ordering of the parameters in dataflow were not in order. Some of the parameters were in the additional options. I also received an error when trying to submit the dataflow job. It was an error relating to interval minutes not being supported. I had to pivot to using a gcloud command. This worked for me: gcloud dataflow jobs run streaming-taxi-pipeline --gcs-location gs://dataflow-templates-us-east1/latest/GCS_Text_to_BigQuery --region us-east1 --max-workers 2 --num-workers 1 --worker-machine-type e2-medium --staging-location gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp --parameters inputFilePattern=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp/rt_taxidata.csv,JSONPath=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp/schema.json,outputTable=qwiklabs-gcp-02-73dac1891e2c:taxirides.realtime,bigQueryLoadingTemporaryDirectory=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp,javascriptTextTransformGcsPath=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp/transform.js,javascriptTextTransformFunctionName=transform

Ryan L. · Reviewed מעל 2 שנים ago

Peter T. · Reviewed מעל 2 שנים ago

Juan C. · Reviewed מעל 2 שנים ago

Cristina - Georgiana H. · Reviewed מעל 2 שנים ago

rework tutorial, I'm getting errors

Winfried B. · Reviewed מעל 2 שנים ago

Rahul T. · Reviewed מעל 2 שנים ago

Satisfied

Saurabh M. · Reviewed מעל 2 שנים ago

Satisfied

Saurabh M. · Reviewed מעל 2 שנים ago

Peter G. · Reviewed מעל 2 שנים ago

It improved when I tried loading data directly at the table creation from Cloud Storage instead of Data flow

Andres L. · Reviewed מעל 2 שנים ago

when choosing the exact mentioned template, then it is a batch job rather than the streaming job, and even when choosing the streaming job called something like "Cloud Storage text to Bigquery" then some of the titles of the parts here in the description have other names than the labels of the input fields. (eg output table)

Melanie D. · Reviewed מעל 2 שנים ago

Template can not work

Aya a N. · Reviewed מעל 2 שנים ago

Nilupuli N. · Reviewed מעל 2 שנים ago

Kasun H. · Reviewed מעל 2 שנים ago

Tomasz S. · Reviewed מעל 2 שנים ago

Samiksha G. · Reviewed מעל 2 שנים ago

Bilgehan K. · Reviewed מעל 2 שנים ago

nice

Mulpuru J. · Reviewed מעל 2 שנים ago

Akash P. · Reviewed מעל 2 שנים ago

Vera G. · Reviewed מעל 2 שנים ago

Shrujan R. · Reviewed מעל 2 שנים ago

Tanabut T. · Reviewed מעל 2 שנים ago

Kristina Pia D. · Reviewed מעל 2 שנים ago

Shashika K. · Reviewed מעל 2 שנים ago