Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow Reviews

Laksh J. · Reviewed 2 yıldan fazla ago

The ordering of the parameters in dataflow were not in order. Some of the parameters were in the additional options. I also received an error when trying to submit the dataflow job. It was an error relating to interval minutes not being supported. I had to pivot to using a gcloud command. This worked for me: gcloud dataflow jobs run streaming-taxi-pipeline --gcs-location gs://dataflow-templates-us-east1/latest/GCS_Text_to_BigQuery --region us-east1 --max-workers 2 --num-workers 1 --worker-machine-type e2-medium --staging-location gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp --parameters inputFilePattern=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp/rt_taxidata.csv,JSONPath=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp/schema.json,outputTable=qwiklabs-gcp-02-73dac1891e2c:taxirides.realtime,bigQueryLoadingTemporaryDirectory=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp,javascriptTextTransformGcsPath=gs://qwiklabs-gcp-02-73dac1891e2c-bucket/tmp/transform.js,javascriptTextTransformFunctionName=transform

Ryan L. · Reviewed 2 yıldan fazla ago

Peter T. · Reviewed 2 yıldan fazla ago

Juan C. · Reviewed 2 yıldan fazla ago

Cristina - Georgiana H. · Reviewed 2 yıldan fazla ago

rework tutorial, I'm getting errors

Winfried B. · Reviewed 2 yıldan fazla ago

Rahul T. · Reviewed 2 yıldan fazla ago

Satisfied

Saurabh M. · Reviewed 2 yıldan fazla ago

Satisfied

Saurabh M. · Reviewed 2 yıldan fazla ago

Peter G. · Reviewed 2 yıldan fazla ago

It improved when I tried loading data directly at the table creation from Cloud Storage instead of Data flow

Andres L. · Reviewed 2 yıldan fazla ago

when choosing the exact mentioned template, then it is a batch job rather than the streaming job, and even when choosing the streaming job called something like "Cloud Storage text to Bigquery" then some of the titles of the parts here in the description have other names than the labels of the input fields. (eg output table)

Melanie D. · Reviewed 2 yıldan fazla ago

Template can not work

Aya a N. · Reviewed 2 yıldan fazla ago

Nilupuli N. · Reviewed 2 yıldan fazla ago

Kasun H. · Reviewed 2 yıldan fazla ago

Tomasz S. · Reviewed 2 yıldan fazla ago

Samiksha G. · Reviewed 2 yıldan fazla ago

Bilgehan K. · Reviewed 2 yıldan fazla ago

nice

Mulpuru J. · Reviewed 2 yıldan fazla ago

Akash P. · Reviewed 2 yıldan fazla ago

Vera G. · Reviewed 2 yıldan fazla ago

Shrujan R. · Reviewed 2 yıldan fazla ago

Tanabut T. · Reviewed 2 yıldan fazla ago

Kristina Pia D. · Reviewed 2 yıldan fazla ago

Shashika K. · Reviewed 2 yıldan fazla ago