리뷰 Serverless Data Processing with Dataflow - Writing an ETL Pipeline using Apache Beam and Dataflow (Python)개

Timepass B. · 10개월 전에 리뷰됨

Laura Maria F. · 10개월 전에 리뷰됨

Krishna S. · 10개월 전에 리뷰됨

krishna s. · 10개월 전에 리뷰됨

riya N. · 10개월 전에 리뷰됨

Mystic T. · 10개월 전에 리뷰됨

Gehan G. · 10개월 전에 리뷰됨

the check progress wasn't working properly at part1 task6. even after having a complete run of the code it was still saying that it wasn't complete. (df-env) root@f39ba5596aca:~/training-data-analyst/quests/dataflow_python/1_Basic_ETL/lab# python3 my_pipeline.py \ > --project=${PROJECT_ID} \ > --region=us-central1 \ > --stagingLocation=gs://$PROJECT_ID/staging/ \ > --tempLocation=gs://$PROJECT_ID/temp/ \ > --runner=DirectRunner WARNING:apache_beam.transforms.core:('No iterator is returned by the process method in %s.', <class 'apache_beam.io.gcp.bigquery_file_loads.UpdateDestinationSchema'>) WARNING:apache_beam.transforms.core:('No iterator is returned by the process method in %s.', <class 'apache_beam.io.gcp.bigquery_file_loads.TriggerCopyJobs'>) WARNING:apache_beam.transforms.core:('No iterator is returned by the process method in %s.', <class 'apache_beam.io.gcp.bigquery_file_loads.DeleteTablesFn'>) INFO:root:Building pipeline ... WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--stagingLocation=gs://qwiklabs-gcp-00-126df7f76e70/staging/', '--tempLocation=gs://qwiklabs-gcp-00-126df7f76e70/temp/'] INFO:apache_beam.runners.worker.statecache:Creating state cache with size 104857600 INFO:apache_beam.io.gcp.bigquery_file_loads:Load job has 1 files. Job name is beam_bq_job_LOAD_mypipeline1737719492842404708_LOAD_STEP_7cc2c07304ba412f90134b5cce704f6c_267c29c1e3e84814d2d22447115ead5a_pane0_partition0. INFO:apache_beam.io.gcp.bigquery_file_loads:Triggering job beam_bq_job_LOAD_mypipeline1737719492842404708_LOAD_STEP_7cc2c07304ba412f90134b5cce704f6c_267c29c1e3e84814d2d22447115ead5a_pane0_partition0 to load data to BigQuery table <TableReference datasetId: 'logs' projectId: 'qwiklabs-gcp-00-126df7f76e70' tableId: 'logs'>.Schema: {'fields': [{'name': 'ip', 'type': 'STRING'}, {'name': 'user_id', 'type': 'STRING'}, {'name': 'lat', 'type': 'FLOAT'}, {'name': 'lng', 'type': 'FLOAT'}, {'name': 'timestamp', 'type': 'STRING'}, {'name': 'http_request', 'type': 'STRING'}, {'name': 'http_response', 'type': 'INTEGER'}, {'name': 'num_bytes', 'type': 'INTEGER'}, {'name': 'user_agent', 'type': 'STRING'}]}. Additional parameters: {}. Source format: NEWLINE_DELIMITED_JSON INFO:apache_beam.io.gcp.bigquery_tools:Started BigQuery job: <JobReference jobId: 'beam_bq_job_LOAD_mypipeline1737719492842404708_LOAD_STEP_7cc2c07304ba412f90134b5cce704f6c_267c29c1e3e84814d2d22447115ead5a_pane0_partition0' location: 'US' projectId: 'qwiklabs-gcp-00-126df7f76e70'> bq show -j --format=prettyjson --project_id=qwiklabs-gcp-00-126df7f76e70 beam_bq_job_LOAD_mypipeline1737719492842404708_LOAD_STEP_7cc2c07304ba412f90134b5cce704f6c_267c29c1e3e84814d2d22447115ead5a_pane0_partition0 INFO:apache_beam.io.gcp.bigquery_tools:Job qwiklabs-gcp-00-126df7f76e70:US.beam_bq_job_LOAD_mypipeline1737719492842404708_LOAD_STEP_7cc2c07304ba412f90134b5cce704f6c_267c29c1e3e84814d2d22447115ead5a_pane0_partition0 status: RUNNING INFO:apache_beam.io.gcp.bigquery_tools:Job qwiklabs-gcp-00-126df7f76e70:US.beam_bq_job_LOAD_mypipeline1737719492842404708_LOAD_STEP_7cc2c07304ba412f90134b5cce704f6c_267c29c1e3e84814d2d22447115ead5a_pane0_partition0 status: DONE

Edoardo G. · 10개월 전에 리뷰됨

Ali M. · 10개월 전에 리뷰됨

Arjun P. · 10개월 전에 리뷰됨

Novan N. · 10개월 전에 리뷰됨

Deepak J. · 10개월 전에 리뷰됨

Abhang M. · 10개월 전에 리뷰됨

umar k. · 10개월 전에 리뷰됨

this lab needs more time please

Raman J. · 10개월 전에 리뷰됨

Abhishek S. · 10개월 전에 리뷰됨

Poorvaj K. · 10개월 전에 리뷰됨

BAHRUL H. · 10개월 전에 리뷰됨

Trilochan B. · 10개월 전에 리뷰됨

good

Uday Mayank D. · 10개월 전에 리뷰됨

Dhanraj S. · 10개월 전에 리뷰됨

Rahil k. · 10개월 전에 리뷰됨

some fails with roles

GABRIEL R. · 10개월 전에 리뷰됨

Jie Yeng L. · 10개월 전에 리뷰됨

Abhijit G. · 10개월 전에 리뷰됨