Serverless Data Processing with Dataflow - CI/CD with Dataflow Ulasan

Ketaki K. · Diulas lebih dari 1 tahun lalu

TASK:- 2-4 gcloud composer environments create data-pipeline-composer \ --location us-central1 \ --image-version composer-1.20.7-airflow-1.10.15 cd ~/ci-cd-for-data-processing-workflow/env-setup chmod +x set_composer_variables.sh ./set_composer_variables.sh export COMPOSER_DAG_BUCKET=$(gcloud composer environments describe $COMPOSER_ENV_NAME \ --location $COMPOSER_REGION \ --format="get(config.dagGcsPrefix)") export COMPOSER_SERVICE_ACCOUNT=$(gcloud composer environments describe $COMPOSER_ENV_NAME \ --location $COMPOSER_REGION \ --format="get(config.nodeConfig.serviceAccount)") cd ~/ci-cd-for-data-processing-workflow/env-setup chmod +x create_buckets.sh ./create_buckets.sh gcloud source repos create $SOURCE_CODE_REPO cp -r ~/ci-cd-for-data-processing-workflow/source-code ~/$SOURCE_CODE_REPO cd ~/$SOURCE_CODE_REPO git config --global credential.'https://source.developers.google.com'.helper gcloud.sh git config --global user.email $(gcloud config list --format 'value(core.account)') git config --global user.name $(gcloud config list --format 'value(core.account)') git init git remote add google \ https://source.developers.google.com/p/$GCP_PROJECT_ID/r/$SOURCE_CODE_REPO git add . git commit -m 'initial commit' git push google master gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \ --member=serviceAccount:$PROJECT_NUMBER@cloudbuild.gserviceaccount.com \ --role=roles/composer.admin gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \ --member=serviceAccount:$PROJECT_NUMBER@cloudbuild.gserviceaccount.com \ --role=roles/composer.worker TASK 5:- cd ~/ci-cd-for-data-processing-workflow/source-code/build-pipeline gcloud builds submit --config=build_deploy_test.yaml --substitutions=\ REPO_NAME=$SOURCE_CODE_REPO,\ _DATAFLOW_JAR_BUCKET=$DATAFLOW_JAR_BUCKET_TEST,\ _COMPOSER_INPUT_BUCKET=$INPUT_BUCKET_TEST,\ _COMPOSER_REF_BUCKET=$REF_BUCKET_TEST,\ _COMPOSER_DAG_BUCKET=$COMPOSER_DAG_BUCKET,\ _COMPOSER_ENV_NAME=$COMPOSER_ENV_NAME,\ _COMPOSER_REGION=$COMPOSER_REGION,\ _COMPOSER_DAG_NAME_TEST=$COMPOSER_DAG_NAME_TEST gsutil ls gs://$DATAFLOW_JAR_BUCKET_TEST/dataflow_deployment*.jar gcloud composer environments describe $COMPOSER_ENV_NAME \ --location $COMPOSER_REGION \ --format="get(config.airflowUri)" TASK :- Create the production pipeline export DATAFLOW_JAR_FILE_LATEST=$(gcloud composer environments run $COMPOSER_ENV_NAME \ --location $COMPOSER_REGION variables -- \ --get dataflow_jar_file_test 2>&1 | grep -i '.jar') cd ~/ci-cd-for-data-processing-workflow/source-code/build-pipeline gcloud builds submit --config=deploy_prod.yaml --substitutions=\ REPO_NAME=$SOURCE_CODE_REPO,\ _DATAFLOW_JAR_BUCKET_TEST=$DATAFLOW_JAR_BUCKET_TEST,\ _DATAFLOW_JAR_FILE_LATEST=$DATAFLOW_JAR_FILE_LATEST,\ _DATAFLOW_JAR_BUCKET_PROD=$DATAFLOW_JAR_BUCKET_PROD,\ _COMPOSER_INPUT_BUCKET=$INPUT_BUCKET_PROD,\ _COMPOSER_ENV_NAME=$COMPOSER_ENV_NAME,\ _COMPOSER_REGION=$COMPOSER_REGION,\ _COMPOSER_DAG_BUCKET=$COMPOSER_DAG_BUCKET,\ _COMPOSER_DAG_NAME_PROD=$COMPOSER_DAG_NAME_PROD TASK 6:- echo "_DATAFLOW_JAR_BUCKET : ${DATAFLOW_JAR_BUCKET_TEST} _COMPOSER_INPUT_BUCKET : ${INPUT_BUCKET_TEST} _COMPOSER_REF_BUCKET : ${REF_BUCKET_TEST} _COMPOSER_DAG_BUCKET : ${COMPOSER_DAG_BUCKET} _COMPOSER_ENV_NAME : ${COMPOSER_ENV_NAME} _COMPOSER_REGION : ${COMPOSER_REGION} _COMPOSER_DAG_NAME_TEST : ${COMPOSER_DAG_NAME_TEST}" TASK :- Create a Trigger in cloud console For this task follow the lab instructions. TASK :- Test the trigger echo "testword" >> ~/$SOURCE_CODE_REPO/workflow-dag/support-files/input.txt echo "testword: 1" >> ~/$SOURCE_CODE_REPO/workflow-dag/support-files/ref.txt cd ~/$SOURCE_CODE_REPO git add . git commit -m 'change in test files' git push google master

Abhinandh J. · Diulas lebih dari 1 tahun lalu

good

Samvar V. · Diulas lebih dari 1 tahun lalu

Suresh G. · Diulas lebih dari 1 tahun lalu

Hoang N. · Diulas lebih dari 1 tahun lalu

Sara Consuelo C. · Diulas lebih dari 1 tahun lalu

Anurag P. · Diulas lebih dari 1 tahun lalu

took several tries to be able to complete the lab as kept running into different issues in different steps

Janne M. · Diulas lebih dari 1 tahun lalu

Artem B. · Diulas lebih dari 1 tahun lalu

grt

Pankaj R. · Diulas lebih dari 1 tahun lalu

Sadaf S. · Diulas lebih dari 1 tahun lalu

not able to open with incongto window

Suganya R. · Diulas lebih dari 1 tahun lalu

Meenakshi S. · Diulas hampir 2 tahun lalu

Oleksii C. · Diulas hampir 2 tahun lalu

Javier Z. · Diulas hampir 2 tahun lalu

open with incongto window

Suganya R. · Diulas hampir 2 tahun lalu

Pravin K. · Diulas hampir 2 tahun lalu

karim f. · Diulas hampir 2 tahun lalu

Vasil K. · Diulas hampir 2 tahun lalu

Lab fails in task 5, tried re-running it 4 times with: ERROR: Step #9 - "wait-for-dag-deployed-on-composer": Already have image (with digest): gcr.io/cloud-builders/gcloud Finished Step #9 - "wait-for-dag-deployed-on-composer" ERROR ERROR: build step 9 "gcr.io/cloud-builders/gcloud" failed: step exited with non-zero status: 1

Janne M. · Diulas hampir 2 tahun lalu

sakshi R. · Diulas hampir 2 tahun lalu

Sudhanya G. · Diulas hampir 2 tahun lalu

Tobia M. · Diulas hampir 2 tahun lalu

karim f. · Diulas hampir 2 tahun lalu

Meenakshi S. · Diulas hampir 2 tahun lalu