Serverless Data Processing with Dataflow - CI/CD with Dataflow Rezensionen

Alex W. · Vor mehr als ein Jahr überprüft

sreehari R. · Vor mehr als ein Jahr überprüft

s.blok1@chello.nl B. · Vor mehr als ein Jahr überprüft

levi a. · Vor mehr als ein Jahr überprüft

ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Invalid image version: 'composer-1.20.12-airflow-2.3.4' derived based on your input: 'composer-1.20.12-airflow-2.3.4'. Currently suggested new environment image versions are: [composer-2.6.4-airflow-2.6.3, composer-2.6.4-airflow-2.5.3]. Cloud Composer 1 will reach the end of support on March 25, 2024. Creation of new Cloud Composer 1 environments is only possible if the project already contains a Composer 1 environment in the same location. We recommend migration to Cloud Composer 2. ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Configuring node location is not supported for Cloud Composer environments in versions 2.0.0 and newer.

Serhii Z. · Vor mehr als ein Jahr überprüft

a lot of bugs

Umang M. · Vor mehr als ein Jahr überprüft

Lab out of date: Multiple errors due to composer v1 not v2: cd ~/ci-cd-for-data-processing-workflow/env-setup chmod +x set_composer_variables.sh ./set_composer_variables.sh Welcome to Cloud Shell! Type "help" to get started. Your Cloud Platform project in this session is set to qwiklabs-gcp-03-9b7a5801b2d0. Use “gcloud config set project [PROJECT_ID]” to change to a different project. student_00_ea9e4e2b2fa3@cloudshell:~ (qwiklabs-gcp-03-9b7a5801b2d0)$ git clone https://github.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow.git Cloning into 'ci-cd-for-data-processing-workflow'... remote: Enumerating objects: 172, done. remote: Counting objects: 100% (100/100), done. remote: Compressing objects: 100% (60/60), done. remote: Total 172 (delta 47), reused 68 (delta 33), pack-reused 72 Receiving objects: 100% (172/172), 55.99 KiB | 2.80 MiB/s, done. Resolving deltas: 100% (66/66), done. student_00_ea9e4e2b2fa3@cloudshell:~ (qwiklabs-gcp-03-9b7a5801b2d0)$ cd ~/ci-cd-for-data-processing-workflow/env-setup source set_env.sh student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME \ --location $COMPOSER_REGION \ --zone $COMPOSER_ZONE_ID \ --python-version 3 \ --image-version composer-1.20.8-airflow-1.10.15 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Invalid image version: 'composer-1.20.8-airflow-1.10.15' derived based on your input: 'composer-1.20.8-airflow-1.10.15'. Currently suggested new environment image versions are: [composer-2.6.5-airflow-2.7.3, composer-2.6.5-airflow-2.6.3, composer-2.6.5-airflow-2.5.3]. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-1.20.8-airflow-1.10.15 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Invalid image version: 'composer-1.20.8-airflow-1.10.15' derived based on your input: 'composer-1.20.8-airflow-1.10.15'. Currently suggested new environment image versions are: [composer-2.6.5-airflow-2.7.3, composer-2.6.5-airflow-2.6.3, composer-2.6.5-airflow-2.5.3]. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-1.20.8-airflow-1.10.15 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Invalid image version: 'composer-1.20.8-airflow-1.10.15' derived based on your input: 'composer-1.20.8-airflow-1.10.15'. Currently suggested new environment image versions are: [composer-2.6.5-airflow-2.7.3, composer-2.6.5-airflow-2.6.3, composer-2.6.5-airflow-2.5.3]. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-2.6.5-airflow-2.7.3 ERROR: (gcloud.composer.environments.create) Cannot specify --python-version with Composer 2.X or greater. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-2.6.5-airflow-2.7.3^C --zone $COMPOSER_ZONE_ID \ --python-version 3 \ --image-version composer-1.20.8-airflow-1.10.15 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Invalid image version: 'composer-1.20.8-airflow-1.10.15' derived based on your input: 'composer-1.20.8-airflow-1.10.15'. Currently suggested new environment image versions are: [composer-2.6.5-airflow-2.7.3, composer-2.6.5-airflow-2.6.3, composer-2.6.5-airflow-2.5.3]. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-1.20.8-airflow-1.10.15 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Invalid image version: 'composer-1.20.8-airflow-1.10.15' derived based on your input: 'composer-1.20.8-airflow-1.10.15'. Currently suggested new environment image versions are: [composer-2.6.5-airflow-2.7.3, composer-2.6.5-airflow-2.6.3, composer-2.6.5-airflow-2.5.3]. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-1.20.8-airflow-1.10.15ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Invalid image version: 'composer-1.20.8-airflow-1.10.15' derived based on your input: 'composer-1.20.8-airflow-1.10.15'. Currently suggested new environment image versions are: [composer-2.6.5-airflow-2.7.3, composer-2.6.5-airflow-2.6.3, composer-2.6.5-airflow-2.5.3]. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-2.6.5-airflow-2.7.3 ERROR: (gcloud.composer.environments.create) Cannot specify --python-version with Composer 2.X or greater. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --python-version 3 --image-version composer-2.6.5-airflow-2.7.3^C student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --im age-version composer-2.6.5-airflow-2.7.3 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Configuring node location is not supported for Cloud Composer environments in versions 2.0.0 and newer. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --zone $COMPOSER_ZONE_ID --image-version composer-2.6.5-air flow-2.7.3 ERROR: (gcloud.composer.environments.create) Error parsing [environment]. The [environment] resource is not properly specified. Failed to find attribute [location]. The attribute can be set in the following ways: - provide the argument `environment` on the command line with a fully specified name - provide the argument `--location` on the command line - set the property `composer/location` student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --image-version composer-2.6.5-airflow-2.7.3 ERROR: (gcloud.composer.environments.create) FAILED_PRECONDITION: Composer API Service Agent service account (service-922910890009@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer v2 API Service Agent Extension role might be missing. Please refer to https://cloud.google.com/composer/docs/composer-2/create-environments#grant-permissions and Composer Creation Troubleshooting pages to resolve this issue. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --image-version composer-2.6.5-airflow-2.7.3 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Configuring node location is not supported for Cloud Composer environments in versions 2.0.0 and newer. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID WARNING: Not defining --image-version resolves to Cloud Composer current default version, which is presently Composer 2 and is subject to further changes in the future. Consider using --image-version=composer-A-airflow-X[.Y[.Z]]. More info at https://cloud.google.com/composer/docs/concepts/versioning/composer-versioning-overview#version-aliases ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Configuring node location is not supported for Cloud Composer environments in versions 2.0.0 and newer. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$ gcloud composer environments create $COMPOSER_ENV_NAME --location $COMPOSER_REGION --zone $COMPOSER_ZONE_ID --image-version composer-2.6.5-airflow-2.7.3 ERROR: (gcloud.composer.environments.create) INVALID_ARGUMENT: Found 1 problem: 1) Configuring node location is not supported for Cloud Composer environments in versions 2.0.0 and newer. student_00_ea9e4e2b2fa3@cloudshell:~/ci-cd-for-data-processing-workflow/env-setup (qwiklabs-gcp-03-9b7a5801b2d0)$

Mark G. · Vor mehr als ein Jahr überprüft

Bryan G. · Vor mehr als ein Jahr überprüft

given wrong composer image

JAQUELINE R. · Vor mehr als ein Jahr überprüft

Aakruti r. · Vor mehr als ein Jahr überprüft

Major issue/bug in this lab, commands are not working in the lab(outdated image versions used). Also, failing at step 9 of building pipeline even running after several times.

Priyank G. · Vor mehr als ein Jahr überprüft

Lab not working

Samson O. · Vor mehr als ein Jahr überprüft

Unable to create Cloud Composer environment with the steps in task 2. Task 5 commands fail

Brandt D. · Vor mehr als ein Jahr überprüft

Naveen S. · Vor mehr als ein Jahr überprüft

Naveen S. · Vor mehr als ein Jahr überprüft

JaisainadhReddy B. · Vor mehr als ein Jahr überprüft

Not able to proceed as the current lab have only permission to create composer 1 but composer v1 ended on march 25th 2024. Please correct the lab.

Arun J. · Vor mehr als ein Jahr überprüft

Naveen S. · Vor mehr als ein Jahr überprüft

This is one of the worst labs I have ever taken. So many bugs in the code (maybe it used to work with Airflow v1) that make it impossible to finish in time when using Composer with Airflow v2, and that is the only supported version as returned by the gcloud command creating the Composer environment. Please review and update this lab or remove it from the curriculum.

Michal B. · Vor mehr als ein Jahr überprüft

Priyank G. · Vor mehr als ein Jahr überprüft

Composer version is not supported anymore. lots of error in solutions too.

Hemant K. · Vor mehr als ein Jahr überprüft

Hemant K. · Vor mehr als ein Jahr überprüft

Unable to create Cloud Composer environment with the steps in task 2. Images are not compatible

Brandt D. · Vor mehr als ein Jahr überprüft

Hemant K. · Vor mehr als ein Jahr überprüft

EXCELENTE LABORATORIO

Carlos Mauricio C. · Vor mehr als ein Jahr überprüft