GSP692

Overview

In the dynamic world of data engineering and workflow orchestration, organizations are increasingly migrating from legacy enterprise schedulers like Control-M to the open-source powerhouse, Apache Airflow. This transition often involves a complex and time-consuming process of converting existing job definitions. DAGify in a Managed Airflow environment accelerates and simplifies Control-M workflows to a cloud native environment, which allows the focus to be on optimizing workflows, and building and orchestrating data pipelines in the new environment.

DAGify is an open-source solution to automate the conversion of Control-M XML files into Airflow's native DAG format. It's also a migration accelerator as DAGify significantly reduces the manual effort, and therefore potential errors associated with the transition to Airflow.

In the flow below, Dagify converts XML files from a legacy Enterprise scheduler into Python DAG files. This expedites the transition to Airflow, in this case Airflow on Managed Airflow.

Cloud Managed Service for Airflow offers organizations a fully managed Airflow experience. It eliminates the complexities of managing Airflow infrastructure.

In this lab, you learn about using DAGify to convert Control-M export files into a Python native DAG and running migrated DAGs in a Managed Airflow Gen 3 environment.

What you'll learn

In this lab, you learn how to perform the following tasks:

• Create a Managed Airflow Gen 3 environment
• Download and configure DAGify
• Use DAGify to convert Control-M export files into a Python native DAG
• Deploy DAGs to the Managed Airflow environment
• Make modifications to the DAGify configuration and templates

Setup and requirements

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources are made available to you.

This hands-on lab lets you do the lab activities in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito (recommended) or private browser window to run this lab. This prevents conflicts between your personal account and the student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab—remember, once you start, you cannot pause a lab.

Note: Use only the student account for this lab. If you use a different Google Cloud account, you may incur charges to that account.

How to start your lab and sign in to the Google Cloud console

1. Click the Start Lab button. If you need to pay for the lab, a dialog opens for you to select your payment method. On the right is the Lab setup and access panel with the following:

   • The Open Google Cloud console button
   • The temporary credentials (username and password) that you must use for this lab
   • Other information, if needed, to step through this lab

   Note that the lab timer is located near the top of the page, showing the remaining time.

2. Click Open Google Cloud console (or right-click and select Open Link in Incognito Window if you are running the Chrome browser).

   The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username below and paste it into the Sign in dialog.

   {{{user_0.username | "Username"}}}

   You can also find the Username in the Lab setup and access panel.

4. Click Next.

5. Copy the Password below and paste it into the Welcome dialog.

   {{{user_0.password | "Password"}}}

   You can also find the Password in the Lab setup and access panel.

6. Click Next.

   Important: You must use the credentials the lab provides you. Do not use your Google Cloud account credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

7. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Google Cloud console opens in this tab.

Note: To access Google Cloud products and services, click the Navigation menu or type the service or product name in the Search field.

Task 1. Create a Managed Airflow environment

In this task, you create a Managed Airflow Gen 3 environment.

1. On the Google Cloud console title bar, in the Search field, type Managed Airflow, and then click on it.

2. On the Environments page, for Create environment, select Managed Airflow Gen 3.

3. For Name, type .

4. For Location, select .

5. For Service account, select .

6. For Resilience mode select Standard resilience.

7. Select Allow access from all IP addresses for Web server network access control.

8. Scroll to the bottom and click Show Advanced configuration.

9. For Environment bucket, select Custom bucket.

10. For Bucket name, click Browse, select , and then click Select.

Leave all other fields as default.

11. Click Create to create the environment.

NOTE: It can take up to 20 minutes to create the environment. You can continue with the lab and return to recheck this task before you deploy DAG to Managed Airflow.

Click Check my progress to verify the objective. Create a Managed Airflow Environment

Task 2. Download and configure DAGify

In this task, you download and set up the DAGify tool.

1. In the console, on the Navigation menu (), click Compute Engine > VM instances.

2. For the instance named lab-setup, in the Connect column, click SSH to open an SSH-in-browser terminal window.

3. In the terminal window, run the following command to clone the DAGify repository:

git clone https://github.com/GoogleCloudPlatform/dagify

3. Navigate to the folder dagify and execute the command below:

cd ~/dagify make dagify-clean

Wait a minute for the Python Packages to be installed.

4. Once completed, activate the Python virtual environment by running the command:

source venv/bin/activate

5. Verify DAGify setup by running the following command:

python ./DAGify.py -h

Output:

Usage: DAGify.py [OPTIONS] Run dagify. Options: -s, --source-path TEXT Path to source files for conversion [default: (./source)] -o, --output-path TEXT Path to output files after conversion. [default: (./output)] -c, --config-file TEXT Path to dagify configuration file. [default: (./config.yaml)] -t, --templates TEXT Path to dagify configuration file. [default: (./dagify/templates)] -d, --dag-divider TEXT Which field in Job Definition should be used to divide up DAGS. [default: (PARENT_FOLDER)] -r, --report Generate report in txt and json format which gives an overview of job_types converted -h, --help Show this message and exit.

Click Check my progress to verify the objective. Download and configure DAGify

Task 3. Run the DAGify command with sample data in default mode

In this task, you use DAGify in default mode to convert the sample Control-M export file in XML format into a Python native DAG.

1. In the lab-setup terminal, view the Control-M job XML file 001-tfatf.xml by running the following command:

cat ./sample_data/control-m/001-tfatf.xml

Output:

<?xml version="1.0" encoding="UTF-8"?> <DEFTABLE xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Folder.xsd"> <SMART_FOLDER FOLDER_NAME="fx_fld_001"> <!-- Folder 001, Application 001, Sub Application 001, Job 001 --> <JOB APPLICATION="fx_fld_001_app_001" SUB_APPLICATION="fx_fld_001_app_001_subapp_001" JOBNAME="fx_fld_001_app_001_subapp_001_job_001" DESCRIPTION="fx_fld_001_app_001_subapp_001_job_001_reports" TASKTYPE="Command" CMDLINE="echo I am task A" PARENT_FOLDER="fx_fld_001"> <OUTCOND NAME="fx_fld_001_app_001_subapp_001_job_001_ok" ODATE="ODAT" SIGN="+" /> </JOB> <!-- Folder 001, Application 001, Sub Application 001, Job 002 --> <JOB APPLICATION="fx_fld_001_app_001" SUB_APPLICATION="fx_fld_001_app_001_subapp_001" JOBNAME="fx_fld_001_app_001_subapp_001_job_002" DESCRIPTION="fx_fld_001_app_001_subapp_001_job_002_reports" TASKTYPE="Command" CMDLINE="echo I am task B" PARENT_FOLDER="fx_fld_001"> <INCOND NAME="fx_fld_001_app_001_subapp_001_job_001_ok" ODATE="ODAT" AND_OR="A" /> <OUTCOND NAME="fx_fld_001_app_001_subapp_001_job_002_ok" ODATE="ODAT" SIGN="+" /> </JOB> ... </SMART_FOLDER>

2. Run DAGify using the default configuration:

python ./DAGify.py -s ./sample_data/control-m/001-tfatf.xml -o ./output/lab-output-task-3/

This converts the sample Control-M export file in XML format into a Python native DAG and stores it in the output format lab-output-task-3 folder.

This also stores the final DAG Python file within a subfolder of the name of the Control-M project folder, which in this case, is 001-tfatf.

3. List the output DAG files in the ./output/lab-output-task-3/001-tfatf/ directory by running the command below:

ls ./output/lab-output-task-3/001-tfatf/

Output:

fx_fld_001.py

This is your converted Python DAG and it's ready for deployment within Managed Airflow.

4. View the converted Python DAG file fx_fld_001.py by running the command below:

cat ./output/lab-output-task-3/001-tfatf/fx_fld_001.py

Output:

# Apache Airflow Base Imports from airflow import DAG from airflow.decorators import task from airflow.sensors.external_task import ExternalTaskMarker from airflow.sensors.external_task import ExternalTaskSensor import datetime # Apache Airflow Custom & DAG/Task Specific Imports from airflow.operators.bash import BashOperator default_args = { 'owner': 'airflow', } with DAG( dag_id="fx_fld_001", start_date=datetime.datetime(2024, 1, 1), schedule_interval="@daily", # TIMEFROM not found, default schedule set to @daily, catchup=False, ) as dag: # DAG Tasks fx_fld_001_app_001_subapp_001_job_001 = BashOperator( task_id="fx_fld_001_app_001_subapp_001_job_001", bash_command="echo I am task A", trigger_rule="all_success", dag=dag, ) ...

Click Check my progress to verify the objective. Run DAGify command with sample data in the default mode

Task 4. Deploy DAG to Managed Airflow

In this task, you check that the Managed Airflow environment is up and then deploy the converted DAG file into the environment.

1. In the console title bar, type Managed Airflow in the Search field and then click on it.

Validate that your Managed Airflow environment is up and running and ready for use.

Note: If the Managed Airflow environment is still being created, wait for the creation process to complete.

2. From the environment list, click , to open the Environment details page.

3. Click Open DAGs folder to access the DAGs folder in the Cloud Storage bucket .

Here you can see the example default DAG airflow_monitoring.py that is issued when a managed airflow environment is created.

4. Return to the SSH terminal window.

5. Upload the new Python DAG fx_fld_001.py to the Cloud Storage bucket associated with Managed Airflow environment using the gcloud storage cp command:

gcloud storage cp -r ~/dagify/output/lab-output-task-3/001-tfatf/* gs://{{{primary_project.startup_script.bucket_name|filled in at lab start}}}/dags/lab-output-task-3/

6. Inspect that the Python DAG file has been uploaded into the correct storage bucket.

gcloud storage ls gs://{{{primary_project.startup_script.bucket_name | filled in at lab start}}}/dags/lab-output-task-3/*

Output:

gs://{{{primary_project.startup_script.bucket_name|filled in at lab start}}}/dags/lab-output-task-3/fx_fld_001.py

7. Go back to the Environment details page for environment.

8. Click Open Airflow UI.

9. Authenticate using user.

Verify that fx_fld_001 DAG is visible in the DAGs list.

NOTE: It can take up to 2-3 minutes for uploaded DAGs to Synchronize into Google Managed Airflow

Explore Managed Airflow

In this task you explore the detailed view of fx_fld_001 in Managed Airflow.

1. Click fx_fld_001 to open the detailed view of the DAG.

2. Click Code to view the original converted Python source code.

3. Click Graph to view the graph of all tasks and dependencies.

Click Check my progress to verify the objective. Deploy DAG to Managed Airflow

Task 5. Run the DAGify command with sample data using a DAG divider

In this task, you use DAGify with the DAG divider flag ( -d) to divide control-m workflow into multiple DAG files based on the XML Key APPLICATION.

1. Switch to the lab-setup terminal window.

2. View Control-M job XML file 002-tftf.xml by running the command:

cat ./sample_data/control-m/002-tftf.xml

Output:

<?xml version="1.0" encoding="UTF-8"?> <DEFTABLE xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Folder.xsd"> <SMART_FOLDER FOLDER_NAME="fx_fld_001"> <!-- Folder 001, Application 001, Sub Application 001, Job 001 --> <JOB APPLICATION="fx_fld_001_app_001" SUB_APPLICATION="fx_fld_001_app_001_subapp_001" JOBNAME="fx_fld_001_app_001_subapp_001_job_001" DESCRIPTION="fx_fld_001_app_001_subapp_001_job_001_reports" TASKTYPE="Command" CMDLINE="" PARENT_FOLDER="fx_fld_001"> <OUTCOND NAME="fx_fld_001_app_001_subapp_001_job_001_ok" ODATE="ODAT" SIGN="+" /> </JOB> <!-- Folder 001, Application 001, Sub Application 001, Job 002 --> <JOB APPLICATION="fx_fld_001_app_001" SUB_APPLICATION="fx_fld_001_app_001_subapp_001" JOBNAME="fx_fld_001_app_001_subapp_001_job_002" DESCRIPTION="fx_fld_001_app_001_subapp_001_job_002_reports" TASKTYPE="Command" CMDLINE="" PARENT_FOLDER="fx_fld_001"> <INCOND NAME="fx_fld_001_app_001_subapp_001_job_001_ok" ODATE="ODAT" AND_OR="A" /> <OUTCOND NAME="fx_fld_001_app_001_subapp_001_job_002_ok" ODATE="ODAT" SIGN="+" /> </JOB> ... </SMART_FOLDER>

3. Run the DAGify command using the DAG divider flag:

python ./DAGify.py -s ./sample_data/control-m/002-tftf.xml -o ./output/lab-output-task-5/ -d APPLICATION

This converts the sample Control-M export file in XML format into a Python native DAG and stores it in the output format lab-output-task-5 folder.

The final DAG Python file is stored within a subfolder in the Control-M project folder. For this case, that folder is 002-tftf.

4. List the output DAG files by running the command:

ls ./output/lab-output-task-5/002-tftf/

You should see multiple Python files created in the directory ./output/lab-output-task-5/002-tftf/ similar to following:

Output:

fx_fld_001_app_001.py fx_fld_001_app_002.py

These are new Python DAGs that are ready to deploy within Managed Airflow.

5. View a converted Python DAG file, in this case fx_fld_001_app_001.py, by running the command:

cat ./output/lab-output-task-5/002-tftf/fx_fld_001_app_001.py

Output:

# Apache Airflow Base Imports from airflow import DAG from airflow.decorators import task from airflow.sensors.external_task import ExternalTaskMarker from airflow.sensors.external_task import ExternalTaskSensor import datetime # Apache Airflow Custom & DAG/Task Specific Imports from airflow.operators.bash import BashOperator default_args = { 'owner': 'airflow', } with DAG( dag_id="fx_fld_001_app_001", start_date=datetime.datetime(2024, 1, 1), schedule_interval="@daily", # TIMEFROM not found, default schedule set to @daily, catchup=False, ) as dag: # DAG Tasks fx_fld_001_app_001_subapp_001_job_001 = BashOperator( task_id="fx_fld_001_app_001_subapp_001_job_001", bash_command="", trigger_rule="all_success", dag=dag, ) ...

6. Deploy the converted Python DAG files by running the gcloud storage cp command to move them to the Cloud Storage bucket associated with the Managed Airflow environment:

gcloud storage cp -r ~/dagify/output/lab-output-task-5/002-tftf/* gs://{{{primary_project.startup_script.bucket_name|filled in at lab start}}}/dags/lab-output-task-5/

7. Verify that the Python DAG files are uploaded into the correct storage bucket by running the command:

gcloud storage ls gs://{{{primary_project.startup_script.bucket_name | filled in at lab start}}}/dags/lab-output-task-5/*

Output:

gs://{{{primary_project.startup_script.bucket_name|filled in at lab start}}}/dags/lab-output-task-5/fx_fld_001_app_001.py gs://{{{primary_project.startup_script.bucket_name|filled in at lab start}}}/dags/lab-output-task-5/fx_fld_001_app_002.py

8. Go back to the Environment details page for environment.

9. Click Open Airflow UI.

Verify that fx_fld_001_app_001 and fx_fld_001_app_002 DAGs are visible in the DAGs list.

NOTE: It can take up to 2-3 minutes for uploaded DAGs to synchronize into Google Managed Airflow.

Click Check my progress to verify the objective. Run DAGify command with sample data using a DAG divider

Congratulations!

You’ve successfully used DAGify to convert Control-M export files into a Python DAG and uploaded migrated DAGs in the Managed Airflow environment.

Next steps / Learn more

For more information about the DAGify, refer to DAGify Github repository

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated May 18, 2026

Lab Last Tested April 16, 2026

Copyright 2026 Google LLC. All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.