Viacheslav Melnyk
Member since 2022
Gold League
25305 points
Member since 2022
In the last installment of the Dataflow course series, we will introduce the components of the Dataflow operational model. We will examine tools and techniques for troubleshooting and optimizing pipeline performance. We will then review testing, deployment, and reliability best practices for Dataflow pipelines. We will conclude with a review of Templates, which makes it easy to scale Dataflow pipelines to organizations with hundreds of users. These lessons will help ensure that your data platform is stable and resilient to unanticipated circumstances.
In this second installment of the Dataflow course series, we are going to be diving deeper on developing pipelines using the Beam SDK. We start with a review of Apache Beam concepts. Next, we discuss processing streaming data using windows, watermarks and triggers. We then cover options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs. We move onto reviewing best practices that help maximize your pipeline performance. Towards the end of the course, we introduce SQL and Dataframes to represent your business logic in Beam and how to iteratively develop pipelines using Beam notebooks.
This course is part 1 of a 3-course series on Serverless Data Processing with Dataflow. In this first course, we start with a refresher of what Apache Beam is and its relationship with Dataflow. Next, we talk about the Apache Beam vision and the benefits of the Beam Portability framework. The Beam Portability framework achieves the vision that a developer can use their favorite programming language with their preferred execution backend. We then show you how Dataflow allows you to separate compute and storage while saving money, and how identity, access, and management tools interact with your Dataflow pipelines. Lastly, we look at how to implement the right security model for your use case on Dataflow.
Complete the intermediate Build a Data Warehouse with BigQuery skill badge course to demonstrate skills in the following: joining data to create new tables, troubleshooting joins, appending data with unions, creating date-partitioned tables, and working with JSON, arrays, and structs in BigQuery.
Complete the intermediate Implement Cloud Security Fundamentals on Google Cloud skill badge course to demonstrate skills in the following: creating and assigning roles with Identity and Access Management (IAM); creating and managing service accounts; enabling private connectivity across virtual private cloud (VPC) networks; restricting application access using Identity-Aware Proxy; managing keys and encrypted data using Cloud Key Management Service (KMS); and creating a private Kubernetes cluster.
Complete the introductory Migrate MySQL Data to Cloud SQL Using Database Migration Service skill badge course to demonstrate skills in the following: migrating MySQL data to Cloud SQL using different job types and connectivity options available in Database Migration Service and migrating MySQL user data when running Database Migration Service jobs.
Complete the intermediate Perform Predictive Data Analysis in BigQuery skill badge course to demonstrate skills in the following: creating datasets in BigQuery by importing CSV and JSON files; harnessing the power of BigQuery with sophisticated SQL analytical concepts, including using BigQuery ML to train an expected goals model on soccer event data and evaluate the impressiveness of World Cup goals.
Earn a skill badge by completing the Share Data Using Google Data Cloud skill badge course, where you will gain practical experience with Google Cloud Data Sharing Partners, which have proprietary datasets that customers can use for their analytics use cases. Customers subscribe to this data, query it within their own platform, then augment it with their own datasets and use their visualization tools for their customer facing dashboards.
Complete the intermediate Manage Data Models in Looker skill badge course to demonstrate skills in the following: maintaining LookML project health; utilizing SQL runner for data validation; employing LookML best practices; optimizing queries and reports for performance; and implementing persistent derived tables and caching policies.
Complete the introductory Build LookML Objects in Looker skill badge course to demonstrate skills in the following: building new dimensions and measures, views, and derived tables; setting measure filters and types based on requirements; updating dimensions and measures; building and refining Explores; joining views to existing Explores; and deciding which LookML objects to create based on business requirements.
Complete the intermediate Create ML Models with BigQuery ML skill badge to demonstrate skills in creating and evaluating machine learning models with BigQuery ML to make data predictions.
Giriş düzeyindeki BigQuery Verilerinden Analiz Elde Etme beceri rozetini alarak şu konulardaki becerilerinizi gösterin: SQL sorguları yazma, herkese açık tabloları sorgulama, örnek verileri BigQuery'ye yükleme, BigQuery'deki sorgu doğrulayıcı ile yaygın söz dizimi sorunlarını giderme ve BigQuery verilerine bağlanarak Data Studio'da rapor oluşturma.
Complete the introductory Prepare Data for Looker Dashboards and Reports skill badge course to demonstrate skills in the following: filtering, sorting, and pivoting data; merging results from different Looker Explores; and using functions and operators to build Looker dashboards and reports for data analysis and visualization.
Complete the intermediate Develop Serverless Apps with Firebase skill badge course to demonstrate skills in the following: architecting and building serverless web applications with Firebase, utilizing Firestore for database management, automating deployment processes using Cloud Build, and integrating Google Assistant functionality into your applications.
Complete the intermediate Develop Serverless Applications on Cloud Run skill badge course to demonstrate skills in the following: integrating Cloud Run with Cloud Storage for data management, architecting resilient asynchronous systems using Cloud Run and Pub/Sub, constructing REST API gateways powered by Cloud Run, and building and deploying services on Cloud Run.
Complete the intermediate Optimize Costs for Google Kubernetes Engine skill badge course to demonstrate skills in the following: creating and managing multi-tenant clusters, monitoring resource usage by namespace, configuring cluster and pod autoscaling for efficiency, setting up load balancing for optimal resource distribution, and implementing liveness and readiness probes to ensure application health and cost-effectiveness.
Complete the intermediate Build Infrastructure with Terraform on Google Cloud skill badge to demonstrate skills in the following: Infrastructure as Code (IaC) principles using Terraform, provisioning and managing Google Cloud resources with Terraform configurations, effective state management (local and remote), and modularizing Terraform code for reusability and organization.
Learn how to use Gemini Notebook to create a personalized study guide for the Professional Cloud Architect certification exam. You'll review Gemini Notebook features, create a notebook in Gemini Notebook, and learn how to use a study guide to practice for a certification exam.
Complete the intermediate Deploy Kubernetes Applications on Google Cloud skill badge course to demonstrate skills in the following: Configuring and building Docker container images.Creating and managing Google Kubernetes Engine (GKE) clusters.Utilizing kubectl for efficient cluster management.Deploying Kubernetes applications with robust continuous delivery (CD) practices.
Giriş düzeyindeki Compute Engine İçin Cloud Load Balancing'i Uygulama beceri rozetini tamamlayarak şu konulardaki becerilerinizi gösterin: Compute Engine'de sanal makineler oluşturma ve dağıtma. Ağ ve uygulama yük dengeleyicileri yapılandırma.
Google Cloud'da Web Sitesi Derleme beceri rozeti kursunu tamamlayarak giriş düzeyindeki beceri rozetini kazanın. Get Cooking in Cloud serisine dayanan bu kursta şu konular ele alınır:Cloud Run'da web sitesi dağıtmaCompute Engine'de web uygulaması barındırmaGoogle Kubernetes Engine'de web sitenizi oluşturma, dağıtma ve ölçeklendirmeCloud Build'i kullanarak monolitik bir uygulamadan mikro hizmetler mimarisine geçiş
Complete the intermediate Engineer Data for Predictive Modeling with BigQuery ML skill badge to demonstrate skills in the following: building data transformation pipelines to BigQuery using Dataprep by Trifacta; using Cloud Storage, Dataflow, and BigQuery to build extract, transform, and load (ETL) workflows; and building machine learning models using BigQuery ML.
Earn a skill badge by completing the Set Up a Google Cloud Network skill badge course, where you will learn how to perform basic networking tasks on Google Cloud Platform - create a custom network, add subnets firewall rules, then create VMs and test the latency when they communicate with each other.
Google Cloud Ağınızı Geliştirme kursunu tamamlayarak bir beceri rozeti kazanın. IAM rollerini keşfetme ve proje erişimi ekleme/kaldırma, VPC ağları oluşturma, Compute Engine sanal makinelerini dağıtma ve izleme, SQL sorguları yazma ve çeşitli dağıtım yaklaşımlarıyla Kubernetes'i kullanarak uygulama dağıtma gibi uygulamaları dağıtıp izlemeyle ilgili birden çok yöntemi öğreneceksiniz.
Giriş düzeyindeki Google Cloud'da Makine Öğrenimi API'leri İçin Veri Hazırlama beceri rozetini tamamlayarak şu konulardaki becerilerinizi gösterin: Dataprep by Trifacta ile veri temizleme, Dataflow'da veri ardışık düzenleri çalıştırma, Managed Service for Apache Spark'ta küme oluşturma ve Apache Spark işleri çalıştırma ve makine öğrenimi API'lerini (Cloud Natural Language API, Google Cloud Speech-to-Text API ve Video Intelligence API dahil olmak üzere) çağırma.
Google Cloud'da Uygulama Geliştirme Ortamı Oluşturma kursunu tamamlayarak beceri rozeti kazanın. Bu kursta Cloud Storage, Identity and Access Management, Cloud Functions ve Pub/Sub gibi teknolojilerin temel özelliklerini kullanarak depolama odaklı bulut altyapısı oluşturma ve bu altyapıyla bağlantı kurmayı öğreneceksiniz.