[DEPRECATED] Serverless Data Processing with Dataflow - Using Dataflow SQL for Batch Analytics (Java) Reviews

Silvio C. · Reviewed over 1 year ago

Dmitry P. · Reviewed over 1 year ago

Iurie S. · Reviewed over 1 year ago

Bala Abinaya N. · Reviewed over 1 year ago

Understand the batch pipeline flow

Batthi V. · Reviewed over 1 year ago

Rares R. · Reviewed over 1 year ago

Diego M. · Reviewed over 1 year ago

Vinoth R. · Reviewed over 1 year ago

Marco H. · Reviewed over 1 year ago

Shaik S. · Reviewed over 1 year ago

Riya G. · Reviewed over 1 year ago

Marco H. · Reviewed over 1 year ago

Maximo C. · Reviewed over 1 year ago

Abhishek G. · Reviewed over 1 year ago

OMG. Very frustrating: 1. VM had to be reset multiple times. 2. When trying to run the two pipelines received the following error: "[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/home/theia). Please verify you invoked Maven from the correct directory. -> [Help 1]" I fixed the problem by (1) compiling, (2) then setting the variables, (3) compiling againg, and (4) finally running the pipelines: Compile the Project: Run Maven to clean and compile the project to ensure that all classes are properly compiled. sh Copy code mvn clean compile Run the Maven Command with Debug Logging: If the problem persists, run Maven with the -e or -X switch to get more detailed error information. This will help in diagnosing the issue further. sh Copy code mvn -e exec:java \ -Dexec.mainClass=${MAIN_CLASS_NAME} \ -Dexec.cleanupDaemonThreads=false \ -Dexec.args=" \ --project=${PROJECT_ID} \ --region=${REGION} \ --stagingLocation=${PIPELINE_FOLDER}/staging \ --tempLocation=${PIPELINE_FOLDER}/temp \ --runner=${RUNNER} \ --inputPath=${INPUT_PATH} \ --tableName=${TABLE_NAME}" Example Command Sequence Navigate to the Project Directory: sh Copy code cd /home/project/training-data-analyst/quests/dataflow/4_SQL_Batch_Analytics/labs Set Environment Variables: sh Copy code export PROJECT_ID=$(gcloud config get-value project) export REGION='us-west1' export BUCKET=gs://${PROJECT_ID} export PIPELINE_FOLDER=${BUCKET} export MAIN_CLASS_NAME=com.mypackage.pipeline.BatchMinuteTrafficSQLPipeline export RUNNER=DataflowRunner export INPUT_PATH=${PIPELINE_FOLDER}/events.json export TABLE_NAME=${PROJECT_ID}:logs.minute_traffic Compile the Project: sh Copy code mvn clean compile Run the Project: sh Copy code mvn exec:java \ -Dexec.mainClass=${MAIN_CLASS_NAME} \ -Dexec.cleanupDaemonThreads=false \ -Dexec.args=" \ --project=${PROJECT_ID} \ --region=${REGION} \ --stagingLocation=${PIPELINE_FOLDER}/staging \ --tempLocation=${PIPELINE_FOLDER}/temp \ --runner=${RUNNER} \ --inputPath=${INPUT_PATH} \ --tableName=${TABLE_NAME}"

Andres Felipe G. · Reviewed over 1 year ago

VM constantly crashes

Andres Felipe G. · Reviewed over 1 year ago

manjing m. · Reviewed over 1 year ago

The lab doesnt work

Wiehan W. · Reviewed over 1 year ago

Iman E. · Reviewed over 1 year ago

completed

Shabana K. · Reviewed over 1 year ago

Thamu M. · Reviewed over 1 year ago

<?xml version="1.0" encoding="UTF-8" ?>  <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.mypackage</groupId> <artifactId>advanced-dataflow-lab-4-solution</artifactId> <version>1.0</version> <properties>  <beam.version>2.32.0</beam.version> <java.version>1.8</java.version> <slf4j.version>1.7.25</slf4j.version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven-dependency-plugin.version>3.0.1</maven-dependency-plugin.version> </properties> <dependencies> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-core</artifactId> <version>${beam.version}</version> </dependency> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-runners-direct-java</artifactId> <version>${beam.version}</version> </dependency> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId> <version>${beam.version}</version> </dependency> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-runners-google-cloud-dataflow-java</artifactId> <version>${beam.version}</version> </dependency> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId> <version>${beam.version}</version> </dependency> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-extensions-sql</artifactId> <version>${beam.version}</version> </dependency> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-extensions-sql-zetasql</artifactId> <version>${beam.version}</version> </dependency>  <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>${slf4j.version}</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <version>${slf4j.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.8.1</version> <configuration> <source>${java.version}</source> <target>${java.version}</target> </configuration> </plugin> </plugins> </build> </project> yet it says not completed ma be glitch in check progress & actual completion validating

Shabana K. · Reviewed over 1 year ago

vchemodurov@griddynamics.com S. · Reviewed over 1 year ago

Was stuck with some plugin failure no matter how much i tried to fix it just failed-The example in solution also failed - when you say if you are stuck look at the example

Shabana K. · Reviewed over 1 year ago

David Lamin G. · Reviewed over 1 year ago