[DEPRECATED] - Recommending Products Using Cloud SQL and Spark Rezensionen

Krishnakumar R. · Vor fast 3 Jahre überprüft

Charles Y. · Vor fast 3 Jahre überprüft

Alfonso V. · Vor fast 3 Jahre überprüft

Adriano Ramos A. · Vor fast 3 Jahre überprüft

Got error at the last step where the job was being run and it threw a SQL exception which was related to some missing drivers. Attaching the error here : 23/06/25 17:05:02 INFO org.apache.spark.SparkEnv: Registering MapOutputTracker 23/06/25 17:05:02 INFO org.apache.spark.SparkEnv: Registering BlockManagerMaster 23/06/25 17:05:02 INFO org.apache.spark.SparkEnv: Registering BlockManagerMasterHeartbeat 23/06/25 17:05:02 INFO org.apache.spark.SparkEnv: Registering OutputCommitCoordinator 23/06/25 17:05:02 INFO org.sparkproject.jetty.util.log: Logging initialized @4744ms to org.sparkproject.jetty.util.log.Slf4jLog 23/06/25 17:05:03 INFO org.sparkproject.jetty.server.Server: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_372-b07 23/06/25 17:05:03 INFO org.sparkproject.jetty.server.Server: Started @4923ms 23/06/25 17:05:03 INFO org.sparkproject.jetty.server.AbstractConnector: Started ServerConnector@42752b74{HTTP/1.1, (http/1.1)}{0.0.0.0:37663} 23/06/25 17:05:04 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at rentals-m/10.128.0.3:8032 23/06/25 17:05:04 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at rentals-m/10.128.0.3:10200 23/06/25 17:05:06 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found 23/06/25 17:05:06 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'. 23/06/25 17:05:06 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1687711250774_0002 23/06/25 17:05:08 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at rentals-m/10.128.0.3:8030 23/06/25 17:05:11 INFO com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl: Ignoring exception of type GoogleJsonResponseException; verified object already exists with desired state. 23/06/25 17:05:12 WARN org.apache.spark.SparkContext: Spark is not running in local mode, therefore the checkpoint directory must not be on the local filesystem. Directory 'checkpoint/' appears to be on the local filesystem. Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. Traceback (most recent call last): File "/tmp/job-9b4cd098_attempt_1/train_and_apply.py", line 48, in <module> dfRates = sqlContext.read.format('jdbc').options(driver=jdbcDriver, url=jdbcUrl, dbtable='Rating', useSSL='false').load() File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 210, in load File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o68.load. : java.sql.SQLException: Access denied for user 'root'@'35.193.136.60' (using password: YES) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:130) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:825) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:446) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:239) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:188) at org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider.getConnection(BasicConnectionProvider.scala:49) at org.apache.spark.sql.execution.datasources.jdbc.connection.ConnectionProvider$.create(ConnectionProvider.scala:77) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$createConnectionFactory$1(JdbcUtils.scala:62) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:56) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226) at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:355) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325) at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750) 23/06/25 17:05:14 INFO org.sparkproject.jetty.server.AbstractConnector: Stopped Spark@42752b74{HTTP/1.1, (http/1.1)}{0.0.0.0:0}

Paritosh M. · Vor fast 3 Jahre überprüft

Elena B. · Vor fast 3 Jahre überprüft

Rui S. · Vor fast 3 Jahre überprüft

Prakash P. · Vor fast 3 Jahre überprüft

Dinesh C. · Vor fast 3 Jahre überprüft

Naresh K. · Vor fast 3 Jahre überprüft

wasn't sure what cluster to create

Chris F. · Vor fast 3 Jahre überprüft

Daniel P. · Vor fast 3 Jahre überprüft

Christian Robert V. · Vor fast 3 Jahre überprüft

Xiao H. · Vor fast 3 Jahre überprüft

Francis Rey S. · Vor fast 3 Jahre überprüft

Gabrielle S. · Vor fast 3 Jahre überprüft

Vemal M. · Vor fast 3 Jahre überprüft

Great

Ajay A. · Vor fast 3 Jahre überprüft

Some description in the doc is outdated. E.g., need to choose where to create the Dataproc but it is not mentioned here

Jun M. · Vor fast 3 Jahre überprüft

Some parts of the tutorial are out of date with what is in the GCP console, making it harder to follow.

Tami G. · Vor fast 3 Jahre überprüft

Hari M. · Vor fast 3 Jahre überprüft

Takashi H. · Vor fast 3 Jahre überprüft

Several freezes in cluster and cloud shell creation. Too much copy and paste and not enough explanation or opportunity to arrive at answers yourself

Domantas V. · Vor fast 3 Jahre überprüft

Charles D. · Vor fast 3 Jahre überprüft

Manas C. · Vor fast 3 Jahre überprüft