Exécution de tâches Apache Spark sur Cloud Dataproc avis

Aakriti J. · Examiné il y a plus de 6 ans

Nan W. · Examiné il y a plus de 6 ans

zmou z. · Examiné il y a plus de 6 ans

Good lab

Asaduddin A. · Examiné il y a plus de 6 ans

David H. · Examiné il y a plus de 6 ans

Rajmani R. · Examiné il y a plus de 6 ans

Ao B. · Examiné il y a plus de 6 ans

Priya C. · Examiné il y a plus de 6 ans

Blake L. · Examiné il y a plus de 6 ans

Mahomet N. · Examiné il y a plus de 6 ans

Cool!

cherifi a. · Examiné il y a plus de 6 ans

LTI S. · Examiné il y a plus de 6 ans

total crap

Denys R. · Examiné il y a plus de 6 ans

Dennys A R. · Examiné il y a plus de 6 ans

Taylor L. · Examiné il y a plus de 6 ans

Sarath J. · Examiné il y a plus de 6 ans

Prakhar K. · Examiné il y a plus de 6 ans

Nicole S. · Examiné il y a plus de 6 ans

20/04/28 19:56:10 INFO org.spark_project.jetty.util.log: Logging initialized @2563ms 20/04/28 19:56:10 INFO org.spark_project.jetty.server.Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown 20/04/28 19:56:11 INFO org.spark_project.jetty.server.Server: Started @2649ms 20/04/28 19:56:11 WARN org.apache.spark.util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 20/04/28 19:56:11 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector@1fe928c8{HTTP/1.1,[http/1.1]}{0.0.0.0:4041} 20/04/28 19:56:11 WARN org.apache.spark.scheduler.FairSchedulableBuilder: Fair Scheduler configuration file not found so jobs will be scheduled in FIFO order. To use fair scheduling, configure pools in fairscheduler.xml or set spark.scheduler.allocation.file to a file that contains the configuration. 20/04/28 19:56:12 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at sparktodp-m/10.128.0.3:8032 20/04/28 19:56:12 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at sparktodp-m/10.128.0.3:10200 20/04/28 19:56:14 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1588102115361_0005 Traceback (most recent call last): File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o51.partitions. : java.lang.IllegalArgumentException: java.net.URISyntaxException: Malformed IPv6 address at index 6: gs://[Your-Bucket-Name]/kddcup.data_10_percent.gz at org.apache.hadoop.fs.Path.initialize(Path.java:254) at org.apache.hadoop.fs.Path.<init>(Path.java:212) at org.apache.hadoop.util.StringUtils.stringToPath(StringUtils.java:243) at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:425) at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$30.apply(SparkContext.scala:1036) at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$30.apply(SparkContext.scala:1036) at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$5$$anonfun$apply$3.apply(HadoopRDD.scala:180) at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$5$$anonfun$apply$3.apply(HadoopRDD.scala:180) at scala.Option.foreach(Option.scala:257) at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$5.apply(HadoopRDD.scala:180) at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$5.apply(HadoopRDD.scala:177) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:171) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:269) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:269) at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61) at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.URISyntaxException: Malformed IPv6 address at index 6: gs://[Your-Bucket-Name]/kddcup.data_10_percent.gz at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.parseIPv6Reference(URI.java:3469) at java.net.URI$Parser.parseServer(URI.java:3219) at java.net.URI$Parser.parseAuthority(URI.java:3155) at java.net.URI$Parser.parseHierarchical(URI.java:3097) at java.net.URI$Parser.parse(URI.java:3053) at java.net.URI.<init>(URI.java:746) at org.apache.hadoop.fs.Path.initialize(Path.java:251) ... 35 more During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/a3b0f32081e2473bb7bbd34cb9a0f11c/spark_analysis.py", line 18, in <module> raw_rdd.take(5) File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1327, in take File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 391, in getNumPartitions File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco pyspark.sql.utils.IllegalArgumentException: 'java.net.URISyntaxException: Malformed IPv6 address at index 6: gs://[Your-Bucket-Name]/kddcup.data_10_percent.gz' 20/04/28 19:56:19 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@1fe928c8{HTTP/1.1,[http/1.1]}{0.0.0.0:4041} Job output is complete

Ramesh G. · Examiné il y a plus de 6 ans

Eric B. · Examiné il y a plus de 6 ans

unable to launch

Girish G. · Examiné il y a plus de 6 ans

Good lab, lots to learn and read through from jupyter notebook / python.

Dean P. · Examiné il y a plus de 6 ans

Trinnawat C. · Examiné il y a plus de 6 ans

David S. · Examiné il y a plus de 6 ans

unable to launch

Girish G. · Examiné il y a plus de 6 ans