Running Jobs on Managed Apache Spark Reviews

Isaac C. · Reviewed 5 yıldan fazla ago

sima s. · Reviewed 5 yıldan fazla ago

Hemalatha R. · Reviewed 5 yıldan fazla ago

Kevin J. · Reviewed 5 yıldan fazla ago

Rahul R. · Reviewed 5 yıldan fazla ago

Lo H. · Reviewed 5 yıldan fazla ago

SAGAR S. · Reviewed 5 yıldan fazla ago

Ingrid V. · Reviewed 5 yıldan fazla ago

Minh V. · Reviewed 5 yıldan fazla ago

thibault b. · Reviewed 5 yıldan fazla ago

Charita M. · Reviewed 5 yıldan fazla ago

this lesson is not sensefull for a user like me who has no expectations with jupyter and hadoop. I have no clue what i am doing here.

Christian G. · Reviewed 5 yıldan fazla ago

Neha C. · Reviewed 5 yıldan fazla ago

Amazing !

Rafael B. · Reviewed 5 yıldan fazla ago

Indra S. · Reviewed 5 yıldan fazla ago

So M. · Reviewed 5 yıldan fazla ago

Followed instructions and still got a failed job

Osaze J. · Reviewed 5 yıldan fazla ago

couldn't start jupyter notebook

Osaze J. · Reviewed 5 yıldan fazla ago

Michał G. · Reviewed 5 yıldan fazla ago

Charles C. · Reviewed 5 yıldan fazla ago

The last step of the process partially ran, then failed. Log File: 21/04/08 19:44:40 INFO org.sparkproject.jetty.util.log: Logging initialized @3163ms to org.sparkproject.jetty.util.log.Slf4jLog 21/04/08 19:44:41 INFO org.sparkproject.jetty.server.Server: jetty-9.4.36.v20210114; built: 2021-01-14T16:44:28.689Z; git: 238ec6997c7806b055319a6d11f8ae7564adc0de; jvm 1.8.0_282-b08 21/04/08 19:44:41 INFO org.sparkproject.jetty.server.Server: Started @3266ms 21/04/08 19:44:41 INFO org.sparkproject.jetty.server.AbstractConnector: Started ServerConnector@7a1522a1{HTTP/1.1, (http/1.1)}{0.0.0.0:38967} 21/04/08 19:44:41 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at sparktodp-m/10.128.0.4:8032 21/04/08 19:44:42 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at sparktodp-m/10.128.0.4:10200 21/04/08 19:44:42 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found 21/04/08 19:44:42 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'. 21/04/08 19:44:43 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1617907104315_0004 21/04/08 19:44:44 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at sparktodp-m/10.128.0.4:8030 21/04/08 19:44:48 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Thread (Thread[GetFileInfo #0,5,main]) interrupted: java.lang.InterruptedException at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:510) at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88) at org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48) at org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21/04/08 19:44:48 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Thread (Thread[GetFileInfo #1,5,main]) interrupted: java.lang.InterruptedException at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:510) at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88) at org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48) at org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21/04/08 19:44:48 INFO org.apache.hadoop.mapred.FileInputFormat: Total input files to process : 1 +-------------+------+ |protocol_type| count| +-------------+------+ | icmp|283602| | tcp|190065| | udp| 20354| +-------------+------+ +-------------+---------+----------+--------------+--------------+-------------+-------------------+-----------------+--------------------+-------------------+------------------+ |protocol_type| state|total_freq|mean_src_bytes|mean_dst_bytes|mean_duration|total_failed_logins|total_compromised|total_file_creations|total_root_attempts|total_root_acceses| +-------------+---------+----------+--------------+--------------+-------------+-------------------+-----------------+--------------------+-------------------+------------------+ | icmp| attack| 282314| 932.14| 0.0| 0.0| 0| 0| 0| 0.0| 0| | tcp| attack| 113252| 9880.38| 881.41| 23.19| 57| 2269| 76| 1.0| 152| | tcp|no attack| 76813| 1439.31| 4263.97| 11.08| 18| 2776| 459| 17.0| 5456| | udp|no attack| 19177| 98.01| 89.89| 1054.63| 0| 0| 0| 0.0| 0| | icmp|no attack| 1288| 91.47| 0.0| 0.0| 0| 0| 0| 0.0| 0| | udp| attack| 1177| 27.5| 0.23| 0.0| 0| 0| 0| 0.0| 0| +-------------+---------+----------+--------------+--------------+-------------+-------------------+-----------------+--------------------+-------------------+------------------+ 21/04/08 19:45:17 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception java.util.ConcurrentModificationException at java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:424) at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:420) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.mutable.MapLike.toSeq(MapLike.scala:75) at scala.collection.mutable.MapLike.toSeq$(MapLike.scala:72) at scala.collection.mutable.AbstractMap.toSeq(Map.scala:84) at org.apache.spark.scheduler.EventLoggingListener.redactProperties(EventLoggingListener.scala:290) at org.apache.spark.scheduler.EventLoggingListener.onJobStart(EventLoggingListener.scala:162) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) Traceback (most recent call last): File "/tmp/4bbe4086093c488292d8cb578c38dc1f/spark_analysis.py", line 75, in <module> bucket = gcs.Client().get_bucket(BUCKET) File "/opt/conda/default/lib/python3.8/site-packages/google/cloud/storage/client.py", line 393, in get_bucket bucket = self._bucket_arg_to_bucket(bucket_or_name) File "/opt/conda/default/lib/python3.8/site-packages/google/cloud/storage/client.py", line 231, in _bucket_arg_to_bucket bucket = Bucket(self, name=bucket_or_name) File "/opt/conda/default/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 583, in __init__ name = _validate_name(name) File "/opt/conda/default/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 66, in _validate_name if not all([name[0].isalnum(), name[-1].isalnum()]): IndexError: string index out of range 21/04/08 19:45:18 INFO org.sparkproject.jetty.server.AbstractConnector: Stopped Spark@7a1522a1{HTTP/1.1, (http/1.1)}{0.0.0.0:0}

John M. · Reviewed 5 yıldan fazla ago

Novita S. · Reviewed 5 yıldan fazla ago

Dave H. · Reviewed 5 yıldan fazla ago

Rodrigo J. · Reviewed 5 yıldan fazla ago

Ivan R. · Reviewed 5 yıldan fazla ago