Have a pyspark code which is running succesfully as per scehdule every month. It failed this month; we have not made any changes to the code or the configuration. Have looked at all the similar threads but one seems to help.
Any suggestions are greatly appreciated!
The complete traceback is as - ''' Significant Credit Monthly starts at 2020-12-04 17:55:04.761266
Traceback (most recent call last): File "/data/code/sig_cr_dc/Sig_CD_Monthly1.py", line 105, in <module> Sig_Cdt_Stats.write.mode("overwrite").save(Sig_Cdt_Path) File "/var/opt/teradata/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p2245.2467/lib/spark/python/lib/", line 397, in save File "/var/opt/teradata/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p2245.2467/lib/spark/python/lib/", line 813, in __call__ File "/var/opt/teradata/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p2245.2467/lib/spark/python/lib/", line 45, in deco File "/var/opt/teradata/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p2245.2467/lib/spark/python/lib/", line 308, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o115.save. : org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:154) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:106) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:56) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:56) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:256) at org.apache.spark.sql.DataFrameWriter.dataSource$lzycompute$1(DataFrameWriter.scala:181) at org.apache.spark.sql.DataFrameWriter.org$apache$spark$sql$DataFrameWriter$$dataSource$1(DataFrameWriter.scala:181) at org.apache.spark.sql.DataFrameWriter$$anonfun$save$1.apply$mcV$sp(DataFrameWriter.scala:188) at org.apache.spark.sql.DataFrameWriter.executeAndCallQEListener(DataFrameWriter.scala:154) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:188) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:172) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 164 in stage 9.0 failed 4 times, most recent failure: Lost task 164.3 in stage 9.0 (TID 18018, kmblhdpdata2.kbank.kotakgroup.com, executor 149): ExecutorLostFailure (executor 149 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1433) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1421) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1420) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1420) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1644) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1603) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1592) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1862) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1875) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1952) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:148) ... 31 more ''' Related questions 0 Py4JJavaError in Pyspark 1 Facing Py4JJavaError on executing PySpark Code 10 Py4JJavaError: An error occurred while calling Related questions 0 Py4JJavaError in Pyspark 1 Facing Py4JJavaError on executing PySpark Code 10 Py4JJavaError: An error occurred while calling 2 Py4JJavaError running Pyspark program 5 Py4JJavaError: An error occurred while calling o1670.collectToPython 0 py4j.protocol.Py4JavaError: An error occured while calling o22.start 2 PySpark : py4j.protocol.Py4JJavaError: An error occurred while calling o65.save 0 Spark:Py4JJavaError: An error occurred while calling o142.saveAsTextFile 0 Py4JError: An error occurred while calling o230.and 1 py4j.protocol.Py4JJavaError: An error occurred while calling o49.csv Load 7 more related questions Show fewer related questions
Reset to default
