SparkLauncher 3.2.2 and spark 2.4.4 jobs

[jozefbakus]

Test compatibility of SparkLauncher 3.2.2 and spark 2.4.4 jobs

[kevinwallimann]

Submitting a Spark 2.4.4 job with spark-submit 3.2.1 fails with the following exception:

Unknown/unsupported param List(--dist-cache-conf, /redacted-path/application_1659437389982_73072/container_e878_1659437389982_73072_02_000001/__spark_conf__/__spark_dist_cache__.properties)

Usage: org.apache.spark.deploy.yarn.ApplicationMaster [options]
Options:
  --jar JAR_PATH       Path to your application's JAR file
  --class CLASS_NAME   Name of your application's main class
  --primary-py-file    A main Python file
  --primary-r-file     A main R file
  --arg ARG            Argument to be passed to your application's main class.
                       Multiple invocations are possible, each will be passed in order.
  --properties-file FILE Path to a custom Spark properties file.

Unfortunately, there's no way to ignore the extra parameter. See https://github.com/apache/spark/blob/v2.4.4/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala#L32
There is also no way not to add the dist-cache-conf parameter. See https://github.com/apache/spark/blob/v3.2.2/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1047

[kevinwallimann]

Submitting a Spark 3.2.1 job with spark-submit 2.4.4 fails with the following error message

Error: Could not find or load main class org.apache.spark.executor.YarnCoarseGrainedExecutorBackend

It is a symptom of the --dist-cache-conf parameter missing. Obviously, spark-submit 2.4.4 doesn't set this parameter.

YarnCoarseGrainedExecutorBackend is not present in spark-yarn_2.12-3.2.1.jar, however, the resources are not correctly configured. Comparing the logs for the same job submitted with spark-submit 3.2.1, we see in the executor logs:

Default YARN executor launch context:
  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__
    SPARK_YARN_STAGING_DIR -> <... omitted ...>
    SPARK_USER -> <... omitted ...>

  command:
      {{JAVA_HOME}}/bin/java \ 

<... omitted ...>

      -XX:OnOutOfMemoryError='kill %p' \ 
      org.apache.spark.executor.YarnCoarseGrainedExecutorBackend \ 
      --user-class-path \ 
      file:$PWD/__app__.jar \ 
      1><LOG_DIR>/stdout \ 
      2><LOG_DIR>/stderr

  resources:
    __spark_libs__/spire-platform_2.12-0.17.0.jar -> resource <... omitted ...>
    __spark_libs__/javolution-5.5.1.jar -> resource <... omitted ...>
    __spark_libs__/algebra_2.12-2.0.1.jar -> resource <... omitted ...>
    __spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar -> <... omitted ...>

<... omitted / more spark jars ...>

The logs for a job submitted with spark-submit 2.4.4 look the same, but the resources list is empty. That's because the resources list is determined by the --dist-cache-conf parameter in Spark 3.2.1, see https://github.com/apache/spark/blob/v3.2.2/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L461

[kevinwallimann]

With the change in apache/spark#23338 (fix version 3.0.0), the distributed cache configs (prefixed by spark.yarn.cache) are sent in a separate file to YARN. Previously, they were part of __spark_conf__.properties, now they are sent in a separate file __spark_dist_cache__.properties. E.g. spark.yarn.cache.filenames contains the paths for the spark jars, which explains why they are missing for spark-submit 2.4.4

Therefore, submitting a Spark 2.4 Job with Spark-Submit 3.x and vice versa does not work.