Kerberos Auth With SparkLauncher - "Cannot locate KDC" Error

We are trying to move from using spark-submit to a programmatic solution using SparkLauncher (Spark 2.4 running on Cloudera Hadoop), and have encountered this problem:

Exception javax.security.auth.login.LoginException: Cannot locate KDC 

on the error output stream of a job process after calling startApplication().

The launcher is configured with the same settings as spark-submit, including setting "spark.kerberos.principal" and "spark.kerberos.keytab" (also tried using the spark.yarn prefix, and setting --principle and --keytab with setSparkArg() as well - but these properties are not the issue).

Even though krb5.conf is in the default location on all machines and the driver, I tried setting

.setConf("spark.executor.extraJavaOptions", "-Djava.security.krb5.conf=/etc/krb5.conf") .setConf("spark.driver.extraJavaOptions", "-Djava.security.krb5.conf=/etc/krb5.conf") 

as well, with the same result.

There are no other auth/kerberos-related conf settings with the spark-submit call (only principal and keytab are set), and all other conf settings are replicated on the Launcher.

The output from SparkLauncher confirms all Spark properties are set as expected.

"Cannot locate KDC" indicates that the krb5.conf file is not found, but how can this be if it's in the default location and spark-submit evidently finds it?

1 Answer

This error does not necessarily mean that it cannot locate your krb5.conf file. It can also arise when your krb5.conf file does not contain the correct domain or realm specifications for your principal.

For example, if your principal is [email protected] but your krb5.conf file only specifies [email protected], you will also get this error.

Additionally, if your krb5.conf is located in a location other than default (/etc/krb5.conf), I was having issues with setting the javaExtraOpts. I found pointing updating the env var prior to running helped: export SPARK_SUBMIT_OPTS="-Djava.security.krb5.conf=/path/to/krb5-new.conf"

Lastly, I found setting --principle and --keytab args in spark-submit actually caused issues, and removing these can help.