Review overridable Spark configuration properties for SparkJobConfiguration

Description

Make it possible to append user configuration for:

  • spark.driver.extraJavaOptions

  • spark.executor.extraJavaOptions

  • spark.yarn.dist.pyFiles

  • environment variables (LD_LIBRARY_PATH, PATH, PYTHONPATH)

Disallow users from setting environment variables such as HOPSWORKS_VERSION, HADOOP_USER_NAME, HADOOP_HOME

Assignee

Robin

Reporter

Robin

Labels

None

Fix versions

Priority

Medium
Configure