Spark 的三种参数配置

一、spark-defults.conf

Example:
	spark.master                     spark://master:7077
	spark.eventLog.enabled           true
	spark.eventLog.dir               hdfs://namenode:8021/directory
	spark.serializer                 org.apache.spark.serializer.KryoSerializer
	spark.driver.memory              5g
	spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

​

二、spark submit

./bin/spark-submit 
--name "My app" 
--master local[4] 
--conf spark.shuffle.spill=false  
--conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" myApp.jar
​

三、sparkconf

SparkConf conf =  new SparkConf()
        .setMaster("local[*]")
        .setAppName("sqlVsDsl")
        .set("spark.executor.memory", "1g");
​