Spark应用程序，使用spark-submit，是用于部署在集群中的Spark应用程序中的shell命令。它采用全各自的集群管理器通过一个统一的接口。因此，您不必配置为每一个应用程序。

例

让我们以字数相同的例子，我们以前使用的，使用shell命令。在这里，我们考虑同样的例子，一个Spark的应用程序。

样本输入

下面的文字是输入数据，并命名该文件是in.txt。

people are not as beautiful as they look, 
as they walk or as they talk. 
they are only as beautiful  as they love, 
as they care as they share.

请看下面的程序 -

SparkWordCount.scala

import org.apache.spark.SparkContext 
import org.apache.spark.SparkContext._ 
import org.apache.spark._  

object SparkWordCount { 
   def main(args: Array[String]) { 

      val sc = new SparkContext( "local", "Word Count", "/usr/local/spark", Nil, Map(), Map()) 
		
      /* local = master URL; Word Count = application name; */  
      /* /usr/local/spark = Spark Home; Nil = jars; Map = environment */ 
      /* Map = variables to work nodes */ 
      /*creating an inputRDD to read text file (in.txt) through Spark context*/ 
      val input = sc.textFile("in.txt") 
      /* Transform the inputRDD into countRDD */ 
		
      valcount = input.flatMap(line ⇒ line.split(" ")) 
      .map(word ⇒ (word, 1)) 
      .reduceByKey(_ + _) 
       
      /* saveAsTextFile method is an action that effects on the RDD */  
      count.saveAsTextFile("outfile") 
      System.out.println("OK"); 
   } 
} 

上面的程序保存到一个名为SparkWordCount.scala文件，并将其放置在一个用户自定义命名的目录Spark的应用程序 。

注 -转变成inputRDD countRDD，我们使用flatMap（）的线（从文本文件）的记号化到字计数的词频和reduceByKey（）方法用于计算每个单词的重复，map（）方法。

使用以下步骤来提交申请。通过终端执行的Spark应用程序目录中的所有步骤。

第1步：下载SparkJA

Spark核心罐需要编译，因此，下载Sparkcore_2.10-1.3.0.jar从以下链接Spark核心瓶子 ，并从下载目录移动jar文件引发的应用程序目录。

第2步：编译程序

编译使用如下命令上面的程序。这个命令应该从Spark应用程序目录下执行。在这里，/usr/local/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar是Spark库采取了Hadoop的支持罐子。

$ scalac -classpath "spark-core_2.10-1.3.0.jar:/usr/local/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar" SparkPi.scala

第3步：创建一个JAR

创建使用下面的命令Spark应用程序的jar文件。这里， 单词计数为jar文件的文件名。

jar -cvf wordcount.jar SparkWordCount*.class spark-core_2.10-1.3.0.jar/usr/local/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar

第4步：提交申请Spark

使用以下命令提交Spark应用 -

spark-submit --class SparkWordCount --master local wordcount.jar

如果成功执行，那么你会发现下面给出的输出。在下面的输出确定出租是用于用户识别，这是程序的最后一行。如果你仔细阅读下面的输出，你会发现不同的东西，比如 -

• 在端口42954成功启动服务“sparkDriver”
• MemoryStore的开始与容量267.3 MB
• 在http://192.168.1.217:4040开始SparkUI
• 新增JAR文件：/home/hadoop/piapplication/count.jar
• ResultStage 1（saveAsTextFile在SparkPi.scala：11）0.566织完了
• 在http://192.168.1.217:4040停止Sparkweb用户界面
• MemoryStore的清除

15/07/08 13:56:04 INFO Slf4jLogger: Slf4jLogger started 
15/07/08 13:56:04 INFO Utils: Successfully started service 'sparkDriver' on port 42954. 
15/07/08 13:56:04 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.217:42954] 
15/07/08 13:56:04 INFO MemoryStore: MemoryStore started with capacity 267.3 MB 
15/07/08 13:56:05 INFO HttpServer: Starting HTTP Server 
15/07/08 13:56:05 INFO Utils: Successfully started service 'HTTP file server' on port 56707. 
15/07/08 13:56:06 INFO SparkUI: Started SparkUI at http://192.168.1.217:4040 
15/07/08 13:56:07 INFO SparkContext: Added JAR file:/home/hadoop/piapplication/count.jar at http://192.168.1.217:56707/jars/count.jar with timestamp 1436343967029 
15/07/08 13:56:11 INFO Executor: Adding file:/tmp/spark-45a07b83-42ed-42b3b2c2-823d8d99c5af/userFiles-df4f4c20-a368-4cdd-a2a7-39ed45eb30cf/count.jar to class loader 
15/07/08 13:56:11 INFO HadoopRDD: Input split: file:/home/hadoop/piapplication/in.txt:0+54 
15/07/08 13:56:12 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2001 bytes result sent to driver 
 (MapPartitionsRDD[5] at saveAsTextFile at SparkPi.scala:11), which is now runnable 
15/07/08 13:56:12 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at saveAsTextFile at SparkPi.scala:11) 
15/07/08 13:56:13 INFO DAGScheduler: ResultStage 1 (saveAsTextFile at SparkPi.scala:11) finished in 0.566 s 
15/07/08 13:56:13 INFO DAGScheduler: Job 0 finished: saveAsTextFile at SparkPi.scala:11, took 2.892996 s
OK 
15/07/08 13:56:13 INFO SparkContext: Invoking stop() from shutdown hook 
15/07/08 13:56:13 INFO SparkUI: Stopped Spark web UI at http://192.168.1.217:4040 
15/07/08 13:56:13 INFO DAGScheduler: Stopping DAGScheduler 
15/07/08 13:56:14 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 
15/07/08 13:56:14 INFO Utils: path = /tmp/spark-45a07b83-42ed-42b3-b2c2823d8d99c5af/blockmgr-ccdda9e3-24f6-491b-b509-3d15a9e05818, already present as root for deletion. 
15/07/08 13:56:14 INFO MemoryStore: MemoryStore cleared 
15/07/08 13:56:14 INFO BlockManager: BlockManager stopped 
15/07/08 13:56:14 INFO BlockManagerMaster: BlockManagerMaster stopped 
15/07/08 13:56:14 INFO SparkContext: Successfully stopped SparkContext 
15/07/08 13:56:14 INFO Utils: Shutdown hook called 
15/07/08 13:56:14 INFO Utils: Deleting directory /tmp/spark-45a07b83-42ed-42b3b2c2-823d8d99c5af 
15/07/08 13:56:14 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!  

第5步：检查输出

该计划的成功执行后，你会发现在Spark的应用程序目录中名为OUTFILE的目录。

下面的命令用于在OUTFILE目录中打开和检查的文件列表。

$ cd outfile 
$ ls 
Part-00000 part-00001 _SUCCESS

在部分00000文件检查输出的命令是-

$ cat part-00000 
(people,1) 
(are,2) 
(not,1) 
(as,8) 
(beautiful,2) 
(they, 7) 
(look,1)

在部分00001文件检查输出的命令是 -

$ cat part-00001 
(walk, 1) 
(or, 1) 
(talk, 1) 
(only, 1) 
(love, 1) 
(care, 1) 
(share, 1)

通过下面的章节去了解更多关于“spark-submit”命令。

Spark提交语法

spark-submit [options] <app jar | python file> [app arguments]

选项