本篇文章給大家分享的是有關(guān)macOS Spark 2.4.3 standalone 搭建的示例分析，小編覺得挺實用的，因此分享給大家學(xué)習(xí)，希望大家閱讀完這篇文章后可以有所收獲，話不多說，跟著小編一起來看看吧。

我們提供的服務(wù)有：網(wǎng)站設(shè)計制作、成都網(wǎng)站設(shè)計、微信公眾號開發(fā)、網(wǎng)站優(yōu)化、網(wǎng)站認證、茅箭ssl等。為成百上千企事業(yè)單位解決了網(wǎng)站和推廣的問題。提供周到的售前咨詢和貼心的售后服務(wù)，是有科學(xué)管理、有技術(shù)的茅箭網(wǎng)站制作公司

based on

jdk 1.8

Mac os

1、解壓

nancylulululu:local nancy$ tar -zxvf /Users/nancy/Downloads/spark-2.4.3-bin-hadoop2.7.tar -C /usr/local/opt/

2、重命名并修改配置文件

nancylulululu:conf nancy$ cp slaves.template slaves

nancylulululu:conf nancy$ cp spark-defaults.conf.template spark-defaults.conf

nancylulululu:conf nancy$ cp spark-env.sh.template spark-env.sh

nancylulululu:conf nancy$ vi slaves

localhost

nancylulululu:conf nancy$ vi spark-env.sh

SPARK_LOCAL_IP=127.0.0.1

SPARK_MASTER_HOST=127.0.0.1

SPARK_MASTER_PORT=7077

SPARK_WORKER_CORES=2

SPARK_WORKER_MEMORY=1G

3、啟動spark

nancylulululu:spark-2.4.3 nancy$ sbin/start-all.sh

starting org.apache.spark.deploy.master.Master, logging to /usr/local/opt/spark-2.4.3/logs/spark-nancy-org.apache.spark.deploy.master.Master-1-nancylulululu.out

localhost: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/opt/spark-2.4.3/logs/spark-nancy-org.apache.spark.deploy.worker.Worker-1-nancylulululu.out

jps 查看一個 master 和一個 worker服務(wù)

nancylulululu:conf nancy$ jps

3908 CoarseGrainedExecutorBackend

3478 Master

4166 Jps

3896 SparkSubmit

3514 Worker

4、Launching Applications with spark-submit

Once a user application is bundled, it can be launched using the bin/spark-submit script. This script takes care of setting up the classpath with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark supports:

./bin/spark-submit \   --class <main-class> \   --master <master-url> \   --deploy-mode <deploy-mode> \   --conf <key>=<value> \   ... # other options   <application-jar> \   [application-arguments]

Some of the commonly used options are:

• --class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)

• --master: The  master URL  for the cluster (e.g. spark://23.195.26.187:7077)

• --deploy-mode: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client) ? 

• --conf: Arbitrary Spark configuration property in key=value format. For values that contain spaces wrap “key=value” in quotes (as shown).

• application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.

• application-arguments: Arguments passed to the main method of your main class, if any

腳本：

./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://localhost:7077 \
 /usr/local/opt/spark-2.4.3/examples/jars/spark-examples_2.11-2.4.3.jar \
100

結(jié)果

Pi is roughly 3.1413047141304715

19/05/14 14:23:06 INFO SparkUI: Stopped Spark web UI at http://localhost:4040

5、spark-shell 模式

nancylulululu:spark-2.4.3 nancy$ bin/spark-shell --master spark://localhost:7077

6、測試wordcount

測試文件

nancylulululu:opt nancy$ vi 3.txt

hello scala hello spark hello MySQL hello java hello java hello scala

測試結(jié)果

scala> sc.textFile("/usr/local/opt/3.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect

res1: Array[(String, Int)] = Array((scala,2), (mysql,1), (hello,6), (java,2), (spark,1))

scala> 

以上就是macOS Spark 2.4.3 standalone 搭建的示例分析，小編相信有部分知識點可能是我們?nèi)粘９ぷ鲿姷交蛴玫降?。希望你能通過這篇文章學(xué)到更多知識。更多詳情敬請關(guān)注創(chuàng)新互聯(lián)行業(yè)資訊頻道。

網(wǎng)站欄目：macOSSpark2.4.3standalone搭建的示例分析
標(biāo)題來源：http://sd-ha.com/article6/jsjcig.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供品牌網(wǎng)站制作、虛擬主機、自適應(yīng)網(wǎng)站、做網(wǎng)站、品牌網(wǎng)站設(shè)計、網(wǎng)站收錄

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會在第一時間刪除。文章觀點不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時需注明來源： 創(chuàng)新互聯(lián)