在Master上安装Spark环境：
Spark集群环境搭建：
搭建hadoop集群使用hadoop版本是hadoop2.6.4
搭建spark这里使用spark版本是spark1.6.2（spark-1.6.2-bin-hadoop2.6.tgz）
1、下载安装包到master虚拟服务器：
在线下载：
hadoop@master:~/wget
离线上传到集群：
?
2、解压spark安装包到master虚拟服务器/usr/local/spark下，并分配权限：
#解压到/usr/local/下hadoop@master:~$sudotar-zxvfspark-1.6.2-bin-hadoop2.6.tgz-C/usr/local/
hadoop@master:~$cd/usr/local/
hadoop@master:/usr/local$ls
bingamesincludemansharesrc
etchadooplibsbinspark-1.6.2-bin-hadoop2.6#重命名为spark
hadoop@master:/usr/local$sudomvspark-1.6.2-bin-hadoop2.6/spark/
hadoop@master:/usr/local$ls
binetcgameshadoopincludelibmansbinsharesparksrc#分配权限
hadoop@master:/usr/local$sudochown-Rhadoop:hadoopspark
hadoop@master:/usr/local$
3、在master虚拟服务器/etc/profile中添加Spark环境变量：
编辑/etc/profile文件
sudovim/etc/profile
在尾部添加$SPARK_HOME变量，添加后，目前我的/etc/profile文件尾部内容如下：
exportJAVA_HOME=/usr/lib/jvm/java-8-oracle
exportJRE_HOME=/usr/lib/jvm/java-8-oracle
exportSCALA_HOME=/opt/scala/scala-2.10.5
#addhadoopbin/directorytoPATH
exportHADOOP_HOME=/usr/local/hadoop
exportSPARK_HOME=/usr/local/spark
exportPATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$JAVA_HOME:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin:$SPARK_HOME/bin:$PATH
exportCLASSPATH=$CLASS_PATH::$JAVA_HOME/lib:$JAVA_HOME/jre/lib
生效：
source/etc/profile
在Master配置Spark：
1、配置master虚拟服务器hadoop-env.sh文件：
sudovim/usr/local/spark/conf/hadoop-env.sh
注意：默认情况下没有hadoop-env.sh和slaves文件，而是.template文件，需要重命名：
hadoop@master:/usr/local/spark/conf$ls
docker.properties.templatemetrics.properties.templatespark-env.sh
fairscheduler.xml.templateslaves.template
log4j.properties.templatespark-defaults.conf.template
hadoop@master:/usr/local/spark/conf$sudovimspark-env.sh
hadoop@master:/usr/local/spark/conf$mvslaves.templateslaves
在文件末尾追加如下内容：
exportSTANDALONE_SPARK_MASTER_HOST=192.168.1.200
exportSPARK_MASTER_IP=192.168.1.200
exportSPARK_WORKER_CORES=1
#everyslavenodestartworkinstancecount
exportSPARK_WORKER_INSTANCES=1
exportSPARK_MASTER_PORT=7077
exportSPARK_WORKER_MEMORY=1g
exportMASTER=spark://${SPARK_MASTER_IP}:${SPARK_MASTER_PORT}
exportSCALA_HOME=/opt/scala/scala-2.10.5
exportJAVA_HOME=/usr/lib/jvm/java-8-oracle
exportSPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://172.21.7.10:9000/SparkEventLog"
exportSPARK_WORKDER_OPTS="-Dspark.worker.cleanup.enabled=true"
exportHADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
2、配置master虚拟服务器下slaves文件：
sudovim/usr/local/spark/conf/slaves
在slaves文件中内容如下：
192.168.1.201
192.168.1.202
注意：每行写一个机器的ip。
3、Master虚拟机下/usr/local/spark/目录下创建logs文件夹，并分配777权限：
hadoop@master:/usr/local/spark$mkdirlogs
hadoop@master:/usr/local/spark$chmod777logs
复制Master虚拟服务器上的/usr/loca/spark下文件到所有slaves节点（node1、node2）下：
1、复制Master虚拟服务器上/usr/local/spark/安装文件到各个salves(node1、node2)上：
注意：拷贝钱需要在ssh到所有salves节点（node1、node2）上，创建/usr/local/spark/目录，并分配777权限。
hadoop@master:/usr/local/spark/conf$cd~/
hadoop@master:~$sudochmod777/usr/local/spark
hadoop@master:~$scp-r/usr/local/sparkhadoop@node1:/usr/local
scp:/usr/local/spark:Permissiondenied
hadoop@master:~$sudoscp-r/usr/local/sparkhadoop@node1:/usr/local
hadoop@node1''spassword:
scp:/usr/local/spark:Permissiondenied
hadoop@master:~$sudochmod777/usr/local/spark
hadoop@master:~$sshnode1
WelcometoUbuntu14.04.2LTS(GNU/Linux3.16.0-30-genericx86_64)
Documentation:www.sm136.com/
SysteminformationasofFriSep2316:40:31UTC2016
Systemload:0.08Processes:400
Usageof/:12.2%of17.34GBUsersloggedin:0
Memoryusage:5%IPaddressforeth0:192.168.1.201
Swapusage:0%
Graphthisdataandmanagethissystemat:
https://landscape.canonical.com/
Newrelease''16.04.1LTS''available.
Run''do-release-upgrade''toupgradetoit.
Lastlogin:WedSep2116:19:252016frommaster
hadoop@node1:~$cd/usr/local/
hadoop@node1:/usr/local$sudomkdirspark
[sudo]passwordforhadoop:
hadoop@node1:/usr/local$ls
binetcgameshadoopincludelibmansbinsharesparksrc
hadoop@node1:/usr/local$sudochmod777./spark
hadoop@node1:/usr/local$exit
hadoop@master:~$scp-r/usr/local/sparkhadoop@node1:/usr/local
...........
hadoop@master:~$sshnode2
WelcometoUbuntu14.04.2LTS(GNU/Linux3.16.0-30-genericx86_64)
Documentation:www.hunanwang.net/
SysteminformationasofFriSep2316:15:03UTC2016
Systemload:0.08Processes:435
Usageof/:13.0%of17.34GBUsersloggedin:0
Memoryusage:6%IPaddressforeth0:192.168.1.202
Swapusage:0%
Graphthisdataandmanagethissystemat:
https://landscape.canonical.com/
Lastlogin:WedSep2116:19:472016frommaster
hadoop@node2:~$cd/usr/local
hadoop@node2:/usr/local$sudomkdirspark
[sudo]passwordforhadoop:
hadoop@node2:/usr/local$sudochmod777./spark
hadoop@node2:/usr/local$exit
logout
Connectiontonode2closed.
hadoop@master:~$scp-r/usr/local/sparkhadoop@node2:/usr/local
...........
2、修改所有salves节点(node1、node2)上/etc/profile，追加$SPARK_HOME环境变量:
注意：一般都会遇到权限问题。最好登录到各个salves节点(node1、node2)上手动编辑/etc/profile。
hadoop@master:~$sshnode1
WelcometoUbuntu14.04.2LTS(GNU/Linux3.16.0-30-genericx86_64)
Documentation:www.shanxiwang.net/
SysteminformationasofFriSep2316:42:44UTC2016
Systemload:0.01Processes:400
Usageof/:12.2%of17.34GBUsersloggedin:0
Memoryusage:5%IPaddressforeth0:192.168.1.201
Swapusage:0%
Graphthisdataandmanagethissystemat:
www.visa158.com
Newrelease''16.04.1LTS''available.
Run''do-release-upgrade''toupgradetoit.
Lastlogin:FriSep2316:40:522016frommaster
hadoop@node1:~$sudovim/etc/profile
[sudo]passwordforhadoop:
hadoop@node1:~$exit
logout
Connectiontonode1closed.
hadoop@master:~$sshnode2
WelcometoUbuntu14.04.2LTS(GNU/Linux3.16.0-30-genericx86_64)
Documentation:help.ubuntu.com/
SysteminformationasofFriSep2316:44:42UTC2016
Systemload:0.0Processes:400
Usageof/:13.0%of17.34GBUsersloggedin:0
Memoryusage:5%IPaddressforeth0:192.168.1.202
Swapusage:0%
Graphthisdataandmanagethissystemat:
landscape.canonical.com/
Newrelease''16.04.1LTS''available.
Run''do-release-upgrade''toupgradetoit.
Lastlogin:FriSep2316:43:312016frommaster
hadoop@node2:~$sudovim/etc/profile
[sudo]passwordforhadoop:
hadoop@node2:~$exit
logout
Connectiontonode2closed.
hadoop@master:~$
修改后的所有salves上/etc/profile文件与master节点上/etc/profile文件配置一致。
在Master启动spark并验证是否配置成功：
1、启动命令：
一般要确保hadoop已经启动，之后才启动spark
hadoop@master:~$cd/usr/local/spark/
hadoop@master:/usr/local/spark$./sbin/start-all.sh
2、验证是否启动成功：
方法一、jps
hadoop@master:/usr/local/spark$./sbin/start-all.sh
startingorg.apache.spark.deploy.master.Master,loggingto/usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-master.out
192.168.1.201:startingorg.apache.spark.deploy.worker.Worker,loggingto/usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-node1.out
192.168.1.202:startingorg.apache.spark.deploy.worker.Worker,loggingto/usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-node2.out
hadoop@master:/usr/local/spark$jps
1650NameNode
1875SecondaryNameNode
3494Jps
2025ResourceManager
3423Master
hadoop@master:/usr/local/spark$cd~/
hadoop@master:~$sshnode1
WelcometoUbuntu14.04.2LTS(GNU/Linux3.16.0-30-genericx86_64)
Documentation:help.ubuntu.com/
SysteminformationasofFriSep2317:33:40UTC2016
Systemload:0.06Processes:402
Usageof/:13.9%of17.34GBUsersloggedin:0
Memoryusage:21%IPaddressforeth0:192.168.1.201
Swapusage:0%
Graphthisdataandmanagethissystemat:
landscape.canonical.com/
Newrelease''16.04.1LTS''available.
Run''do-release-upgrade''toupgradetoit.
Lastlogin:FriSep2317:33:102016frommaster
hadoop@node1:~$jps
1392DataNode
2449Jps
2330Worker
2079NodeManager
hadoop@node1:~$exit
logout
Connectiontonode1closed.
hadoop@master:~$sshnode2
WelcometoUbuntu14.04.2LTS(GNU/Linux3.16.0-30-genericx86_64)
Documentation://help.ubuntu.com/
SysteminformationasofFriSep2317:33:40UTC2016
Systemload:0.07Processes:404
Usageof/:14.7%of17.34GBUsersloggedin:0
Memoryusage:20%IPaddressforeth0:192.168.1.202
Swapusage:0%
Graphthisdataandmanagethissystemat:
landscape.canonical.com/
Newrelease''16.04.1LTS''available.
Run''do-release-upgrade''toupgradetoit.
Lastlogin:FriSep2316:51:362016frommaster
hadoop@node2:~$jps
2264Worker
2090NodeManager
1402DataNode
2383Jps
hadoop@node2:~$
?
方法二、web方式http://192.168.1.200:8080看是否正常:
|
|
