1、启动yarn
[root@master sbin]# ./start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /opt/softWare/hadoop/hadoop-2.7.3/logs/hadoop-root-namenode-master.out
slaves1: starting datanode, logging to /opt/softWare/hadoop/hadoop-2.7.3/logs/hadoop-root-datanode-slaves1.out
slaves2: starting datanode, logging to /opt/softWare/hadoop/hadoop-2.7.3/logs/hadoop-root-datanode-slaves2.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /opt/softWare/hadoop/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-master.out
[root@master sbin]# ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/softWare/hadoop/hadoop-2.7.3/logs/yarn-root-resourcemanager-master.out
slaves2: starting nodemanager, logging to /opt/softWare/hadoop/hadoop-2.7.3/logs/yarn-root-nodemanager-slaves2.out
slaves1: starting nodemanager, logging to /opt/softWare/hadoop/hadoop-2.7.3/logs/yarn-root-nodemanager-slaves1.out
2、访问yarn的web
ip:8088
3、测试spark自带Demo，提交抖音小店yarn
[root@master bin]# ./spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi ../examples/jars/spark-examples_2.11-2.2.0.jar 1
[root@master bin]# ./spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi ../examples/jars/spark-examples_2.11-2.2.0.jar 1
19/12/05 22:24:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/12/05 22:24:58 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.230.21:8032
19/12/05 22:24:59 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
19/12/05 22:24:59 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (4096 MB per container)
19/12/05 22:24:59 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
19/12/05 22:24:59 INFO yarn.Client: Setting up container launch context for our AM
19/12/05 22:24:59 INFO yarn.Client: Setting up the launch environment for our AM container
19/12/05 22:24:59 INFO yarn.Client: Preparing resources for our AM container
19/12/05 22:25:02 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/12/05 22:25:06 INFO yarn.Client: Uploading resource file:/tmp/spark-3a82aa7c-e7f6-4094-8ce0-f5a25f57d128/__spark_libs__8566170070460254509.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1575555841019_0001/__spark_libs__8566170070460254509.zip
19/12/05 22:25:10 INFO yarn.Client: Uploading resource file:/opt/softWare/spark/spark-2.20/examples/jars/spark-examples_2.11-2.2.0.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1575555841019_0001/spark-examples_2.11-2.2.0.jar
19/12/05 22:25:10 INFO yarn.Client: Uploading resource file:/tmp/spark-3a82aa7c-e7f6-4094-8ce0-f5a25f57d128/__spark_conf__5855181180415715459.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1575555841019_0001/__spark_conf__.zip
19/12/05 22:25:10 WARN yarn.Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
19/12/05 22:25:10 INFO spark.SecurityManager: Changing view acls to: root
19/12/05 22:25:10 INFO spark.SecurityManager: Changing modify acls to: root
19/12/05 22:25:10 INFO spark.SecurityManager: Changing view acls groups to:
19/12/05 22:25:10 INFO spark.SecurityManager: Changing modify acls groups to:
19/12/05 22:25:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/12/05 22:25:10 INFO yarn.Client: Submitting application application_1575555841019_0001 to ResourceManager
19/12/05 22:25:11 INFO impl.YarnClientImpl: Submitted application application_1575555841019_0001
19/12/05 22:25:12 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:12 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1575555910997
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1575555841019_0001/
user: root
19/12/05 22:25:13 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:14 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:15 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:16 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:17 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:18 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:19 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:20 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:21 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:22 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:23 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:24 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:25 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:26 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:27 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:28 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:29 INFO yarn.Client: Application report for application_1575555841019_0001 (state: ACCEPTED)
19/12/05 22:25:30 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:30 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192..xx
ApplicationMaster RPC port: 0
queue: default
start time: 1575555910997
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1575555841019_0001/
user: root
19/12/05 22:25:31 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:32 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:33 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:34 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:35 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:36 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:37 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:38 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:39 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:40 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:41 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:42 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:43 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:44 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:45 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:46 INFO yarn.Client: Application report for application_1575555841019_0001 (state: RUNNING)
19/12/05 22:25:47 INFO yarn.Client: Application report for application_1575555841019_0001 (state: FINISHED)
19/12/05 22:25:47 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192..xx
ApplicationMaster RPC port: 0
queue: default
start time: 1575555910997
final status: SUCCEEDED
tracking URL: http://master:8088/proxy/application_1575555841019_0001/
user: root
19/12/05 22:25:47 INFO util.ShutdownHookManager: Shutdown hook called
19/12/05 22:25:47 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3a82aa7c-e7f6-4094-8ce0-f5a25f57d128
4、测试spark-shell提交yarn
[root@master bin]# ./spark-shell --master yarn --deploy-mode client
[root@master bin]# ./spark-shell --master yarn --deploy-mode client
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/12/05 22:37:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/12/05 22:38:01 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/12/05 22:38:38 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
19/12/05 22:38:39 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
19/12/05 22:38:40 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192..xx:4040
Spark context available as 'sc' (master = yarn, app id = application_1575555841019_0002).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141)
Type in expressions to have them evaluated.
Type :help for more information.
scala> 1+1
res0: Int = 2
scala>
5、提交yarn之后web图
|
