Apache Hadoop集群安装记录-基于CentOS虚拟机(2主4从)

2017-01-05 11:08:54来源:oschina作者:Goopand人点击


一、平台环境
1、VMware虚拟机

6台虚拟机,每台分配2CPU(线程)、4GB内存、50GB磁盘空间


2、CentOS 6.8系统

内核版本:2.6.32-642.el6.x86_64


3、集群网络配置

主机名
固定IP(Host-only 内网)
动态IP(NAT 外网)
用途
tpA
192.168.100.101
192.168.200.13x
Master1


tpB
192.168.100.102
192.168.200.13x
Master2


tpC
192.168.100.103
192.168.200.13x
Slave1


tpD
192.168.100.104
192.168.200.13x
Slave2


tpE
192.168.100.105
192.168.200.13x
Slave3


tpF
192.168.100.106
192.168.200.13x
Slave4

其中:


Master1承载Namenode


Master2承载SecondaryNamenode和ResourceManager


其余Slaves承载Datanode和NodeManager。

这里假设每台主机配置好了:


主机名(/etc/sysconfig/network)


网卡IP(/etc/sysconfig/network-scripts/ifcfg-eth*)


域名服务器(/etc/resolv.conf)


主机映射列表(/etc/hosts)


其中/etc/hosts文件配置如下:


127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.101 tpA
192.168.100.102 tpB
192.168.100.103 tpC
192.168.100.104 tpD
192.168.100.105 tpE
192.168.100.106 tpF
4、安装介质下载

分别从官网下载JDK(要求1.7以上)、Hadoop安装包:


JDK官网下载地址


软件版本:jdk-8u102-linux-x64.tar.gz


Hadoop官网下载地址


软件版本:hadoop-2.7.3.tar.gz

二、安装步骤
1、创建hadoop用户组、用户

分别在6台主机上创建hadoop用户组、用户,并创建相同密码:


[root@tpX ~]# groupadd hadoop
[root@tpX ~]# useradd -g hadoop -m -d /home/hadoop -s /bin/bash hadoop
[root@tpX ~]# passwd hadoop 2、关闭防火墙、SELINUX
[root@tpX ~]# service iptables stop
[root@tpX ~]# chkconfig iptables off
[root@tpX ~]# service ip6tables stop
[root@tpX ~]# chkconfig ip6tables off
[root@tpX ~]#
// 修改/etc/sysconfig/selinux配置文件
[root@tpX ~]# vim /etc/sysconfig/selinux
SELINUX=disabled
// 并执行如下命令
[root@tpX ~]# setenforce 0
[root@tpX ~]# getenforce
// 可选操作:关闭不需要的服务
[root@tpX ~]# for SERVICES in abrtd acpid auditd avahi-daemon cpuspeed haldaemon mdmonitor messagebus udev-post; do chkconfig ${SERVICES} off; done
// 如果安装了PackageKit,则按如下方法,设置enabled值为0,永久禁止packagekit运行
# vim /etc/yum/pluginconf.d/refresh-packagekit.conf
3、安装JDK、配置环境变量

以下依次在每台主机上操作


(1)以root用户解压、安装到/usr/local目录:


# cp jdk-8u102-linux-x64.tar.gz /usr/local
# cd /usr/local
# tar -zxvf jdk-8u102-linux-x64.tar.gz
# chown -R root:root jdk1.8.0_102

(2)在hadoop用户下配置Java环境变量并验证:


$ cd /home/hadoop
$ vi .bashrc
// 添加以下内容
export JAVA_HOME=/usr/local/jdk1.8.0_102
export PATH=${JAVA_HOME}/bin:${PATH}
export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar:${CLASSPATH}
$ . ./.bashrc
$ java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)$ javac -version
javac 1.8.0_102
$ echo $JAVA_HOME
/usr/local/jdk1.8.0_102

备注:Java 1.5以上版本,可以不配置CLASSPATH,只需配置JAVA_HOME和PATH即可。

4、配置SSH无密码互信访问

以下同样在每一台主机上操作:


(1)SSH无密码访问原理


搭建Hadoop环境需要设置无密码登陆,所谓无密码登陆其实是指通过证书认证的方式登陆,使用一种被称为“公私钥”认证的方式来进行ssh登录。 在Linux系统中,ssh是远程登录的默认工具,因为该工具的协议使用了RSA/DSA的加密算法。该工具做Linux系统的远程管理是非常安全的。Telnet因为其不安全性,在最新的Linux系统中被弃用了。 “公私钥”认证方式简单的解释:首先在客户端上创建一对公私钥(公钥文件:~/.ssh/id_rsa.pub ;私钥文件:~/.ssh/id_rsa)。然后把公钥放到服务器上(~/.ssh/authorized_keys),自己保留好私钥。在使用ssh登录时,ssh程序会发送私钥去和服务器上的公钥做匹配。如果匹配成功就可以登录了。


(2)先确保ssh的server和client,以及rsync已经安装


# rpm -qa | grep openssh
openssh-5.3p1-117.el6.x86_64
openssh-askpass-5.3p1-117.el6.x86_64
openssh-server-5.3p1-117.el6.x86_64
openssh-clients-5.3p1-117.el6.x86_64
# rpm -qa | grep rsync
rsync-3.0.6-12.el6.x86_64
// 如果没有安装,可以通过如下命令进行安装
# yum install openssh#安装SSH协议
# yum install rsync#(rsync是一个远程数据同步工具,可通过LAN/WAN快速同步多台主机间的文件)
# service sshd restart #重启服务

(3)修改sshd配置文件,并验证(开启RSAAuthentication、PubkeyAuthentication、AuthorizedKeysFile)


# vi /etc/ssh/sshd_config
// 找到如下内容,并把开头的注释#去掉
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
// 修改了配置文件之后,需要重启sshd服务
# service sshd restart
// 在hadoop用户下,测试连接本机
# su - hadoop
$ ssh localhost
// 会提示输入密码,因为此时尚未生成证书

(4)生成证书“公私钥”


// 在hadoop用户下执行
// 或者直接执行ssh-keygen -t rsa -P ''命令,询问保存路径时直接回车采用默认路径存储在"/home/hadoop/.ssh"目录下
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
// 注意,一定要把.ssh/authorized_keys文件的权限设置为600。
// 否则,ssh连接本机、远端主机,仍然需要密码!这是Linux安全机制决定的。
$ chmod 600 ~/.ssh/authorized_keys
// 注意,如果报和.ssh目录权限相关的报错,则按如下修改
$ chmod 700 ~/.ssh

A、测试ssh连接本机:


// 测试ssh连接本机成功
$ ssh localhost
Last login: Fri Aug 26 22:05:47 2016 from localhost

B、测试ssh连接远程服务器:


(注:远程服务器OpenSSH服务当然要开启)


先拷贝本机产生的key文件,到远程服务器centX(这里X是指其余3台主机,因此以下要执行三次)


$ scp ~/.ssh/id_rsa.pub hadoop@centX:/home/hadoop/
// 登录远程服务器centX,执行如下命令
$ cat id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys
$ rm id_rsa.pub

小结,最终目的是把本机的id_rsa.pub公钥文件,放在本机的.ssh/authorized_keys文件中,同时放到另外三台主机的.ssh/authorized_keys文件中。大家互相拥有彼此的公钥!


[备注1] 可以使用ssh-copy-id命令工具,添加公钥到远程的~/.ssh/authorized_keys。


// 用法如下 RSA
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@centX

[备注2] 可以使用Linux自带的rsync工具来完成秘钥文件的同步。


[备注3] 另外可以使用Greenplum的gpssh-exkeys工具来完成。


[附注] 关于SSH秘钥的类型说明(rsa、dsa 或者 ecc),可以参考:


SSH密钥类型的的选择(RSA,DSAorOther)

5、Hadoop安装、配置、验证

(1)解压hadoop安装包到/usr/local,设置属主(master1主机)


# cp hadoop-2.7.3.tar.gz /usr/local
# cd /usr/local
# tar -zxvf hadoop-2.7.3.tar.gz
# rm -f hadoop-2.7.3.tar.gz
# ln -s hadoop-2.7.3 hadoop
# chown -R hadoop:hadoop hadoop*

(2)配置hadoop用户的环境变量(所有主机)


注意:Hadoop 2.0及以上的版本,MapReduce运行在YARN之上,配置文件的目录由原先的conf子目录,统一改为etc/hadoop子目录。参考:hadoop-release-missing-conf-directory


In Hadoop 2.0 and later, MapReduce runs on YARN and there is an additional configuration file called yarn-site.xml. All the configuration files should go in the etc/hadoop subdirectory. // vim编辑hadoop用户下的.bashrc文件,添加环境变量
$ vim ~/.bashrc
export HADOOP_HOME=/usr/local/hadoop
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${PATH}$ source ~/.bashrc

(3)创建相关目录name、data、temp


# mkdir -p /appdata/hadoop/name#Master1和Master2
# mkdir -p /appdata/hadoop/data#Slaves
# mkdir -p /appdata/hadoop/temp#Master1、Master2和Slaves
# mkdir -p /appdata/hadoop/logs#Master1、Master2和Slaves
# mkdir -p /appdata/hadoop/pids#Master1、Master2和Slaves,存放pid文件
# chown -R hadoop:hadoop /appdata/hadoop

备注:创建temp的原因在于,如没有配置该temp临时目录,系统默认以/tmp/hadoop-hadoop作为临时目录。这样在每次主机重启后有可能被清除,这种情况只有重新format才行,否则启动hadoop会有报错。


(4)配置Hadoop,涉及到$HADOOP_HOME/etc/hadoop目录下的8个文件:(master1主机)


hadoop-env.sh mapred-env.sh yarn-env.sh slaves core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml


A、修改hadoop-env.sh、mapred-env.sh、yarn-env.sh中的JAVA_HOME、LOG配置


$ vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh
$ vim $HADOOP_HOME/etc/hadoop/mapred-env.sh
$ vim $HADOOP_HOME/etc/hadoop/yarn-env.sh
// hadoop-env.sh涉及修改内容如下
export JAVA_HOME=/usr/local/jdk1.8.0_102
export HADOOP_LOG_DIR=/appdata/hadoop/logs
export HADOOP_PID_DIR=/appdata/hadoop/pids
// mapred-env.sh涉及修改内容如下
export JAVA_HOME=/usr/local/jdk1.8.0_102
export HADOOP_MAPRED_LOG_DIR=/appdata/hadoop/logs
export HADOOP_MAPRED_PID_DIR=/appdata/hadoop/pids
// yarn-env.sh涉及修改内容如下
export JAVA_HOME=/usr/local/jdk1.8.0_102
if [ "$YARN_LOG_DIR" = "" ]; then
YARN_LOG_DIR=/appdata/hadoop/logs
fi

B、配置slaves


tpC
tpD
tpE
tpF

C、配置core-site.xml




fs.defaultFS
hdfs://tpA:9000


io.file.buffer.size
131072


hadoop.tmp.dir
file:/appdata/hadoop/temp
A base for other temporary directories.


D、配置hdfs-site.xml




dfs.namenode.secondary.http-address
tpB:9001


dfs.namenode.name.dir
file:/appdata/hadoop/name


dfs.datanode.data.dir
file:/appdata/hadoop/data


dfs.replication
3


dfs.webhdfs.enabled
true


E、配置mapred-site.xml如下


$ cp mapred-site.xml.template mapred-site.xml
$ vim mapred-site.xml


mapreduce.framework.name
yarn


mapreduce.jobhistory.address
tpA:10020


mapreduce.jobhistory.webapp.address
tpA:19888


F、配置yarn-site.xml




yarn.nodemanager.aux-services
mapreduce_shuffle


yarn.nodemanager.aux-services.mapreduce_shuffle.class
org.apache.hadoop.mapred.ShuffleHandler


yarn.resourcemanager.hostname
tpB


yarn.resourcemanager.scheduler.address
tpB:8030


yarn.resourcemanager.resource-tracker.address
tpB:8031


yarn.resourcemanager.address
tpB:8032


yarn.resourcemanager.admin.address
tpB:8033


yarn.resourcemanager.webapp.address
tpB:8088


G、打包、复制/usr/local/hadoop-2.7.3到其他节点(master2、slaves)


# cd /usr/local
# tar -cvf hadoop-2.7.3.tar hadoop-2.7.3
# scp hadoop-2.7.3.tar root@tpX:/usr/local
# ssh tpX
# cd /usr/local
# tar -xvf hadoop-2.7.3.tar
# ln -s hadoop-2.7.3 hadoop
# chown -R hadoop:hadoop hadoop*
# rm -f hadoop-2.7.3.tar

(5)启动及验证


A、格式化NameNode


[hadoop@tpA ~]$ hdfs namenode -format
16/09/26 14:04:23 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = tpA/192.168.100.101
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.3
STARTUP_MSG: classpath = /usr/local/hadoop-2.7.3/etc/hadoop::/usr/local/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z
STARTUP_MSG: java = 1.8.0_102
************************************************************/
16/09/26 14:04:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/09/26 14:04:23 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-66c5da4e-770e-4941-b5fe-7f89c0ad5533
16/09/26 14:04:23 INFO namenode.FSNamesystem: No KeyProvider found.
16/09/26 14:04:23 INFO namenode.FSNamesystem: fsLock is fair:true
16/09/26 14:04:23 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/09/26 14:04:23 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/09/26 14:04:23 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/09/26 14:04:23 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Sep 26 14:04:23
16/09/26 14:04:23 INFO util.GSet: Computing capacity for map BlocksMap
16/09/26 14:04:23 INFO util.GSet: VM type= 64-bit
16/09/26 14:04:23 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
16/09/26 14:04:23 INFO util.GSet: capacity= 2^21 = 2097152 entries
16/09/26 14:04:23 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/09/26 14:04:23 INFO blockmanagement.BlockManager: defaultReplication= 3
16/09/26 14:04:23 INFO blockmanagement.BlockManager: maxReplication= 512
16/09/26 14:04:23 INFO blockmanagement.BlockManager: minReplication= 1
16/09/26 14:04:23 INFO blockmanagement.BlockManager: maxReplicationStreams= 2
16/09/26 14:04:23 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/09/26 14:04:23 INFO blockmanagement.BlockManager: encryptDataTransfer = false
16/09/26 14:04:23 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
16/09/26 14:04:23 INFO namenode.FSNamesystem: fsOwner= hadoop (auth:SIMPLE)
16/09/26 14:04:23 INFO namenode.FSNamesystem: supergroup = supergroup
16/09/26 14:04:23 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/09/26 14:04:23 INFO namenode.FSNamesystem: HA Enabled: false
16/09/26 14:04:23 INFO namenode.FSNamesystem: Append Enabled: true
16/09/26 14:04:23 INFO util.GSet: Computing capacity for map INodeMap
16/09/26 14:04:23 INFO util.GSet: VM type= 64-bit
16/09/26 14:04:23 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
16/09/26 14:04:23 INFO util.GSet: capacity= 2^20 = 1048576 entries
16/09/26 14:04:23 INFO namenode.FSDirectory: ACLs enabled? false
16/09/26 14:04:23 INFO namenode.FSDirectory: XAttrs enabled? true
16/09/26 14:04:23 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
16/09/26 14:04:23 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/26 14:04:23 INFO util.GSet: Computing capacity for map cachedBlocks
16/09/26 14:04:23 INFO util.GSet: VM type= 64-bit
16/09/26 14:04:23 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
16/09/26 14:04:23 INFO util.GSet: capacity= 2^18 = 262144 entries
16/09/26 14:04:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/09/26 14:04:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/09/26 14:04:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
16/09/26 14:04:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
16/09/26 14:04:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
16/09/26 14:04:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
16/09/26 14:04:23 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/09/26 14:04:23 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/09/26 14:04:23 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/09/26 14:04:23 INFO util.GSet: VM type= 64-bit
16/09/26 14:04:23 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
16/09/26 14:04:23 INFO util.GSet: capacity= 2^15 = 32768 entries
16/09/26 14:04:23 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1954606952-192.168.100.101-1474869863868
16/09/26 14:04:23 INFO common.Storage: Storage directory /appdata/hadoop/name has been successfully formatted.
16/09/26 14:04:23 INFO namenode.FSImageFormatProtobuf: Saving image file /appdata/hadoop/name/current/fsimage.ckpt_0000000000000000000 using no compression
16/09/26 14:04:23 INFO namenode.FSImageFormatProtobuf: Image file /appdata/hadoop/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds.
16/09/26 14:04:24 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/09/26 14:04:24 INFO util.ExitUtil: Exiting with status 0
16/09/26 14:04:24 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at tpA/192.168.100.101
************************************************************/

"Exitting with status 0" 表示成功,"Exitting with status 1" 表示失败。


在这一步完成之后,可以看到Master节点主机的/appdata/hadoop/hdfs/namenode下面已经生成相关文件:


[hadoop@tpA ~]$ ll /appdata/hadoop/*/*/*
-rw-rw-r-- 1 hadoop hadoop 353 Sep 26 13:19 /appdata/hadoop/name/current/fsimage_0000000000000000000
-rw-rw-r-- 1 hadoop hadoop62 Sep 26 13:19 /appdata/hadoop/name/current/fsimage_0000000000000000000.md5
-rw-rw-r-- 1 hadoop hadoop 2 Sep 26 13:19 /appdata/hadoop/name/current/seen_txid
-rw-rw-r-- 1 hadoop hadoop 207 Sep 26 13:19 /appdata/hadoop/name/current/VERSION

而Master2、Slave节点则暂时没有任何文件生成。


B、启动HDFS服务进程


执行命令:start-dfs.sh


[hadoop@tpA ~]$ start-dfs.sh
Starting namenodes on [tpA]
tpA: starting namenode, logging to /appdata/hadoop/logs/hadoop-hadoop-namenode-tpA.out
tpC: starting datanode, logging to /appdata/hadoop/logs/hadoop-hadoop-datanode-tpC.out
tpD: starting datanode, logging to /appdata/hadoop/logs/hadoop-hadoop-datanode-tpD.out
tpF: starting datanode, logging to /appdata/hadoop/logs/hadoop-hadoop-datanode-tpF.out
tpE: starting datanode, logging to /appdata/hadoop/logs/hadoop-hadoop-datanode-tpE.out
Starting secondary namenodes [tpB]
tpB: starting secondarynamenode, logging to /appdata/hadoop/logs/hadoop-hadoop-secondarynamenode-tpB.out

这一步完成之后,可以看到,在各个主机上都有hadoop的进程启动了:


Master1主机:NameNode


Master2主机:SecondaryNameNode


Slaves主机:DataNode


同时,Master节点主机上namenode目录下生成了新的文件,而Slave节点主机上datanode目录下生成了文件。


[hadoop@tpA ~]$ ls -lrt /appdata/hadoop/*/*/*
-rw-rw-r-- 1 hadoop hadoop 208 Sep 26 14:04 /appdata/hadoop/name/current/VERSION
-rw-rw-r-- 1 hadoop hadoop 353 Sep 26 14:04 /appdata/hadoop/name/current/fsimage_0000000000000000000
-rw-rw-r-- 1 hadoop hadoop62 Sep 26 14:04 /appdata/hadoop/name/current/fsimage_0000000000000000000.md5
-rw-rw-r-- 1 hadoop hadoop42 Sep 26 14:08 /appdata/hadoop/name/current/edits_0000000000000000001-0000000000000000002
-rw-rw-r-- 1 hadoop hadoop 1048576 Sep 26 14:08 /appdata/hadoop/name/current/edits_inprogress_0000000000000000003
-rw-rw-r-- 1 hadoop hadoop2 Sep 26 14:08 /appdata/hadoop/name/current/seen_txid
-rw-rw-r-- 1 hadoop hadoop 353 Sep 26 14:08 /appdata/hadoop/name/current/fsimage_0000000000000000002
-rw-rw-r-- 1 hadoop hadoop62 Sep 26 14:08 /appdata/hadoop/name/current/fsimage_0000000000000000002.md5
[hadoop@tpB ~]$ ls -lrt /appdata/hadoop/*/*/*
total 8
-rw-rw-r-- 1 hadoop hadoop8 Sep 26 14:07 in_use.lock
drwxrwxr-x 2 hadoop hadoop 4096 Sep 26 14:08 current
[hadoop@tpX ~]$ ls -lrt /appdata/hadoop/*/*/*
-rw-rw-r-- 1 hadoop hadoop229 Sep 26 14:07 /appdata/hadoop/data/current/VERSION
/appdata/hadoop/data/current/BP-1954606952-192.168.100.101-1474869863868:
total 12
drwxrwxr-x 2 hadoop hadoop 4096 Sep 26 14:07 tmp
drwxrwxr-x 4 hadoop hadoop 4096 Sep 26 14:07 current
-rw-rw-r-- 1 hadoop hadoop166 Sep 26 14:07 scanner.cursor

C、启动YARN服务进程


执行命令:start-yarn.sh


[hadoop@tpB ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-resourcemanager-tpB.out
tpC: starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-tpC.out
tpD: starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-tpD.out
tpF: starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-tpF.out
tpE: starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-tpE.out

这一步完成之后,可以看到,非Master1主机上都有YARN相关的进程启动了:


Master2主机:ResourceManager


Slaves主机:NodeManager


同时,Slave节点主机上的/appdata/hadoop目录下有新增目录。


[hadoop@tpX ~]$ ls -lrt /appdata/hadoop/*/*/*
-rw-rw-r-- 1 hadoop hadoop229 Sep 26 14:07 /appdata/hadoop/data/current/VERSION
/appdata/hadoop/data/current/BP-1954606952-192.168.100.101-1474869863868:
total 12
drwxrwxr-x 2 hadoop hadoop 4096 Sep 26 14:07 tmp
drwxrwxr-x 4 hadoop hadoop 4096 Sep 26 14:07 current
-rw-rw-r-- 1 hadoop hadoop166 Sep 26 14:07 scanner.cursor
/appdata/hadoop/temp/nm-local-dir/nmPrivate:
total 0
/appdata/hadoop/temp/nm-local-dir/filecache:
total 0
/appdata/hadoop/temp/nm-local-dir/usercache:

D、验证服务状态


执行命令:jps


[hadoop@tpA ~]$ jps
15689 NameNode
16415 Jps
[hadoop@tpB ~]$ jps
9160 SecondaryNameNode
9387 ResourceManager
9675 Jps
[hadoop@tpX ~]$ jps
9027 DataNode
9449 NodeManager
9597 Jps

E、查看Hadoop集群状态


执行命令:hadoop dfsadmin -report


[hadoop@tpA ~]$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Configured Capacity: 192444596224 (179.23 GB)
Present Capacity: 155677032448 (144.99 GB)
DFS Remaining: 155676934144 (144.99 GB)
DFS Used: 98304 (96 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (4):
Name: 192.168.100.104:50010 (tpD)
Hostname: tpD
Decommission Status : Normal
Configured Capacity: 48111149056 (44.81 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 9192091648 (8.56 GB)
DFS Remaining: 38919032832 (36.25 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.89%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Sep 26 14:14:59 CST 2016Name: 192.168.100.106:50010 (tpF)
Hostname: tpF
Decommission Status : Normal
Configured Capacity: 48111149056 (44.81 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 9191473152 (8.56 GB)
DFS Remaining: 38919651328 (36.25 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.90%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Sep 26 14:14:59 CST 2016Name: 192.168.100.103:50010 (tpC)
Hostname: tpC
Decommission Status : Normal
Configured Capacity: 48111149056 (44.81 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 9192235008 (8.56 GB)
DFS Remaining: 38918889472 (36.25 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.89%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Sep 26 14:14:59 CST 2016Name: 192.168.100.105:50010 (tpE)
Hostname: tpE
Decommission Status : Normal
Configured Capacity: 48111149056 (44.81 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 9191763968 (8.56 GB)
DFS Remaining: 38919360512 (36.25 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.89%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Sep 26 14:14:59 CST 2016

F、执行Hadoop测试用例


[hadoop@tpA ~]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 2 5
Number of Maps= 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
16/09/26 14:15:31 INFO client.RMProxy: Connecting to ResourceManager at tpB/192.168.100.102:8032
16/09/26 14:15:32 INFO input.FileInputFormat: Total input paths to process : 2
16/09/26 14:15:32 INFO mapreduce.JobSubmitter: number of splits:2
16/09/26 14:15:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1474870368029_0001
16/09/26 14:15:33 INFO impl.YarnClientImpl: Submitted application application_1474870368029_0001
16/09/26 14:15:33 INFO mapreduce.Job: The url to track the job: http://tpB:8088/proxy/application_1474870368029_0001/
16/09/26 14:15:33 INFO mapreduce.Job: Running job: job_1474870368029_0001
16/09/26 14:15:41 INFO mapreduce.Job: Job job_1474870368029_0001 running in uber mode : false
16/09/26 14:15:41 INFO mapreduce.Job:map 0% reduce 0%
16/09/26 14:15:49 INFO mapreduce.Job:map 100% reduce 0%
16/09/26 14:15:56 INFO mapreduce.Job:map 100% reduce 100%
16/09/26 14:15:57 INFO mapreduce.Job: Job job_1474870368029_0001 completed successfully
16/09/26 14:15:57 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=50
FILE: Number of bytes written=356261
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=518
HDFS: Number of bytes written=215
HDFS: Number of read operations=11
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=10842
Total time spent by all reduces in occupied slots (ms)=4575
Total time spent by all map tasks (ms)=10842
Total time spent by all reduce tasks (ms)=4575
Total vcore-milliseconds taken by all map tasks=10842
Total vcore-milliseconds taken by all reduce tasks=4575
Total megabyte-milliseconds taken by all map tasks=11102208
Total megabyte-milliseconds taken by all reduce tasks=4684800
Map-Reduce Framework
Map input records=2
Map output records=4
Map output bytes=36
Map output materialized bytes=56
Input split bytes=282
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=56
Reduce input records=4
Reduce output records=0
Spilled Records=8
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=454
CPU time spent (ms)=3380
Physical memory (bytes) snapshot=693272576
Virtual memory (bytes) snapshot=6266691584
Total committed heap usage (bytes)=478150656
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=236
File Output Format Counters
Bytes Written=97
Job Finished in 25.941 seconds
Estimated value of Pi is 3.60000000000000000000

G、停止Hadoop集群


执行命令:stop-yarn.sh 和 stop-dfs.sh


[hadoop@tpB ~]$ stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
tpD: stopping nodemanager
tpC: stopping nodemanager
tpF: stopping nodemanager
tpE: stopping nodemanager
tpC: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
tpF: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop[hadoop@tpA ~]$ stop-dfs.sh
Stopping namenodes on [tpA]
tpA: stopping namenode
tpC: stopping datanode
tpD: stopping datanode
tpE: stopping datanode
tpF: stopping datanode
Stopping secondary namenodes [tpB]
tpB: stopping secondarynamenode

最后可以看到,各个节点主机上的hadoop进程都停止了。

配置文件中涉及到的端口:


[hadoop@tpA ~]$ grep ":[0-9]" /usr/local/hadoop/etc/hadoop/* | grep "value"
/usr/local/hadoop/etc/hadoop/core-site.xml: hdfs://tpA:9000
/usr/local/hadoop/etc/hadoop/hdfs-site.xml:tpB:9001
/usr/local/hadoop/etc/hadoop/mapred-site.xml:tpA:10020
/usr/local/hadoop/etc/hadoop/mapred-site.xml:tpA:19888
/usr/local/hadoop/etc/hadoop/yarn-site.xml:tpB:8030
/usr/local/hadoop/etc/hadoop/yarn-site.xml:tpB:8031
/usr/local/hadoop/etc/hadoop/yarn-site.xml:tpB:8032
/usr/local/hadoop/etc/hadoop/yarn-site.xml:tpB:8033
/usr/local/hadoop/etc/hadoop/yarn-site.xml:tpB:8088

尝试打开Web管理页面:


①Master1的几个端口均无内容:



②Master2的管理端口:



最新文章

123

最新摄影

微信扫一扫

第七城市微信公众平台