Hadoop环境搭建与测试

375次阅读

没有评论

共计 2800 个字符，预计需要花费 7 分钟才能阅读完成。

Hadoop 环境搭建请参看下面的连接：

CentOS 安装和配置 Hadoop2.2.0 http://www.linuxidc.com/Linux/2014-01/94685.htm

Ubuntu 13.04 上搭建 Hadoop 环境 http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu 12.10 +Hadoop 1.2.1 版本集群配置 http://www.linuxidc.com/Linux/2013-09/90600.htm

Ubuntu 上搭建 Hadoop 环境（单机模式 + 伪分布模式）http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu 下 Hadoop 环境的配置 http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建 Hadoop 环境图文教程详解 http://www.linuxidc.com/Linux/2012-02/53927.htm

搭建 Hadoop 环境（在 Winodws 环境下用虚拟机虚拟两个 Ubuntu 系统进行搭建）http://www.linuxidc.com/Linux/2011-12/48894.htm

查看集群状态：

[root@master bin]# hdfs dfsadmin -report
Configured Capacity: 36729053184 (34.21 GB)
Present Capacity: 13322559491 (12.41 GB)
DFS Remaining: 13322240000 (12.41 GB)
DFS Used: 319491 (312.00 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

————————————————-
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: 192.168.137.103:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 18364526592 (17.10 GB)
DFS Used: 45056 (44 KB)
Non DFS Used: 11702558720 (10.90 GB)
DFS Remaining: 6661922816 (6.20 GB)
DFS Used%: 0.00%
DFS Remaining%: 36.28%
Last contact: Thu Nov 06 21:26:34 CST 2014

Name: 192.168.137.102:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 18364526592 (17.10 GB)
DFS Used: 274435 (268.00 KB)
Non DFS Used: 11703934973 (10.90 GB)
DFS Remaining: 6660317184 (6.20 GB)
DFS Used%: 0.00%
DFS Remaining%: 36.27%
Last contact: Thu Nov 06 21:26:31 CST 2014

查看文件块组成：

[root@master bin]# hdfs fsck / -files -blocks
Status: HEALTHY
Total size: 219351 B
Total dirs: 11
Total files: 12
Total symlinks: 0
Total blocks (validated): 10 (avg. block size 21935 B)
Minimally replicated blocks: 10 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 1
Average block replication: 1.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Thu Nov 06 21:27:34 CST 2014 in 29 milliseconds

The filesystem under path ‘/’ is HEALTHY
[root@master bin]#
[hadoop@master hadoop2.2]$

查看各节点状态：
http://192.168.56.101:50070
查看 ResourceManager 上 cluster 运行状态：
http://192.168.56.101:8088

在环境搭建的过程中，如果出现任何问题，都要去查看日志
日志路径是：/home/hadoop/hadoop2.2/logs

在配置完成 HADOOP_HOME 之后，并且使之生效，那么接下来就进行测试，启动 hadoop
首先在 / 目录下创建 input 文件

[root@master /]# vim input，在该文件中输入如下内容：I am a very good person! I love you America !
将之上传到 hdfs 上：[root@master /]# hadoop fs -put /input /input
在 hadoop 的 bin 目录下执行：[root@master bin]# ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output
这样执行完毕之后可以看到：output 文件夹下有两个文件
[root@master ~]# hadoop fs -ls /output
Found 2 items
-rw-r–r– 1 root supergroup 0 2014-11-06 21:21 /output/_SUCCESS
-rw-r–r– 1 root supergroup 64 2014-11-06 21:21 /output/part-r-00000
接着可以查看 Wordcount 的统计结果：
[root@master bin]# hadoop fs -cat /output/part-r-00000
! 1
America 1
I 2
a 1
am 1
good 1
love 1
person! 1
very 1
you 1
[root@master bin]#