Hadoop2.0 配置yarn成功

114次阅读

没有评论

共计 3384 个字符，预计需要花费 9 分钟才能阅读完成。

Hadoop2.0 配置 yarn 成功，记住这个成功的一刻，把几个主要文件贴出来：

————————————– 分割线 ————————————–

相关阅读：

Ubuntu 13.04 上搭建 Hadoop 环境 http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu 12.10 +Hadoop 1.2.1 版本集群配置 http://www.linuxidc.com/Linux/2013-09/90600.htm

Ubuntu 上搭建 Hadoop 环境（单机模式 + 伪分布模式）http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu 下 Hadoop 环境的配置 http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建 Hadoop 环境图文教程详解 http://www.linuxidc.com/Linux/2012-02/53927.htm

Hadoop LZO 安装教程 http://www.linuxidc.com/Linux/2013-01/78397.htm

Hadoop 集群上使用 Lzo 压缩 http://www.linuxidc.com/Linux/2012-05/60554.htm

————————————– 分割线 ————————————–

core-site.xml:

<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”https://www.linuxidc.com/Linux/2014-05/configuration.xsl”?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://fc20:9000</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/home/ljq/hadoop/tmp</value>
</property>

<property>
<name>hadoop.native.lib</name>
<value>false</value>
<description>Should native hadoop libraries, if present, be used.</description>
</property>
</configuration>

hdfs-site.xml:

<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”https://www.linuxidc.com/Linux/2014-05/configuration.xsl”?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>/home/ljq/hadoop/dfs/name</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>/home/ljq/hadoop/dfs/data</value>
</property>
</configuration>

mapred-site.xml:

<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”https://www.linuxidc.com/Linux/2014-05/configuration.xsl”?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value> 一定是小写 yarn 否则出现错误:java.lang.IllegalStateException: Invalid shuffle port number -1
</property>

</configuration>

yarn-site.xml: 文件里所有 yarn 都必须小写, 否则相应的端口信息找不到, 会在默认端口上启动进程

<?xml version=”1.0″?>
<configuration>

<!– Site specific YARN configuration properties –>

<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>fc20</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>

<property>
<description>The address of the applications manager interface in the RM.</description>
<name>yarn.resourcemanager.address</name>
<value>fc20:18004</value>
</property>

<property>
<description>The address of the scheduler interface.</description>
<name>yarn.resourcemanager.scheduler.address</name>
<value>fc20:18003</value>
</property>

<property>
<description>The address of the RM web application.</description>
<name>yarn.resourcemanager.webapp.address</name>
<value>fc20:18008</value>
</property>

<property>
<description>The address of the resource tracker interface.</description>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>fc20:18006</value>
</property>