阿里云-云小站(无限量代金券发放中)
【腾讯云】云服务器、云数据库、COS、CDN、短信等热卖云产品特惠抢购

64位Ubuntu下重新编译Hadoop2.2.0步骤

464次阅读
没有评论

共计 11215 个字符,预计需要花费 29 分钟才能阅读完成。

Hadoop 官方网站中只提供了 32 位的 Hadoop-2.2.0.tar.gz,如果要在 64 位 Ubuntu 下部署 Hadoop-2.2.0,就需要重新编译源码包,生成 64 位的部署包。

建议以下操作使用 root 账户,避免出现权限不足的问题。

Linux 系统:64 位 Ubuntu 14.04.1

安装 JDK

请参考文章《Ubuntu 14.04.1 中安装 JDK》http://www.linuxidc.com/Linux/2015-03/114618.htm。

安装 Maven

请参考文章《Ubuntu 14.04.1 中安装 Maven》http://www.linuxidc.com/Linux/2015-03/114619.htm。

下载 Hadoop 源码

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz

解压

tar -xzf hadoop-2.2.0-src.tar.gz

编译源代码

cd hadoop-2.2.0-src
mvn package -Pdist,native -DskipTests -Dtar

第 1 次编译:失败(hadoop pom.xml 的 bug)

错误信息:

[ERROR] Failed to execute goal on project hadoop-auth: Could not resolve dependencies for project org.apache.hadoop:hadoop-auth:jar:2.2.0: Could not transfer artifact org.mortbay.jetty:jetty:jar:6.1.26 from/to central (https://repo.maven.apache.org/maven2): GET request of: org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar from central failed: SSL peer shut down incorrectly -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-auth

解决办法:
这是 hadoop 的一个 bug,在 pom.xml 中添加下面 patch 即可,详见 https://issues.apache.org/jira/browse/HADOOP-10110。

编辑 `hadoop-common-project/hadoop-auth/pom.xml` 文件:

vi hadoop-common-project/hadoop-auth/pom.xml

<dependencys></dependencys> 节点中插入:

<dependency>
  <groupId>org.mortbay.jetty</groupId>
  <artifactId>jetty-util</artifactId>
  <scope>test</scope>
</dependency>

第 2 次编译:失败(未安装 protoc)

错误信息:

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.2.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -&gt; [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-common

解决办法:

根据错误信息可以知道是因为没有安装 protoc。

wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar -xzf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure
make
make check
make install

其中,在执行./configure 命令是会报如下错误:

checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... no
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/home/hadoop/protobuf-2.5.0':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

提示我们找不到 C 编译器,因此我们还需要安装 C 编译器。

ubuntu 提供了集成 gcc 等编译器的基本编译工具 `build-essential`,安装起来也比较方便,只需要一行命令:

apt-get install build-essential

安装过程中可能会提示包找不到,建议先更新下软件源:

apt-get update

安装之后验证 protobuf 的时候可能会报错以下错误:

$ protoc --version
protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory

解决如下:

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
$ protoc --version
libprotoc 2.5.0

第 3 次编译:失败(未安装 cmake)

错误信息:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/home/hadoop/hadoop-2.2.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory -&gt; [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-common

解决办法:

apt-get install cmake

第 4 次编译:失败(未安装 libglib2.0-dev)

错误信息:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 -&gt; [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-common

解决办法:

apt-get install libglib2.0-dev

第 5 次编译:失败(未安装 libssl-dev)

错误信息:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1 -&gt; [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-pipes

解决办法:

apt-get install libssl-dev

第 6 次编译:成功

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [13.578 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [5.183 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [9.527 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [1.268 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [4.717 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [9.966 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [7.368 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [3.971 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [02:27 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [14.996 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [0.078 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:32 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [30.260 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [19.083 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [8.313 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [0.071 s]
[INFO] hadoop-yarn ........................................ SUCCESS [0.542 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [01:07 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [48.948 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [0.314 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [18.413 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [23.891 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [5.687 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [24.345 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [0.721 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [8.261 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [0.168 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [6.632 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [0.261 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [40.147 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [3.497 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [0.164 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [6.054 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [29.892 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [5.450 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [18.558 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [9.045 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [7.740 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [2.819 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [12.523 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [5.321 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [8.999 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [13.044 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [3.739 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [11.307 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [8.223 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [6.296 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [6.341 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [14.662 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [2.694 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [0.063 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [44.996 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [16.908 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [5.014 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:23 min
[INFO] Finished at: 2014-10-04T14:54:28+08:00
[INFO] Final Memory: 69M/215M
[INFO] ------------------------------------------------------------------------

编译成果

编译生产的文件在 `~/hadoop-2.2.0-src/hadoop-dist/target` 目录中。

$ ls ~/hadoop-2.2.0-src/hadoop-dist/target
antrun hadoop-2.2.0 hadoop-dist-2.2.0-javadoc.jar test-dir
dist-layout-stitching.sh hadoop-2.2.0.tar.gz javadoc-bundle-options
dist-tar-stitching.sh hadoop-dist-2.2.0.jar maven-archiver

其中 hadoop-2.2.0 是编译后的文件夹,hadoop-2.2.0.tar.gz 是编译后的打包文件。

验证

$ cd ~/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib/native/
$ file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb43b4ebd092ae8b4a427719b8907e6fdb223ed9, not stripped

可以看到,libhadoop.so.1.0.0 已经是 64 位的了。

拷贝

将编译好的 64 位 hadoop-2.2.0.tar.gz 部署包,拷贝到当前用户目录。

cp ~/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0.tar.gz ~

Ubuntu14.04 下 Hadoop2.4.1 单机 / 伪分布式安装配置教程  http://www.linuxidc.com/Linux/2015-02/113487.htm

CentOS 安装和配置 Hadoop2.2.0  http://www.linuxidc.com/Linux/2014-01/94685.htm

Ubuntu 13.04 上搭建 Hadoop 环境 http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu 12.10 +Hadoop 1.2.1 版本集群配置 http://www.linuxidc.com/Linux/2013-09/90600.htm

Ubuntu 上搭建 Hadoop 环境(单机模式 + 伪分布模式)http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu 下 Hadoop 环境的配置 http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建 Hadoop 环境图文教程详解 http://www.linuxidc.com/Linux/2012-02/53927.htm

搭建 Hadoop 环境(在 Winodws 环境下用虚拟机虚拟两个 Ubuntu 系统进行搭建)http://www.linuxidc.com/Linux/2011-12/48894.htm

更多 Hadoop 相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

正文完
星哥玩云-微信公众号
post-qrcode
 0
星锅
版权声明:本站原创文章,由 星锅 于2022-01-20发表,共计11215字。
转载说明:除特殊说明外本站文章皆由CC-4.0协议发布,转载请注明出处。
【腾讯云】推广者专属福利,新客户无门槛领取总价值高达2860元代金券,每种代金券限量500张,先到先得。
阿里云-最新活动爆款每日限量供应
评论(没有评论)
验证码
【腾讯云】云服务器、云数据库、COS、CDN、短信等云产品特惠热卖中

星哥玩云

星哥玩云
星哥玩云
分享互联网知识
用户数
4
文章数
19351
评论数
4
阅读量
7994157
文章搜索
热门文章
星哥带你玩飞牛NAS-6:抖音视频同步工具,视频下载自动下载保存

星哥带你玩飞牛NAS-6:抖音视频同步工具,视频下载自动下载保存

星哥带你玩飞牛 NAS-6:抖音视频同步工具,视频下载自动下载保存 前言 各位玩 NAS 的朋友好,我是星哥!...
星哥带你玩飞牛NAS-3:安装飞牛NAS后的很有必要的操作

星哥带你玩飞牛NAS-3:安装飞牛NAS后的很有必要的操作

星哥带你玩飞牛 NAS-3:安装飞牛 NAS 后的很有必要的操作 前言 如果你已经有了飞牛 NAS 系统,之前...
我把用了20年的360安全卫士卸载了

我把用了20年的360安全卫士卸载了

我把用了 20 年的 360 安全卫士卸载了 是的,正如标题你看到的。 原因 偷摸安装自家的软件 莫名其妙安装...
再见zabbix!轻量级自建服务器监控神器在Linux 的完整部署指南

再见zabbix!轻量级自建服务器监控神器在Linux 的完整部署指南

再见 zabbix!轻量级自建服务器监控神器在 Linux 的完整部署指南 在日常运维中,服务器监控是绕不开的...
飞牛NAS中安装Navidrome音乐文件中文标签乱码问题解决、安装FntermX终端

飞牛NAS中安装Navidrome音乐文件中文标签乱码问题解决、安装FntermX终端

飞牛 NAS 中安装 Navidrome 音乐文件中文标签乱码问题解决、安装 FntermX 终端 问题背景 ...
阿里云CDN
阿里云CDN-提高用户访问的响应速度和成功率
随机文章
你的云服务器到底有多强?宝塔跑分告诉你

你的云服务器到底有多强?宝塔跑分告诉你

你的云服务器到底有多强?宝塔跑分告诉你 为什么要用宝塔跑分? 宝塔跑分其实就是对 CPU、内存、磁盘、IO 做...
免费领取huggingface的2核16G云服务器,超简单教程

免费领取huggingface的2核16G云服务器,超简单教程

免费领取 huggingface 的 2 核 16G 云服务器,超简单教程 前言 HuggingFace.co...
我把用了20年的360安全卫士卸载了

我把用了20年的360安全卫士卸载了

我把用了 20 年的 360 安全卫士卸载了 是的,正如标题你看到的。 原因 偷摸安装自家的软件 莫名其妙安装...
每年0.99刀,拿下你的第一个顶级域名,详细注册使用

每年0.99刀,拿下你的第一个顶级域名,详细注册使用

每年 0.99 刀,拿下你的第一个顶级域名,详细注册使用 前言 作为长期折腾云服务、域名建站的老玩家,星哥一直...
开源MoneyPrinterTurbo 利用AI大模型,一键生成高清短视频!

开源MoneyPrinterTurbo 利用AI大模型,一键生成高清短视频!

  开源 MoneyPrinterTurbo 利用 AI 大模型,一键生成高清短视频! 在短视频内容...

免费图片视频管理工具让灵感库告别混乱

一言一句话
-「
手气不错
把小米云笔记搬回家:飞牛 NAS 一键部署,小米云笔记自动同步到本地

把小米云笔记搬回家:飞牛 NAS 一键部署,小米云笔记自动同步到本地

把小米云笔记搬回家:飞牛 NAS 一键部署,小米云笔记自动同步到本地 大家好,我是星哥,今天教大家在飞牛 NA...
安装并使用谷歌AI编程工具Antigravity(亲测有效)

安装并使用谷歌AI编程工具Antigravity(亲测有效)

  安装并使用谷歌 AI 编程工具 Antigravity(亲测有效) 引言 Antigravity...
免费无广告!这款跨平台AI RSS阅读器,拯救你的信息焦虑

免费无广告!这款跨平台AI RSS阅读器,拯救你的信息焦虑

  免费无广告!这款跨平台 AI RSS 阅读器,拯救你的信息焦虑 在算法推荐主导信息流的时代,我们...
星哥带你玩飞牛NAS硬件02:某鱼6张左右就可拿下5盘位的飞牛圣体NAS

星哥带你玩飞牛NAS硬件02:某鱼6张左右就可拿下5盘位的飞牛圣体NAS

星哥带你玩飞牛 NAS 硬件 02:某鱼 6 张左右就可拿下 5 盘位的飞牛圣体 NAS 前言 大家好,我是星...
300元就能买到的”小钢炮”?惠普7L四盘位小主机解析

300元就能买到的”小钢炮”?惠普7L四盘位小主机解析

  300 元就能买到的 ” 小钢炮 ”?惠普 7L 四盘位小主机解析 最近...