hadoop-2.6.0编译详细步骤,通用于其他版本

Hadoop admin 2年前 (2018-11-03) 249次浏览 0个评论 扫描二维码

一、环境准备

1、JDK安装部署

详细安装地址:[JDK安装部署](https://www.codeobj.com/?p=302)

2、Maven安装

详细安装地址:[Maven安装部署](https://www.codeobj.com/?p=129)

二、安装编译hadoop

1、安装编译protobuf

1)下载:protobuf2.5.0

https://github.com/protocolbuffers/protobuf/releases

2)解压

`tar -zxvf protobuf-2.5.0.tar.gz`

3)编译protobuf

cd protobuf-2.5.0/
./configure 
make
make install
# 查看protobuf版本以测试是否安装成功
protoc --version
①如果出现以下问题
protoc: error while loading shared libraries: libprotobuf.so.8: cannot open shared object file: No such file or directory
则设置如下变量
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
②如果没有异常则会出现如下版本
libprotoc 2.5.0

2、安装snappy

[root@hadopp000 software]# wget http://pkgs.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.1.tar.gz/8887e3b7253b22a31f5486bca3cbc1c2/snappy-1.1.1.tar.gz
# 用root用户执行以下命令
[root@hadopp000 software]#tar -zxvf snappy-1.1.1.tar.gz
[root@hadopp000 software]# cd snappy-1.1.1/
[root@hadopp000 snappy-1.1.1]# ./configure
[root@hadopp000 snappy-1.1.1]# make
[root@hadopp000 snappy-1.1.1]# make install

#查看snappy库文件
[root@hadoop000 snappy-1.1.1]# ls -lh /usr/local/lib |grep snappy
-rw-r--r--. 1 root root 228K Dec  5 20:06 libsnappy.a
-rwxr-xr-x. 1 root root  953 Dec  5 20:06 libsnappy.la
lrwxrwxrwx. 1 root root   18 Dec  5 20:06 libsnappy.so -> libsnappy.so.1.2.0
lrwxrwxrwx. 1 root root   18 Dec  5 20:06 libsnappy.so.1 -> libsnappy.so.1.2.0
-rwxr-xr-x. 1 root root 145K Dec  5 20:06 libsnappy.so.1.2.0

3、安装bzip2

这个包很少有人提到,但是我在安装完毕之后却发现少了bzip2的支持
所以这里特意提醒以下
yum install -y  bzip2-devel

4、安装hadoop2.6.0

1) 下载hadoop2.6.0

https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0/

2) 编译hadoop2.6.0

[hadoop@hadoop000 software]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz

[hadoop@hadoop000 software]$ cd hadoop-2.7.3-src

[hadoop@hadoop000 software]$ mvn clean package -Pdist,native -DskipTests -Dtar
我第一次编译的时候发现出现了异常,然后添加如下参数,跳过javadoc就可以编译成功了
[hadoop@hadoop000 software]$ mvn clean package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true

3) 检查native

[hadoop@hadoop000 native]$ ll
total 4940
-rw-r--r--. 1 hadoop hadoop  106366 Dec  7 20:42 libgplcompression.a
-rw-r--r--. 1 hadoop hadoop    1138 Dec  7 20:42 libgplcompression.la
-rwxr-xr-x. 1 hadoop hadoop   69499 Dec  7 20:42 libgplcompression.so
-rwxr-xr-x. 1 hadoop hadoop   69499 Dec  7 20:42 libgplcompression.so.0
-rwxr-xr-x. 1 hadoop hadoop   69499 Dec  7 20:42 libgplcompression.so.0.0.0
-rw-r--r--. 1 hadoop hadoop 1308954 Dec  7 00:07 libhadoop.a
-rw-r--r--. 1 hadoop hadoop 1485708 Dec  7 00:07 libhadooppipes.a
lrwxrwxrwx. 1 hadoop hadoop      18 Dec  7 00:07 libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x. 1 hadoop hadoop  765744 Dec  7 00:07 libhadoop.so.1.0.0
-rw-r--r--. 1 hadoop hadoop  582048 Dec  7 00:07 libhadooputils.a
-rw-r--r--. 1 hadoop hadoop  359746 Dec  7 00:07 libhdfs.a
lrwxrwxrwx. 1 hadoop hadoop      16 Dec  7 00:07 libhdfs.so -> libhdfs.so.0.0.0
-rwxr-xr-x. 1 hadoop hadoop  228715 Dec  7 00:07 libhdfs.so.0.0.0

[hadoop@hadoop000 hadoop-2.6.0]$ hadoop checknative
18/12/08 19:01:28 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/12/08 19:01:28 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/local/lib/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
将native目录复制到/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/目录下,重新启动hadoop即可

codeobj , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:hadoop-2.6.0编译详细步骤,通用于其他版本
喜欢 (0)
[a37free@163.com]
分享 (0)
发表我的评论
取消评论
表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址