HDP上安装impala

2017-01-05 11:14:16来源:oschina作者:litterMo人点击

Impala是Cloudera公司主导开发的新型查询系统,它提供SQL语义,能查询存储在Hadoop的HDFS和HBase中的PB级大数据。Impala提供更快的查询速度,性能上号称比Hive快3~10倍。Impala是开源的,但一般都是通过cloudera manager或者在CDH版本上安装,今天主要介绍的是在HDP版本上的安装。


版本


Impala对于Hadoop的版本要求很高,现在说明一下当前安装的版本信息


Impala 2.5


HDP 2.2.8.0 基于Hadoop2.6


安装步骤


1. 在/etc/yum.repo.d 中创建impala.repo


[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/
gpgkey =https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1

2. 执行yum命令


yum installimpala-server impala-catalog impala-state-store impala-shell

3.更改impala链接库信息


ln -sf/usr/hdp/2.2.8.0-3150/hadoop/hadoop-common.jar /usr/lib/impala/lib/hadoop-common.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop/hadoop-annotations.jar /usr/lib/impala/lib/hadoop-annotations.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop/hadoop-auth.jar /usr/lib/impala/lib/hadoop-auth.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop/hadoop-aws.jar /usr/lib/impala/lib/hadoop-aws.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-hdfs/hadoop-hdfs.jar /usr/lib/impala/lib/hadoop-hdfs.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-mapreduce/hadoop-mapreduce-client-common.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-mapreduce/hadoop-mapreduce-client-core.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-api.jar /usr/lib/impala/lib/hadoop-yarn-api.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-client.jar /usr/lib/impala/lib/hadoop-yarn-client.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-common.jar /usr/lib/impala/lib/hadoop-yarn-common.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-server-common.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-server-nodemanager.jar /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar
ln -sf/usr/hdp/2.2.8.0-3150/hadoop-yarn/hadoop-yarn-server-web-proxy.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar
ln -sf /usr/hdp/2.2.8.0-3150/hadoop/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.so
ln -sf /usr/hdp/2.2.8.0-3150/hadoop/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0
ln -sf /usr/hdp/2.2.8.0-3150/usr/lib/libhdfs.so /usr/lib/impala/lib/libhdfs.so
ln -sf /usr/hdp/2.2.8.0-3150/usr/lib/libhdfs.so.0.0.0 /usr/lib/impala/lib/libhdfs.so.0.0.0
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-ant.jar /usr/lib/impala/lib/hive-ant.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-beeline.jar /usr/lib/impala/lib/hive-beeline.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-common.jar /usr/lib/impala/lib/hive-common.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-exec.jar /usr/lib/impala/lib/hive-exec.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-hbase-handler.jar /usr/lib/impala/lib/hive-hbase-handler.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-metastore.jar /usr/lib/impala/lib/hive-metastore.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-serde.jar /usr/lib/impala/lib/hive-serde.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-service.jar /usr/lib/impala/lib/hive-service.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-shims-common.jar /usr/lib/impala/lib/hive-shims-common.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-shims-scheduler.jar /usr/lib/impala/lib/hive-shims-scheduler.jar
ln -sf /usr/hdp/2.2.8.0-3150/hive/lib/hive-shims.jar /usr/lib/impala/lib/hive-shims.jar
ln -sf /usr/hdp/2.2.8.0-3150/zookeeper/zookeeper.jar /usr/lib/impala/lib/zookeeper.jar

4. 增加JAVA_HOME


vi/etc/default/bigtop-utils


增加 EXPORT JAVA_HOME= /usr/jdk64/jdk1.7.0_67


5. 修改配置文件


增加 /etc/hadoop/conf/core-site.xml



dfs.client.read.shortcircuit true


dfs.client.read.shortcircuit.skip.checksum
false


dfs.datanode.hdfs-blocks-metadata.enabled
true

增加 /etc/hadoop/conf/hdfs-site.xml



dfs.datanode.hdfs-blocks-metadata.enabled
true


dfs.block.local-path-access.user
impala


dfs.client.file-block-storage-locations.timeout.millis
60000

执行拷贝


cp /etc/hadoop/conf/*.xml /etc/impala/conf
cp /etc/hive/conf/hive-site.xml /etc/impala/conf

否则报错


E0721 15:52:54.265014 11246 impala-server.cc:247] Unsupported default filesystem. The default filesystem must be a DistributedFileSystem but the configured default filesystem is LocalFileSystem. fs.defaultFS (file:///) might be set incorrectly.ERROR: block location tracking is not properly enabled because
- dfs.datanode.hdfs-blocks-metadata.enabled is not enabled.
- dfs.client.file-block-storage-locations.timeout.millis is too low. It should be at least 10 seconds.

6. 修改 /etc/default/impala


修改IMPALA_CATALOG_SERVICE_HOST,IMPALA_STATE_STORE_HOST 两个变量为你启动catalog,state_store两个服务的host


7. 从网上下载htrace-core-3.0.4.jar改名htrace-core-2.00.jar 替换/usr/lib/impala/lib/htrace-core-2.00.jar


否则会报错


E0711 13:55:00.4515846925 impala-server.cc:247] NoClassDefFoundError: org/htrace/Trace
CAUSED BY: ClassNotFoundException: org.htrace.Trace
E0711 13:55:00.4516536925 impala-server.cc:249] Aborting Impala Server startup due to improper configuration

8 一个host启动 catalog, state_store


所有host启动 impalad

提示:


安装过程中需要学会查看/var/log/impala的日志,可以查看当前的错误信息

我们已经编写好可以用的ambari组件进行impala安装,组件地址见下面的链接


https://github.com/cas-bigdatalab/ambari-impala-service

最新文章

123

最新摄影

微信扫一扫

第七城市微信公众平台