CentOS 安装 Logstash 2.2.0 和 Elasticsearch 2.2.0

431次阅读

没有评论

共计 18834 个字符，预计需要花费 48 分钟才能阅读完成。

本文介绍安装 logstash 2.2.0 和 elasticsearch 2.2.0，操作系统环境版本是 CentOS/Linux 2.6.32-504.23.4.el6.x86_64。

安装 JDK 是必须的，一般操作系统都会有，只是版本的问题，后面会提到。

而 Kibana 只是一个用纯 JavaScript 写的前端 UI，暂不介绍。因为，最近公司需要分析所有系统的日志，才搞 ELK，但人员和时间有限，三个框架都研究，不太现实。

使用 Elasticsearch + Logstash + Kibana 搭建日志集中分析平台实践 http://www.linuxidc.com/Linux/2015-12/126587.htm

Linux 上安装部署 ElasticSearch 全程记录 http://www.linuxidc.com/Linux/2015-09/123241.htm

Elasticsearch 安装使用教程 http://www.linuxidc.com/Linux/2015-02/113615.htm

ElasticSearch 配置文件译文解析 http://www.linuxidc.com/Linux/2015-02/114244.htm

ElasticSearch 集群搭建实例 http://www.linuxidc.com/Linux/2015-02/114243.htm

分布式搜索 ElasticSearch 单机与服务器环境搭建 http://www.linuxidc.com/Linux/2012-05/60787.htm

ElasticSearch 的工作机制 http://www.linuxidc.com/Linux/2014-11/109922.htm

ElasticSearch 的详细介绍 ：请点这里
ElasticSearch 的下载地址 ：请点这里

Elasticsearch

Elasticsearch（简称，ES）提供 ZIP、TAR、DEB 和 RPM 包。但 Github 上提供了一个针对中文环境的 Elasticsearch-RTF，RTF 即 Ready To Fly，它是一个针对中文的发行版，换句话说，帮你入门的。本文针对 Elasticsearch-RTF 为例。基本上，elasticsearch 解压后就能使用。

CentOS 安装 Logstash 2.2.0 和 Elasticsearch 2.2.0

假设你已经从 Github 上下载 elasticsearch-rtf，名为 elasticsearch-master.zip，并上传到你的 Linux 服务器 /usr/local/elasticsearch 目录（如果没有，就用 mkdir 命令创建一个）。
•现在，解压，并重新命名文件夹：

[root@linuxidc local]# cd /usr/local/elasticsearch

[root@linuxidc local]# unzip elasticsearch-master.zip

[root@linuxidc elasticsearch]# ls

elasticsearch-master elasticsearch-master.zip

[root@linuxidc local]# mv elasticsearch-master elasticsearch

[root@linuxidc elasticsearch]# ls

elasticsearch elasticsearch-master.zip
•尝试运行 elasticsearch：

Linux 环境：

[root@linuxidc elasticsearch]# pwd

/usr/local/elasticsearch/elasticsearch

[root@linuxidc elasticsearch]# bin/elasticsearch

windows 环境，执行相应的 .bat 文件，即 elasticsearch.bat。

但报错了：

[root@linuxidc elasticsearch]# bin/elasticsearch

Exception in thread “main” java.lang.RuntimeException: Java version: Oracle Cooration 1.7.0_51 [Java HotSpot(TM) 64-Bit Server VM 24.51-b03] suffers from crical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause dataorruption.

Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/referce/current/_installation.html for current recommendations.

If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JAVA_OPT environment variable.

Upgrading is preferred, this workaround will result in degraded performance.

at org.elasticsearch.bootstrap.JVMCheck.check(JVMCheck.java:123)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:283)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:3

Refer to the log for complete error details.

大意是：Java 运行时异常，本机版本 JDK 有 bug……让升级 JVM。如果实在不能升级，就向 JAVA_OPT 环境变量添加 -XX:-UseSuperWord 选项。

于是，看一下本机的 Java 版本：

[root@linuxidc elasticsearch]# java -version

java version “1.7.0_51”

Java(TM) SE Runtime Environment (build 1.7.0_51-b13)

Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

[root@linuxidc elasticsearch]# echo $JAVA_HOME

/usr/java/jdk1.7.0_51

[root@linuxidc elasticsearch]#

版本是 1.7.0_51。再在官网查了一下，说：“Elasticsearch requires at least Java 7. Specifically as of this writing, it is recommended that you use the Oracle JDK version 1.8.0_72. Java installation varies from platform to platform so we won’t go into those details here. Oracle’s recommended installation documentation can be found on Oracle’s website. Suffice to say, before you install Elasticsearch, please check your Java version first by running (and then install/upgrade accordingly if needed):”，大意是，ES 至少要求 7，推荐使用 1.8.0_72。
•那就删除之前的版本，按个新的吧。先删掉之前的 JDK，然后再用 yum 按个新的：

[root@linuxidc elasticsearch]# yum list installed | grep java

[root@linuxidc elasticsearch]# yum list installed | grep jdk

jdk.x86_64 2000:1.7.0_51-fcs installed

[root@linuxidc elasticsearch]# yum -y remove jdk.x86_64

……

[root@linuxidc elasticsearch]#yum -y install java-1.8.0-openjdk*

……

注意：java-1.8.0-openjdk*”，后面有个星号，即安装 java 全部相关的东西~
•安装完成后，设置 JDK 的环境变量：

[root@linuxidc elasticsearch]# export JAVA_HOME=/usr/lib/jvm/java-1.8.0

[root@linuxidc elasticsearch]# export PATH=$JAVA_HOME/bin:$PATH

[root@linuxidc elasticsearch]# export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

[root@linuxidc elasticsearch]# java -version

openjdk version “1.8.0_71”

OpenJDK Runtime Environment (build 1.8.0_71-b15)

OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)

[root@linuxidc elasticsearch]# echo $JAVA_HOME

/usr/lib/jvm/java-1.8.0

[root@linuxidc elasticsearch]#

另外，JDK 安装在了我机器的 /usr/lib/jvm 目录下，自己确认一下你的路径。
•再次运行：

[root@linuxidc elasticsearch]# bin/elasticsearch

Exception in thread “main” java.lang.RuntimeException: don’t run elasticsearch as root.

at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

[root@linuxidc elasticsearch]#
•又报错，elasticsearch 不能用 root 用户运行，那就建立一个：

[root@linuxidc elasticsearch]# groupadd es

[root@linuxidc elasticsearch]# useradd -g es es

[root@linuxidc elasticsearch]# passwd es

Changing password for user es.

New password:

BAD PASSWORD: it is WAY too short

BAD PASSWORD: is too simple

Retype new password:

passwd: all authentication tokens updated successfully.

[root@linuxidc elasticsearch]#

[root@linuxidc elasticsearch]# chown -R root .

[root@linuxidc elasticsearch]# chown -R es .

[root@linuxidc elasticsearch]# chgrp -R es .

[root@linuxidc elasticsearch]# ls -l

total 4

drwxr-xr-x 7 es es 4096 Mar 1 03:07 elasticsearch

[root@linuxidc elasticsearch]#
•重新打开一个终端，用 es 用户登录，并运行 elasticsearch：

[root@linuxidc ~]$ cd /usr/local/elasticsearch/elasticsearch

[root@linuxidc elasticsearch]$ bin/elasticsearch

[2016-03-01 05:11:48,413][WARN][bootstrap] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-03-01 05:11:48,750][INFO][node] [Googam] version[2.1.1], pid[15042], build[40e2c53/2015-12-15T13:05:55Z]

[2016-03-01 05:11:48,750][INFO][node] [Googam] initializing …

[2016-03-01 05:11:49,088][INFO][plugins] [Googam] loaded [elasticsearch-analysis-ik, elasticsearch-analysis-mmseg, elasticsearch-analysis-stconvert, elasticsearch-analysis-pinyin], sites []

[2016-03-01 05:11:49,121][INFO][env] [Googam] using [1] data paths, mounts [[/ (/dev/mapper/vg_linuxidc-lv_root)]], net usable_space [26.1gb], net total_space [34.8gb], spins? [possibly], types [ext4]

[2016-03-01 05:11:51,119][INFO][mmseg-analyzer] [Dict Loading] chars loaded time=42ms, line=12638, on file=chars.dic

……
•此时，在另一个终端，访问 elasticsearch：

[root@linuxidc elasticsearch]# curl -X GET http://localhost:9200

{

“name” : “Captain Savage”,

“cluster_name” : “elasticsearch”,

“version” : {

“number” : “2.1.1”,

“build_hash” : “40e2c53a6b6c2972b3d13846e450e66f4375bd71”,

“build_timestamp” : “2015-12-15T13:05:55Z”,

“build_snapshot” : false,

“lucene_version” : “5.3.1”

},

“tagline” : “You Know, for Search”

}

[root@linuxidc elasticsearch]#

ES 已经安装成功。

但此时的 ES 不能通过 IP 访问，所以，你要修改 config/elasticsearch.yml。找到“network.host”行，那个示例，添加一行：

network.host: your id address

就能通过 IP，或浏览器访问。

更多详情见请继续阅读下一页的精彩内容 ：http://www.linuxidc.com/Linux/2016-03/128794p2.htm

Logstash

logstash 主页

logstash 提供各种安装包，包括 tar.gz，ZIP，DEB 和 RPM。另外，又提供了一个包含所有插件的压缩包——logstash-all-plugins-2.2.0.tar.gz。本文以它为例。解压后，配置 logstash，运行即可。

CentOS 安装 Logstash 2.2.0 和 Elasticsearch 2.2.0

•在 /usr/local 下创建目录 logstash：

[root@linuxidc local]# cd /usr/local

[root@linuxidc local]# pwd

/usr/local

[root@linuxidc local]# mkdir logstash

[root@linuxidc local]#
•上传 logstash 到该目录。然后，解压并配置 logstash 配置文件：

[root@linuxidc local]# cd logstash

[root@linuxidc logstash]# ls

logstash-all-plugins-2.2.0.tar.gz

[root@linuxidc logstash]# tar zxf logstash-all-plugins-2.2.0.tar.gz

logstash-2.2.0 logstash-all-plugins-2.2.0.tar.gz

[root@linuxidc logstash]# cd logstash-2.2.0

[root@linuxidc logstash-2.2.0]# vi logstash.conf

[root@linuxidc logstash-2.2.0]#

文件内容如下：

input{

stdin{}

}

output{

stdout{}

}

这个配置是最简单，输入是终端命令行，输出也是终端命令行。
•运行 logstash：

[root@linuxidc logstash-2.2.0]# bin/logstash agent -f logstash.conf

hello world

hello world 2

Settings: Default pipeline workers: 2

Logstash startup completed

2016-02-28T22:37:06.130Z linuxidc hello world

2016-02-28T22:37:06.132Z linuxidc hello world 2

logst 已经成功安装。其中，agent 表示运行 Agent 模式，-f 表示指定配置文件，-p 表示端口，命令行参数可以参考 logstash command-lines flags。

另外，你可以查看 logstash 都安装了哪些插件。

[root@linuxidc logstash-2.2.0]# bin/plugin list
logstash-codec-avro
logstash-codec-cef
logstash-codec-cloudfront
logstash-codec-cloudtrail
logstash-codec-collectd
logstash-codec-compress_spooler
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-gzip_lines
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-nmap
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-codec-s3plain
logstash-codec-spool
logstash-filter-aggregate
logstash-filter-alter
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-cidr
logstash-filter-cipher
logstash-filter-clone
logstash-filter-collate
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dns
logstash-filter-drop
logstash-filter-elapsed
logstash-filter-elasticsearch
logstash-filter-environment
logstash-filter-extractnumbers
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-i18n
logstash-filter-json
logstash-filter-json_encode
logstash-filter-kv
logstash-filter-metaevent
logstash-filter-metricize
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-oui
logstash-filter-prune
logstash-filter-punct
logstash-filter-range
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-tld
logstash-filter-translate
logstash-filter-unique
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-filter-zeromq
logstash-input-beats
logstash-input-cloudwatch
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-fluentd
logstash-input-ganglia
logstash-input-gelf
logstash-input-gemfire
logstash-input-generator
logstash-input-github
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-jmx
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-meetup
logstash-input-pipe
logstash-input-puppet_facter
logstash-input-rabbitmq
logstash-input-redis
logstash-input-relp
logstash-input-rss
logstash-input-s3
logstash-input-salesforce
logstash-input-snmptrap
logstash-input-sqlite
logstash-input-sqs
logstash-input-stdin
logstash-input-stomp
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-varnishlog
logstash-input-websocket
logstash-input-wmi
logstash-input-xmpp
logstash-input-zenoss
logstash-input-zeromq
logstash-output-boundary
logstash-output-circonus
logstash-output-cloudwatch
logstash-output-csv
logstash-output-datadog
logstash-output-datadog_metrics
logstash-output-elasticsearch
logstash-output-elasticsearch-ec2
logstash-output-elasticsearch_http
logstash-output-elasticsearch_Java
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-google_bigquery
logstash-output-google_cloud_storage
logstash-output-graphite
logstash-output-graphtastic
logstash-output-hipchat
logstash-output-http
logstash-output-influxdb
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-librato
logstash-output-loggly
logstash-output-lumberjack
logstash-output-metriccatcher
logstash-output-mongodb
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-redmine
logstash-output-riemann
logstash-output-s3
logstash-output-sns
logstash-output-solr_http
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-stomp
logstash-output-syslog
logstash-output-tcp
logstash-output-udp
logstash-output-websocket
logstash-output-xmpp
logstash-output-zabbix
logstash-output-zeromq
logstash-patterns-core
[root@linuxidc logstash-2.2.0]#

本文永久更新链接地址 ：http://www.linuxidc.com/Linux/2016-03/128794.htm