Linux日志分析系统AWStats分析Nignx日志

207次阅读

共计 20101 个字符，预计需要花费 51 分钟才能阅读完成。

一. 简述
AWStats 软件是一个免费的强大的基于 Perl 的 WEB 日志分析工具，显示你所有的网页 / 邮件 / FTP 统计包括访问，访问者，页面，点击，高峰时间，操作系统，浏览器，搜索引擎，关键字，机器人访问，断开的链接和更多的阻力截图排序。

（1）功能: 它可以统计您站点的如下信息：
一：访问量，访问次数，页面浏览量，点击数，数据流量等
二：精确到每月、每日、每小时的数据
三：访问者国家
四：访问者 IP
五：Robots/Spiders 的统计
六：访客持续时间
七：对不同 Files type 的统计信息
八：Pages-URL 的统计
九：访客操作系统浏览器等信息
十：其它信息（搜索关键字等等）

（2）工作原理
工作原理

AWStats 的功能很多，我在此主要用它来分析 apache 服务器的日志。安装使用之前还是说说大致的工作原理，AWStats 提供一系列的 perl 脚本实现：服务配置，日志读取，报表生成等功能。而功能实现的具体执行过程是：首先，当然是 apache 将访问情况记录到日志中，AWStats 每次执行更新时读取这些日志，分析日志数据，将结果存储到数据库中，(这个数据库是 AWStats 自带的（就是一文本文件），并不需要第三方软件支持。)，最后 AWStats 提供一个 cgi 程序通过 web 页面来显示数据库中所统计的数据。
工作模式

AWStats 的工作模式是这样的：
分析日志：运行后将这样的日志统计结果归档到一个 AWStats 的数据库（纯文本）里；
输出日志：分两种形式
一种是通过 cgi 程序读取统计结果数据库输出（Linux 中）；

一种是运行后台脚本将输出导出成静态文件(Windows 中)；

（3）nginx 安装配置详解
一. 配置 nginx 日志格式并重新加载 nginx，修改 nginx.conf 的日志格式，不然 awstats 无法分析。
[root@Monitor server]# vim /usr/local/nginx/conf/nginx.conf
worker_processes 1;
user appuser appuser;
error_log /data/nginx/error.log;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main ‘$remote_addr – $remote_user [$time_local] “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
‘”$http_user_agent” “$http_x_forwarded_for”‘;
access_log /data/nginx/access.log;
[root@Monitor server]# /usr/local/nginx/sbin/nginx -s reload

#### 注意要点：日志格式里的双引号不能漏了且每个参数之间是一个空格分隔，因为只要有细微的差别，awstats 就无法读取日志

二. 安装 Awstats。
Awstats 的安装有两种方式，一种是 yum install awstats。二种是去官方网（https://awstats.sourceforge.io）直接下载安装包来进行安装。本文主要的是通过下载安装包进行安装
（1）通过 yum 直接进行安装
[root@ittestserver1 opt]# yum install -y awstats
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/metalink | 6.0 kB 00:00
* base: mirrors.163.com
* epel: mirror.premi.st
* extras: mirrors.zju.edu.cn
* updates: mirrors.163.com
base | 3.7 kB 00:00
epel | 4.3 kB 00:00
epel/primary_db | 5.9 MB 00:01
extras | 3.4 kB 00:00
jenkins | 2.9 kB 00:00
jenkins/primary_db | 21 kB 00:00
updates | 3.4 kB 00:00
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package awstats.noarch 0:7.0-3.el6 will be installed
–> Processing Dependency: perl(URI::Escape) for package: awstats-7.0-3.el6.noarch
–> Processing Dependency: perl(LWP::UserAgent) for package: awstats-7.0-3.el6.noarch
–> Processing Dependency: perl(HTML::Entities) for package: awstats-7.0-3.el6.noarch
–> Running transaction check
—> Package perl-HTML-Parser.x86_64 0:3.64-2.el6 will be installed
–> Processing Dependency: perl(HTML::Tagset) >= 3.03 for package: perl-HTML-Parser-3.64-2.el6.x86_64
–> Processing Dependency: perl(HTML::Tagset) for package: perl-HTML-Parser-3.64-2.el6.x86_64
—> Package perl-URI.noarch 0:1.40-2.el6 wil
—> Package perl-libs.x86_64 4:5.10.1-141.el6_7.1 will be updated
—> Package perl-libs.x86_64 4:5.10.1-144.el6 will be an update
—> Package perl-version.x86_64 3:0.77-141.el6_7.1 will be updated
—> Package perl-version.x86_64 3:0.77-144.el6 will be an update
–> Finished Dependency Resolution
Verifying : 3:perl-version-0.77-141.el6_7.1.x86_64 25/29
Verifying : 4:perl-libs-5.10.1-141.el6_7.1.x86_64 26/29
Verifying : perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64 27/29
Verifying : 1:perl-ExtUtils-ParseXS-2.2003.0-141.el6_7.1.x86_64 28/29
Verifying : perl-Test-Harness-3.17-141.el6_7.1.x86_64 29/29
Installed:
awstats.noarch 0:7.0-3.el6
Dependency Installed:
perl-Compress-Raw-Zlib.x86_64 1:2.021-144.el6 perl-Compress-Zlib.x86_64 0:2.021-144.el6
perl-HTML-Parser.x86_64 0:3.64-2.el6 perl-HTML-Tagset.noarch 0:3.20-4.el6
perl-IO-Compress-Base.x86_64 0:2.021-144.el6 perl-IO-Compress-Zlib.x86_64 0:2.021-144.el6
perl-URI.noarch 0:1.40-2.el6 perl-libwww-perl.noarch 0:5.833-5.el6
Dependency Updated:
perl.x86_64 4:5.10.1-144.el6 perl-ExtUtils-MakeMaker.x86_64 0:6.55-144.el6
perl-ExtUtils-ParseXS.x86_64 1:2.2003.0-144.el6 perl-Module-Pluggable.x86_64 1:3.90-144.el6
perl-Pod-Escapes.x86_64 1:1.04-144.el6 perl-Pod-Simple.x86_64 1:3.13-144.el6
perl-Test-Harness.x86_64 0:3.17-144.el6 perl-devel.x86_64 4:5.10.1-144.el6
perl-libs.x86_64 4:5.10.1-144.el6 perl-version.x86_64 3:0.77-144.el6
Complete!

（2）通过下载安装包进行安装。下载地址为：https://prdownloads.sourceforge.net/awstats/awstats-7.6.tar.gz
[root@Monitor soft]# wget https://prdownloads.sourceforge.net/awstats/awstats-7.6.tar.gz
–2017-09-18 11:17:28– https://prdownloads.sourceforge.net/awstats/awstats-7.6.tar.gz
正在解析主机 prdownloads.sourceforge.net… 216.34.181.59
正在连接 prdownloads.sourceforge.net|216.34.181.59|:443… 已连接。
已发出 HTTP 请求，正在等待回应 … 301 Moved Permanently
位置：https://downloads.sourceforge.net/project/awstats/AWStats/7.6/awstats-7.6.tar.gz [跟随至新的 URL]
–2017-09-18 11:17:29– https://downloads.sourceforge.net/project/awstats/AWStats/7.6/awstats-7.6.tar.gz
正在解析主机 downloads.sourceforge.net… 216.34.181.59
正在连接 downloads.sourceforge.net|216.34.181.59|:443… 已连接。
已发出 HTTP 请求，正在等待回应 … 302 Found
位置：https://nchc.dl.sourceforge.net/project/awstats/AWStats/7.6/awstats-7.6.tar.gz [跟随至新的 URL]
–2017-09-18 11:17:42– https://nchc.dl.sourceforge.net/project/awstats/AWStats/7.6/awstats-7.6.tar.gz
正在解析主机 nchc.dl.sourceforge.net… 211.79.60.17, 2001:e10:ffff:1f02::17
正在连接 nchc.dl.sourceforge.net|211.79.60.17|:443… 已连接。
已发出 HTTP 请求，正在等待回应 … 200 OK
长度：2955876 (2.8M) [application/x-gzip]
正在保存至:“awstats-7.6.tar.gz”
5% [==>] 163,840 53.1K/s eta(英国中部时 6% [===>] 196,608 56.5K/s eta(英国中部时 7% [===>] 229,376 59.4K/s eta(英国中部时 8% [====>] 245,760 57.8K/s eta(英国中部时 9% [=====>] 278,528 55.4K/s eta(英国中部时 10% [=====>] 311,296 57.5K/s eta(英国中部时 11% [======>] 327,680 56.2K/s eta(英国中部时 11% [======>] 344,064 56.1K/s eta(英国中部时 12% [======>] 360,448 55.6K/s eta(英国中部时 12% [=======>] 376,832 55.1K/s eta(英国中部时 13% [=======>] 393,216 54.7K/s eta(英国中部时 14% [========>] 425,984 56.1K/s eta(英国中部时 14% [========>] 442,368 55.6K/s eta(英国中部时 15% [========>] 458,752 55.8K/s eta(英国中部时 16% [=========>] 475,136 56.2K/s eta(英国中部时 16% [=========>] 491,520 54.4K/s eta(英国中部时 17% [=========>] 507,904 54.8K/s eta(英国中部时 17% [==========>] 524,288 53.0K/s eta(英国中部时 18% [===========>] 557,056 55.4K/s eta(英国中部时 19% [===========>] 573,440 55.8K/s eta(英国中部时 20% [============>] 606,208 58.2K/s eta(英国中部时 21% [============>] 622,592 56.3K/s eta(英国中部时 22% [=============>] 655,360 56.4K/s eta(英国中部时 23% [==============>] 704,512 60.8K/s eta(英国中部时 25% [===============>] 753,664 66.4K/s eta(英国中部时 27% [================>] 802,816 68.7K/s eta(英国中部时 29% [==================>] 884,736 78.2K/s eta(英国中部时 32% [===================>] 966,656 86.6K/s eta(英国中部时 36% [======================>] 1,064,960 97.5K/s eta(英国中部时 39% [========================>] 1,179,648 111K/s eta(英国中部时 42% [=========================>] 1,245,184 117K/s eta(英国中部时 48% [=============================>] 1,425,408 135K/s eta(英国中部时 52% [================================>] 1,540,096 150K/s eta(英国中部时 54% [==================================>] 1,622,016 159K/s eta(英国中部时 57% [===================================>] 1,703,936 166K/s eta(英国中部时 60% [=====================================>] 1,785,856 174K/s eta(英国中部时 62% [=======================================>] 1,851,392 177K/s eta(英国中部时 63% [=======================================>] 1,867,776 178K/s eta(英国中部时 65% [=========================================>] 1,949,696 183K/s eta(英国中部时 69% [===========================================>] 2,048,000 193K/s eta(英国中部时 70% [============================================>] 2,097,152 194K/s eta(英国中部时 72% [=============================================>] 2,146,304 197K/s eta(英国中部时 74% [==============================================>] 2,195,456 199K/s eta(英国中部时 75% [===============================================>] 2,244,608 199K/s eta(英国中部时 77% [================================================>] 2,293,760 199K/s eta(英国中部时 79% [=================================================>] 2,342,912 199K/s eta(英国中部时 80% [==================================================>] 2,392,064 195K/s eta(英国中部时 82% [===================================================>] 2,441,216 187K/s eta(英国中部时 84% [====================================================>] 2,490,368 184K/s eta(英国中部时 85% [=====================================================>] 2,539,520 167K/s eta(英国中部时 88% [=======================================================>] 2,605,056 167K/s eta(英国中部时 89% [========================================================>] 2,637,824 142K/s eta(英国中部时 90% [========================================================>] 2,670,592 131K/s eta(英国中部时 91% [=========================================================>] 2,703,360 124K/s eta(英国中部时 92% [==========================================================>] 2,736,128 125K/s eta(英国中部时 93% [==========================================================>] 2,768,896 118K/s eta(英国中部时 94% [===========================================================>] 2,801,664 109K/s eta(英国中部时 95% [============================================================>] 2,834,432 107K/s eta(英国中部时 97% [=============================================================>] 2,867,200 105K/s eta(英国中部时 98% [=============================================================>] 2,899,968 102K/s eta(英国中部时 99% [==============================================================>] 2,932,736 100K/s eta(英国中部时 100%[===============================================================>] 2,955,876 102K/s in 26s
2017-09-18 11:18:09 (112 KB/s) – 已保存“awstats-7.6.tar.gz”[2955876/2955876])
[root@Monitor soft]# rm -rf /usr/local/awstats/
[root@Monitor soft]# tar xf awstats-7.6.tar.gz
[root@Monitor soft]# mv awstats-7.6
awstats-7.6/ awstats-7.6.tar.gz
[root@Monitor soft]# mv awstats-7.6 /usr/local/
[root@Monitor awstats]# mkdir /data/awstats

三. 配置 Awstats
1, 进入 /usr/local/awstats/tools/ 目录，执行配置脚本 awstats_configure.pl：
[root@Monitor awstats]# cd /usr/local/awstats/tools/
[root@Monitor tools]# ./awstats_configure.pl
—– AWStats awstats_configure 1.0 (build 20140126) (c) Laurent Destailleur —–
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
– You are not an administrator user,
– You want to analyze downloaded log files without web server,
– You want to analyze mail or ftp log files instead of web log files,
– You need to analyze load balanced servers log files,
– You want to ‘understand’ all possible ways to use AWStats…
Read the AWStats documentation (docs/index.html).
—–> Running OS detected: Linux, BSD or Unix
—–> Check for web server install
Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path (‘none’ to skip web server setup):
> none ##### 跳过服务的设置
Your web server config file(s) could not be found.
You will need to setup your web server manually to declare AWStats
script as a CGI, if you want to build reports dynamically.
See AWStats setup documentation (file docs/index.html)
—–> Update model config file ‘/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf’
File awstats.model.conf updated.
—–> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y ##### 点击 y 创建一个新的配置文件
—–> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> www.gjb.com ##### 输入你网站的域名
—–> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
> ##### 直接回车键
—–> Create config file ‘/etc/awstats/awstats.www.gjb.com.conf’
Config file /etc/awstats/awstats.www.gjb.com.conf created. ## 配置文件所在路径
—–> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.gjb.com
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue… ##### 直接按回车键
A SIMPLE config file has been created: /etc/awstats/awstats.www.gjb.com.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for ‘www.gjb.com’ with command:
> perl awstats.pl -update -config=www.gjb.com
You can also build static report pages for ‘www.gjb.com’ with command:
> perl awstats.pl -output=pagetype -config=www.gjb.com
Press ENTER to finish… ##### 直接按回车键

2. 编辑日志配置文件，修改 LogFile 参数路径为你 nginx 日志文件的路径
[root@Monitor tools]# vim /etc/awstats/awstats.www.gjb.com.conf
# AWSTATS CONFIGURE FILE 7.3
#—————————————————————————–
# Copy this file into awstats.www.mydomain.conf and edit this new config file
# to setup AWStats (See documentation in docs/ directory).
# “LogFile” contains the web, ftp or mail server log file to analyze.
# Possible values: A full path, or a relative path from awstats.pl directory.
# Example: “/var/log/apache/access.log”
# Example: “../logs/mycombinedlog.log”
# You can also use tags in this filename if you need a dynamic file name
# depending on date or time (Replacement is made by AWStats at the beginning
# of its execution). This is available tags :
# %YYYY-n is replaced with 4 digits year we were n hours ago
# %YY-n is replaced with 2 digits year we were n hours ago
LogFile=”/data/nginx/0914.log-20170917″ #日志文件存放路径
DirData=”/var/lib/awstats” #创建生成的数据路径
Lang=”cn” #默认语言中文
SkipHosts=”127.0.0.1 REGEX[^192\.168\.]” #本地及内部的访问不做分析统计
LevelForWormsDetection=2 #日志等级，不对警告日志进行统计

四. 执行 awstats 日志分析 awstats.pl 语句
[root@Monitor tools]# /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.gjb.com
Create/Update database for config “/etc/awstats/awstats.www.gjb.com.conf” by AWStats version 7.6 (build 20161204)
From data in log file “/data/nginx/0914.log-20170917″…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Flush history file on disk (unique hosts reach flush limit of 20000)
Flush history file on disk (unique hosts reach flush limit of 20000)
Flush history file on disk (unique hosts reach flush limit of 20000)
Flush history file on disk (unique hosts reach flush limit of 20000)
Flush history file on disk (unique hosts reach flush limit of 20000)
Jumped lines in file: 0
Parsed lines in file: 3753576
Found 2 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 3753574 new qualified records.

## 查看一下 awstats 数据库文件
[root@Monitor awstats]# du -sh *
8.3M awstats092017.www.gjb.com.txt

五. 配置 awstats 生成静态页面
利用 awstats 的工具将统计的结果生成静态文件：
[root@Monitor awstats]# /usr/local/awstats/tools/awstats_buildstaticpages.pl -update -config=www.gjb.com
-lang=cn -dir=/data/awstats -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl
Launch update process : “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -update -configdir=
Build main page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output
Build alldomains page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=alldomains
Build allhosts page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=allhosts
Build lasthosts page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=lasthosts
Build unknownip page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=unknownip
Build allrobots page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=allrobots
Build lastrobots page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=lastrobots
Build session page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=session
Build urldetail page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=urldetail
Build urlentry page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=urlentry
Build urlexit page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=urlexit
Build osdetail page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=osdetail
Build unknownos page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=unknownos
Build browserdetail page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=browserdetail
Build unknownbrowser page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=unknownbrowser
Build downloads page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=downloads
Build refererse page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=refererse
Build refererpages page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=refererpages
Build keyphrases page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=keyphrases
Build keywords page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=keywords
Build errors400 page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=errors400
Build errors403 page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=errors403
Build errors404 page: “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -config=www.gjb.com -staticlinks -lang=cn -output=errors404
23 files built.
Main HTML page is ‘awstats.www.gjb.com.html’.

如果想每天都更新的话设置定时任务：

[root@Monitor awstats]# crontab -e
no crontab for root – using an empty one
1 0 * * * /usr/local/awstats/tools/awstats_buildstaticpages.pl -update -config= -lang=cn -dir=/data/awstats -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl

六. 配置 nginx 代理来查看生成的效果图。
[root@Monitor awstats]# vim /usr/local/nginx/conf/server/server.conf
server {
listen 8000;
server_name xn3.lqb.com;
root /data/;
location ~ ^/awstats {
# root /opt/www/awstats;
index index.html;
access_log /opt/www/awstats.log;
error_log /opt/www/error.log;
#charset utf-8;
charset gb2312;
#auth_basic “admin”;
# auth_basic_user_file conf/admin.pass;
}
location ~ ^/icon/{
root /usr/local/awstats/wwwroot;
index index.html;
access_log off;
error_log off;
}
}
[root@Monitor awstats]# /usr/local/nginx/sbin/nginx -s reload