使用 Go 对 Nginx 进行性能测试

共计 8068 个字符，预计需要花费 21 分钟才能阅读完成。

目前有很多提供 Go 语言 HTTP 应用服务的方法，但其中最好的选择取决于每个应用的实际情况。目前，Nginx 看起来是每个新项目的标准 Web 服务器，即使在有其他许多不错 Web 服务器的情况下。然而，在 Nginx 上提供 Go 应用服务的开销是多少呢？我们需要一些 nginx 的特性参数（vhosts，负载均衡，缓存，等等）或者直接使用 Go 提供服务？如果你需要 nginx，最快的连接机制是什么？这就是在这我试图回答的问题。 该基准测试的目的不是要验证 Go 比 nginx 的快或慢。那将会很愚蠢。

下面是我们要比较不同的设置：

• Go HTTP standalone (as the control group)
• Nginx proxy to Go HTTP
• Nginx fastcgi to Go TCP FastCGI
• Nginx fastcgi to Go Unix Socket FastCGI

硬件

因为我们将在相同的硬件下比较所有设置，硬件选择的是廉价的一个。这不应该是一个大问题。

• Samsung 笔记本 NP550P5C-AD1BR
• Intel Core i7 3630QM @2.4GHz (quad core, 8 threads)
• CPU caches: (L1: 256KiB, L2: 1MiB, L3: 6MiB)
• RAM 8GiB DDR3 1600MHz

软件

• Ubuntu 13.10 amd64 Saucy Salamander (updated)
• Nginx 1.4.4 (1.4.4-1~saucy0 amd64)
• Go 1.2 (linux/amd64)
• wrk 3.0.4

设置

内核

只需很小的一点调整，将内核的 limits 调高。如果你对这一变量有更好的想法，请在写在下面评论处：

fs.file-max                    9999999
fs.nr_open                    9999999
net.core.netdev_max_backlog    4096
net.core.rmem_max              16777216
net.core.somaxconn            65535
net.core.wmem_max              16777216
net.ipv4.ip_forward            0
net.ipv4.ip_local_port_range  1025      65535
net.ipv4.tcp_fin_timeout      30
net.ipv4.tcp_keepalive_time    30
net.ipv4.tcp_max_syn_backlog  20480
net.ipv4.tcp_max_tw_buckets    400000
net.ipv4.tcp_no_metrics_save  1
net.ipv4.tcp_syn_retries      2
net.ipv4.tcp_synack_retries    2
net.ipv4.tcp_tw_recycle        1
net.ipv4.tcp_tw_reuse          1
vm.min_free_kbytes            65536
vm.overcommit_memory          1

Limits

供 root 和 www-data 打开的最大文件数限制被配置为 200000。

Nginx

有几个必需得 Nginx 调整。有人跟我说过，我禁用了 gzip 以保证比较公平。下面是它的配置文件 /etc/nginx/nginx.conf：

user www-data;
worker_processes auto;
worker_rlimit_nofile 200000;
pid /var/run/nginx.pid;

events {
    worker_connections 10000;
    use epoll;
    multi_accept on;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 300;
    keepalive_requests 10000;
    types_hash_max_size 2048;

    open_file_cache max=200000 inactive=300s;
    open_file_cache_valid 300s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    server_tokens off;
    dav_methods off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    access_log /var/log/nginx/access.log combined;
    error_log /var/log/nginx/error.log warn;

    gzip off;
    gzip_vary off;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*.conf;
}

Nginx vhosts

upstream go_http {
    server 127.0.0.1:8080;
    keepalive 300;
}

server {
    listen 80;
    server_name go.http;
    access_log off;
    error_log /dev/null crit;

    location / {
        proxy_pass http://go_http;
        proxy_http_version 1.1;
        proxy_set_header Connection “”;
    }
}

upstream go_fcgi_tcp {
    server 127.0.0.1:9001;
    keepalive 300;
}

server {
    listen 80;
    server_name go.fcgi.tcp;
    access_log off;
    error_log /dev/null crit;

    location / {
        include fastcgi_params;
        fastcgi_keep_conn on;
        fastcgi_pass go_fcgi_tcp;
    }
}

upstream go_fcgi_unix {
    server unix:/tmp/go.sock;
    keepalive 300;
}

server {
    listen 80;
    server_name go.fcgi.unix;
    access_log off;
    error_log /dev/null crit;

    location / {
        include fastcgi_params;
        fastcgi_keep_conn on;
        fastcgi_pass go_fcgi_unix;
    }
}

Go 源码

package main

import (
    “fmt”
    “log”
    “net”
    “net/http”
    “net/http/fcgi”
    “os”
    “os/signal”
    “syscall”
)

var (
    abort bool
)

const (
    SOCK = “/tmp/go.sock”
)

type Server struct {
}

func (s Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    body := “Hello World\n”
    // Try to keep the same amount of headers
    w.Header().Set(“Server”, “gophr”)
    w.Header().Set(“Connection”, “keep-alive”)
    w.Header().Set(“Content-Type”, “text/plain”)
    w.Header().Set(“Content-Length”, fmt.Sprint(len(body)))
    fmt.Fprint(w, body)
}

func main() {
    sigchan := make(chan os.Signal, 1)
    signal.Notify(sigchan, os.Interrupt)
    signal.Notify(sigchan, syscall.SIGTERM)

    server := Server{}

    go func() {
        http.Handle(“/”, server)
        if err := http.ListenAndServe(“:8080”, nil); err != nil {
            log.Fatal(err)
        }
    }()

    go func() {
        tcp, err := net.Listen(“tcp”, “:9001”)
        if err != nil {
            log.Fatal(err)
        }
        fcgi.Serve(tcp, server)
    }()

    go func() {
        unix, err := net.Listen(“unix”, SOCK)
        if err != nil {
            log.Fatal(err)
        }
        fcgi.Serve(unix, server)
    }()

    <-sigchan

    if err := os.Remove(SOCK); err != nil {
        log.Fatal(err)
    }
}

检查 HTTP header

为公平起见，所有的请求必需大小相同。

$ curl -sI http://127.0.0.1:8080/
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 12
Content-Type: text/plain
Server: gophr
Date: Sun, 15 Dec 2013 14:59:14 GMT

$ curl -sI http://127.0.0.1:8080/ | wc -c
141

$ curl -sI http://go.http/
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 15 Dec 2013 14:59:31 GMT
Content-Type: text/plain
Content-Length: 12
Connection: keep-alive

$ curl -sI http://go.http/ | wc -c
141

$ curl -sI http://go.fcgi.tcp/
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 12
Connection: keep-alive
Date: Sun, 15 Dec 2013 14:59:40 GMT
Server: gophr

$ curl -sI http://go.fcgi.tcp/ | wc -c
141

$ curl -sI http://go.fcgi.unix/
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 12
Connection: keep-alive
Date: Sun, 15 Dec 2013 15:00:15 GMT
Server: gophr

$ curl -sI http://go.fcgi.unix/ | wc -c
141

启动引擎

• 使用 sysctl 配置内核
• 配置 Nginx
• 配置 Nginx vhosts
• 用 www-data 启动服务
• 运行基准测试

基准测试

GOMAXPROCS = 1

Go standalone

# wrk -t100 -c5000 -d30s http://127.0.0.1:8080/
Running 30s test @ http://127.0.0.1:8080/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency  116.96ms  17.76ms 173.96ms  85.31%
    Req/Sec  429.16    49.20  589.00    69.44%
  1281567 requests in 29.98s, 215.11MB read
Requests/sec:  42745.15
Transfer/sec:      7.17MB

Nginx + Go through HTTP

# wrk -t100 -c5000 -d30s http://go.http/
Running 30s test @ http://go.http/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency  124.57ms  18.26ms 209.70ms  80.17%
    Req/Sec  406.29    56.94    0.87k    89.41%
  1198450 requests in 29.97s, 201.16MB read
Requests/sec:  39991.57
Transfer/sec:      6.71MB

Nginx + Go through FastCGI TCP

# wrk -t100 -c5000 -d30s http://go.fcgi.tcp/
Running 30s test @ http://go.fcgi.tcp/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency  514.57ms  119.80ms  1.21s    71.85%
    Req/Sec    97.18    22.56  263.00    79.59%
  287416 requests in 30.00s, 48.24MB read
  Socket errors: connect 0, read 0, write 0, timeout 661
Requests/sec:  9580.75
Transfer/sec:      1.61MB

Nginx + Go through FastCGI Unix Socket

# wrk -t100 -c5000 -d30s http://go.fcgi.unix/
Running 30s test @ http://go.fcgi.unix/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency  425.64ms  80.53ms 925.03ms  76.88%
    Req/Sec  117.03    22.13  255.00    81.30%
  350162 requests in 30.00s, 58.77MB read
  Socket errors: connect 0, read 0, write 0, timeout 210
Requests/sec:  11670.72
Transfer/sec:      1.96MB

GOMAXPROCS = 8

Go standalone

# wrk -t100 -c5000 -d30s http://127.0.0.1:8080/
Running 30s test @ http://127.0.0.1:8080/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency    39.25ms    8.49ms  86.45ms  81.39%
    Req/Sec    1.29k  129.27    1.79k    69.23%
  3837995 requests in 29.89s, 644.19MB read
Requests/sec: 128402.88
Transfer/sec:    21.55MB

Nginx + Go through HTTP

# wrk -t100 -c5000 -d30s http://go.http/
Running 30s test @ http://go.http/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency  336.77ms  297.88ms 632.52ms  60.16%
    Req/Sec    2.36k    2.99k  19.11k    84.83%
  2232068 requests in 29.98s, 374.64MB read
Requests/sec:  74442.91
Transfer/sec:    12.49MB

Nginx + Go through FastCGI TCP

# wrk -t100 -c5000 -d30s http://go.fcgi.tcp/
Running 30s test @ http://go.fcgi.tcp/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency  217.69ms  121.22ms  1.80s    75.14%
    Req/Sec  263.09    102.78  629.00    62.54%
  721027 requests in 30.01s, 121.02MB read
  Socket errors: connect 0, read 0, write 176, timeout 1343
Requests/sec:  24026.50
Transfer/sec:      4.03MB

Nginx + Go through FastCGI Unix Socket

# wrk -t100 -c5000 -d30s http://go.fcgi.unix/
Running 30s test @ http://go.fcgi.unix/
  100 threads and 5000 connections
  Thread Stats  Avg      Stdev    Max  +/- Stdev
    Latency  694.32ms  332.27ms  1.79s    62.13%
    Req/Sec  646.86    669.65    6.11k    87.80%
  909836 requests in 30.00s, 152.71MB read
Requests/sec:  30324.77
Transfer/sec:      5.09MB

结论

第一组基准测试时一些 Nginx 的设置还没有很好的优化（启用 gzip，Go 的后端没有使用 keep-alive 连接）。当改为 wrk 以及按建议优化 Nginx 后结果有较大差异。

当 GOMAXPROCS= 1 时，Nginx 的开销不是那么大，但当 OMAXPROCS= 8 时差异就很大了。以后可能会再试一下其他设置。如果你需要使用 Nginx 像虚拟主机，负载均衡，缓存等特性，使用 HTTP proxy，别使用 FastCGI。有些人说 Go 的 FastCGI 还没有被很好优化，这也许就是测试结果中巨大差异的原因。

推荐阅读 ：

Nginx 实现反向代理和负载均衡的配置及优化 http://www.linuxidc.com/Linux/2013-11/92909.htm

Nginx 做负载均衡报:nginx: [emerg] could not build the types_hash http://www.linuxidc.com/Linux/2013-10/92063.htm

Nginx 负载均衡模块 ngx_http_upstream_module 详述 http://www.linuxidc.com/Linux/2013-10/91907.htm

Nginx+Firebug 让浏览器告诉你负载均衡将请求分到了哪台服务器 http://www.linuxidc.com/Linux/2013-10/91824.htm

Ubuntu 安装 Nginx php5-fpm MySQL(LNMP 环境搭建) http://www.linuxidc.com/Linux/2012-10/72458.htm

Nginx 的详细介绍 ：请点这里
Nginx 的下载地址 ：请点这里