阿里云-云小站(无限量代金券发放中)
【腾讯云】云服务器、云数据库、COS、CDN、短信等热卖云产品特惠抢购

Swift开启StatsD后出现上传数据出现返回503的Bug

100次阅读
没有评论

共计 3303 个字符,预计需要花费 9 分钟才能阅读完成。

swift 在版本 2.1.0 之前如果各个服务的配置文件中打开以下配置后,且系统没有配置正确将会出现上传对象出错的情况
log_statsd_host = localhost
log_statsd_port = 8125
log_statsd_default_sample_rate = 1.0
log_statsd_sample_rate_factor = 1.0
log_statsd_metric_prefix =
具体错误 log 信息大概如下:
object-server ERROR __call__ error with PUT /sdc/2468/AUTH_8f9dbbadd64a43a0abb5e832c6ea766a/000008/013781 : #012Traceback (most recent call last):#012  File “/usr/lib/Python2.6/site-packages/swift/obj/server.py”, line 938, in __call__#012    res = method(req)#012  File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 1558, in wrapped#012    return func(*a, **kw)#012  File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File “/usr/lib/python2.6/site-packages/swift/obj/server.py”, line 712, in PUT#012    file.put(fd, metadata)#012  File “/usr/lib64/python2.6/contextlib.py”, line 34, in __exit__#012    self.gen.throw(type, value, traceback)#012  File “/usr/lib/python2.6/site-packages/swift/obj/server.py”, line 286, in mkstemp#012    yield fd#012  File “/usr/lib/python2.6/site-packages/swift/obj/server.py”, line 680, in PUT#012    ‘PUT.’ + device + ‘.timing’, elapsed_time, upload_size)#012  File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 654, in wrapped#012    return func(self.logger.statsd_client, *a, **kw)#012  File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 506, in transfer_rate#012    sample_rate)#012  File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 496, in timing#012    return self._send(metric, timing_ms, ‘ms’, sample_rate)#012  File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 481, in _send#012    return sock.sendto(‘|’.join(parts), self._target)#012  File “/usr/lib/python2.6/site-packages/eventlet/greenio.py”, line 371, in sendto#012    return self.fd.sendto(*args)#012error: [Errno 1] Operation not permitted (txn: tx8d76698250304466817aa99061637421)

根据 log 信息查到是在 swift/common/utils.py 文件的 StatsdClient._send 函数抛出了异常没有被捕捉导致的,该函数代码如下:
    def _send(self, m_name, m_value, m_type, sample_rate):
        if sample_rate is None:
            sample_rate = self._default_sample_rate
        sample_rate = sample_rate * self._sample_rate_factor
        parts = [‘%s%s:%s’ % (self._prefix, m_name, m_value), m_type]
        if sample_rate < 1:
            if self.random() < sample_rate:
                parts.append(‘@%s’ % (sample_rate,))
            else:
                return
        # Ideally, we’d cache a sending socket in self, but that
        # results in a socket getting shared by multiple green threads.
        with closing(self._open_socket()) as sock:
                return sock.sendto(‘|’.join(parts), self._target)    #该函数调用抛出了异常
解决办法:
    在 return sock.sendto(‘|’.join(parts), self._target)中加入异常处理即可,具体代码可参考官方最新代码

同时系统的 /var/log/messages 日志中出现大量的如下信息:
proxy-access Error sending UDP message to (‘127.0.0.1’, 8125): [Errno 1] Operation not permitted
proxy-access Error sending UDP message to (‘127.0.0.1’, 8125): [Errno 1] Operation not permitted
kernel: __ratelimit: 89 callbacks suppressed
kernel: nf_conntrack: table full, dropping packet.
kernel: nf_conntrack: table full, dropping packet.
kernel: nf_conntrack: table full, dropping packet.
解决办法:
根据上面的信息,得知 8125 端口是 StatsD 服务端口,因此是 StatsD 的客户端出了问题。同时内核报出了丢包错误,主要是由于服务器防火墙开启了过滤机制导致的(net.ipv4.netfilter.ip_conntrack_max 太小),在此将防火墙关闭即可(service iptables stop)

 参考:
[1] https://bugs.launchpad.net/swift/+bug/1183152
[2] http://www.cyberciti.biz/faq/ip_conntrack-table-ful-dropping-packet-error/
[3] http://stackoverflow.com/questions/6240951/sendto-operation-not-permitted-netsnmp

Swift 的详细介绍:请点这里

正文完
星哥说事-微信公众号
post-qrcode
 
星锅
版权声明:本站原创文章,由 星锅 2022-01-20发表,共计3303字。
转载说明:除特殊说明外本站文章皆由CC-4.0协议发布,转载请注明出处。
【腾讯云】推广者专属福利,新客户无门槛领取总价值高达2860元代金券,每种代金券限量500张,先到先得。
阿里云-最新活动爆款每日限量供应
评论(没有评论)
验证码
【腾讯云】云服务器、云数据库、COS、CDN、短信等云产品特惠热卖中