• 问题描述

zabbix运行一段大概半年有余,突然有一天发现有台proxy代理过来的主机最新数据显示的是昨天的,今天的数据一直在队列里堆积着。

分享一下我的排查过程

  • 排查过程

1). 查看proxy该台的zabbix proxy服务端口状态(netstat -luntp)

2).查看proxy该台的zabbix proxy日志(并未发现有异常错误)

3).zabbix_get 命令获取键值(可以获取)

4).查看zabbix proxy配置文件参数是否有优化(因为其他proxy正常,唯独该台有问题,所有就没看zabbix server日志)

5).查看数据库错误日志(发现报错)

  • 数据库日志报错如下
25T06:00:34.936084Z 0 [Warning] Could not increase number of max_open_files to more than 5000 (request: 65535)
2020-05-25T06:00:34.936105Z 0 [Warning] Changed limits: max_connections: 4190 (requested 5000)
2020-05-25T06:00:34.936125Z 0 [Warning] Changed limits: table_open_cache: 400 (requested 2048)
2020-05-25T06:00:35.102474Z 0 [Warning] You need to use --log-bin to make --binlog-format work.
2020-05-25T06:00:35.110672Z 0 [Warning] InnoDB: innodb_open_files should not be greater than the open_files_limit.

2020-05-25T06:00:35.844579Z 0 [Warning] CA certificate ca.pem is self signed.
2020-05-25T06:00:35.860792Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2020-05-25T06:00:35.860832Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2020-05-25T06:00:35.860845Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2020-05-25T06:00:35.873878Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2020-05-25T06:00:35.873889Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2020-05-25T06:00:35.873939Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2020-05-25T06:00:35.898765Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2020-05-25T06:00:35.898782Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.

询问才发现是前人用是yum装数据库的,并未优化过。。。

  • 解决
# ulimit -n
131072
vim /usr/lib/systemd/system/mysqld.service
LimitNOFILE = 5000 
#修改为LimitNOFILE = 65535

vim /etc/security/limits.conf
* soft nofile 131072
* hard nofile 131072
* soft memlock unlimited
* hard memlock unlimited

最后修改:2020 年 06 月 02 日 06 : 45 PM
如果觉得我的文章对你有用,请随意赞赏