Wednesday, June 21, 2017

Zabbix and Grafana

zabbix

Install Zabbix 3.0 and Grafana in Ubuntu 14.04

I cannot install Zabbix 3.0 in Ubuntu16.04, since some package dependency problem. So I install Zabbix 3.0 in Ubuntu 14.04.

Pre-Install

apt-get update
apt-get install apache2
apt-get install mysql-server   ## input password
apt-get install php5 php5-cli php5-common php5-mysql

Install Mariadb

If you prefer mariadb db instead of mysql, you can use the following commands.

apt-get install software-properties-common
apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db
add-apt-repository 'deb http://sfo1.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu trusty main'
apt-get update
apt-get install galera mariadb-galera-server-10.0

Edit apache2

edit /etc/php5/apache2/php.ini

[Date]
; http://php.net/date.timezone
date.timezone = 'Asia/Taipei'
echo "ServerName localhost" | tee /etc/apache2/conf-available/fqdn.conf
a2enconf fqdn
wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb
dpkg -i zabbix-release_3.0-1+trusty_all.deb
apt-get update
apt-get install zabbix-server-mysql zabbix-frontend-php

Setup Database

mysql -u root -ppassword
mysql> CREATE DATABASE zabbixdb;
mysql> GRANT ALL on zabbixdb.* to zabbix@localhost IDENTIFIED BY 'password';
mysql> FLUSH PRIVILEGES;
mysql> quit;
cd /usr/share/doc/zabbix-server-mysql
zcat create.sql.gz | mysql -u root -p zabbixdb    ##enter password

Setup Zabbix-Server

edit /etc/zabbix/zabbix_server.conf

 DBHost=localhost
 DBName=zabbixdb
 DBUser=zabbix
 DBPassword=password
service apache2 restart
service zabbix-server restart

Connect to Zabbix Server

connect to Zabbix from Browser

http://172.16.155.167/zabbix

SetUp auto Registration

For Linux

Set up AutoRegistration and its Action

Configuration->Action->Event source->Auto registration-> Create Action
Name: Linux host autoregistration
Conditions: Host metadata like Linux
Operations: Link to templates: Template OS Linux

You might add more Roles of Action likes DB in Zabbix Server, and adding DB to Clinet, see later.

Name: Linux host autoregistration
Conditions: Host metadata like DB
Operations: Link to templates: Template App MySQL

For Windows

Name: Windows host autoregistration
Conditions: Host metadata like Windows
Operations: Link to templates: Template OS Windows

Set up Network Monitoring

Configuration->Templaes->Templte OS Linux->Discovery -> Nework interface discovery-> update interval 30 sec

Install Zabbix Agent

Quick Install

apt-get install zabbix-agent

Install current version

wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix/zabbix-agent_3.0.9-1+trusty_amd64.deb
dpkg -i zabbix-agent_3.0.9-1+trusty_amd64.deb
apt-get update

Edit Zabbix-agent configuration

vim /etc/zabbix/zabbix_agentd.conf

Server=172.16.155.167
ServerActive=172.16.155.167
#Hostname=inqencdir2
HostMetadata=Linux

You can disable Hostname for using system hostname

You can add more roles here, such as DB.

HostMetadata=Linux DB

Restart zabbix-agent

service zabbix-agent restart

Set up Screen

We can plot multiple server

Screen->create Screen->Name:overall->Columns:2 -> Rows:2-> Add
Edit Screen -> Change ->Resource: Data overview ->Group:Discoverd Hosts->ADD

You will see the data. And you can add more blocks with

Edit Screen -> + ->Change 

Set up Alarm

Administration ->Email-> Name:Email->SMTP server:10.60.94.110->SMTP serverport:25->SMTPhelo:company.com->
SMTP email:zabbix@inquartik.com ->Update
Administration -> Users ->Admin-> Media -> add -> jonahwu@gmail.com ->Update
Configuration->action -> EventSource:trigger->create Action -> Name:test1->Condition->New Condition->xxxx

Adding Trigger

Adding trigger to Template "Linux OS"

Configuration->Templates -> Template OS Linux->Triggers->Create Trigger->
Name: testtriggercpu-> Expression (add)->Item->Select->Group:Template->Host:Template OS Linux
->CPU user time->Function:MaxmunValue For Period T is > N ->Last of(T):1m -> N:80 -> Insert
Severity: High ->Add

To see all the Trigger

Configuration -> Templates-> Templates OS Linux ->Triggers

Result

Configuration-> Hosts

You will see the new server added to column and with Templates information. If not, it's wrong.

Download Agent

If you need agent with 3.0, you need download it from the following url.

http://repo.zabbix.com/zabbix/3.2/ubuntu/pool/main/z/zabbix/

In CentOS

http://repo.zabbix.com/zabbix/3.0/rhel/6/x86_64/
 rpm -Uvh http://repo.zabbix.com/zabbix/3.0/rhel/6/x86_64/zabbix-agent-3.0.9-1.el6.x86_64.rpm
 yum install zabbix-agent

enable zabbix-agent after reboot.

chkconfig zabbix-agent on

make sure port enable

[root@centosz ~]# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:10050               0.0.0.0:*                   LISTEN      4564/zabbix_agentd
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      2189/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1985/cupsd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2320/master
tcp        0      0 :::10050                    :::*                        LISTEN      4564/zabbix_agentd
tcp        0      0 :::22                       :::*                        LISTEN      2189/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1985/cupsd
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               1982/dhclient
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1985/cupsd

Make sure remove all iptables.

iptables --flush
service iptables save

Loggin

setfacl -m u:zabbix:r-- /var/log/syslog
or
chmod 777 /var/log/syslog

Trouble Shooting

In server check client agent

where 192.168.30.131 is client IP.

/usr/bin/zabbix_get -s 192.168.30.131 -p 10050 -k "system.hostname"

vi /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 10050 -j ACCEPT

adding above line after 22 port2

-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW,ESTABLISHED -m tcp --dport 10050 -j ACCEPT

Install Grafana

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.3.1_amd64.deb
apt-get install -y adduser libfontconfig
dpkg -i grafana_4.3.1_amd64.deb
service grafana-server start
grafana-cli plugins install alexanderzobnin-zabbix-app
service grafana-server restart
update-rc.d grafana-server defaults 95 10

Connect to Grafana

http://172.16.155.167:3000/
login: admin 
password:admin
select Plugins -> pressing enable button
add data source 

http settings:
url http://your_zabbix_server_ip_address/zabbix/api_jsonrpc.php

Zabbix API details
Username: admin
Password: zabbix

Add

Save&Test

Create Customer Zabbix Dashboard

Home->Create New->Graph
Press Panel Title -> Edit ->DataSource:Default->DataSource:Zabbix
Group->Discoverd Hosts->Host->choose your host

Ctrl+S

Grafana Host Setting

Using Templating Mechanism is necessary, since most of role is cluster. So we need rendering with group concept.

Adding Group filter

Manage Dashboard->Templating -> New
Name:Group
DataSource:zabbix
Regex:        (empty)
Query:*
Multi-value: v
include all option: v
Save

You might see the Preview of values (shows max 20) to make sure your setting is correct.

Adding Host filter.

Name:Host
Data Source: zabbix
Query:$Group.*
Regex:^zabbix.*        #if you want to show all the hostname are zabbix(*) 
Multi-value: v
include All value: v
Save

You might see the Preview of values (shows max 20) to make sure your setting is correct.

Modify Panel

Modify all Host parameter in planel with $Host

Host: $Host

$Host is defined by Host filter

Result

You can choose different host or all to render the metrics.

The Benefit of importing/export template and modify by templating

Export

Share Dashboard->Export->Save to File

Import

Main->Dashborads->imports

Quit Deployment

Using templating can speed up your deploy, you can just import a template and modify

Templating -> Host->Edit -> Regex -> ^[new host type].*

such as
Templating -> Host->Edit -> Regex -> ^mysql.*
or 
Templating -> Host->Edit -> Regex -> .*mysql.*

You then can start to monitor the joined cluster.

No comments:

Post a Comment