How Install Grafana, Graphite, and Statsd on Ubuntu 18.04

Introduction

Monitoring is an important part of production grade systems to ensure that your server infrastructure and applications are healthy and functioning as they should be. This tutorial will help you to set up your own monitoring infrastructure using Grafana, Graphite, and Statsd.

Grafana is an open-source tool to visualize time series data and create alerts on top of it. Grafana supports 50+ types of data sources and Graphite is one of them. Graphite is used to store the time series data and it supports various types of data feeders. Statsd server is one of such data feeders which listens for the incoming data from the clients over UDP connection.

Prerequisites

• Cloud VPS or Dedicated Server with at least 1GB RAM and Ubuntu 18.04 installed.
• You must be logged in via SSH as sudo or root user. This tutorial assumes that you are logged in as a sudo user.

Step 1: Update the System

Update the system with the latest packages and security patches using these commands.

sudo apt update
sudo apt upgrade -y

Step 2: Install Grafana

Grafana can be installed using Linux binaries as well as its official repository. In this tutorial, we will be using the official Grafana repository to install the software so that we can easily upgrade them in the future.

Add Grafana GPG key into the system to ensure unaltered packages are being installed.

curl https://packages.grafana.com/gpg.key | sudo apt-key add -

Add Grafana repository into the system.

sudo apt install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

Update the repository cache and install the software by running the following commands.

sudo apt update
sudo apt install -y grafana

Step 3: Start Grafana

Grafana provides the systemd service files hence you can easily start Grafana by running the command.

sudo systemctl start grafana-server

To enable Grafana to automatically start at boot time and application failures, run.

sudo systemctl enable grafana-server

Grafana is now installed. Navigate to http://192.168.0.101:3000/  in your browser and log in with username admin and password admin Make sure you replace 192.168.0.101 with your actual IP address of Snel VPS or Server.

Step 4: Install Graphite

Graphite consists of three separate application, Carbon, Whisper and Graphite Web.
Install build dependencies which are required to compile a few Graphite components by running.

sudo apt -y install python-dev python-pip libcairo2-dev libffi-dev build-essential

Install Graphite along with all its components by running.

export PYTHONPATH="/opt/graphite/lib/:/opt/graphite/webapp/"
sudo -H pip install --no-binary=:all: https://github.com/graphite-project/whisper/tarball/master
sudo -H pip install --no-binary=:all: https://github.com/graphite-project/carbon/tarball/master
sudo -H pip install --no-binary=:all: https://github.com/graphite-project/graphite-web/tarball/master

Create an initial schema for Graphite by running the command.

sudo -H PYTHONPATH=/opt/graphite/webapp django-admin.py migrate --settings=graphite.settings --run-syncdb

Create superuser for Graphite by running the following command. You will need to provide a username, password, and email for this.

sudo -H PYTHONPATH=/opt/graphite/webapp django-admin.py createsuperuser --settings=graphite.settings

Generate a static file for Graphite web app by running the command.

sudo -H PYTHONPATH=/opt/graphite/webapp django-admin.py collectstatic --noinput --settings=graphite.settings

Finally, copy carbon configuration files by running.

sudo cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
sudo cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf

Step 5: Configure Apache with mod_wsgi

Install Apache with mod_wsgi.

sudo apt install -y apache2 libapache2-mod-wsgi

Enable mod_wsgi module in Apache.

sudo a2enmod wsgi

Create wsgi file at /opt/graphite/conf/graphite.wsgi using your choice of editor.

sudo nano /opt/graphite/conf/graphite.wsgi

Populate the file with the following configuration.

import sys

sys.path.append('/opt/graphite/webapp')
from graphite.wsgi import application

Provide this file execution permission and change the Graphite directory ownership to Apache user by running.

sudo chmod +x /opt/graphite/conf/graphite.wsgi
sudo chown -R www-data:www-data /opt/graphite

Create a new Apache virtual host file using the command.

sudo nano /etc/apache2/sites-available/graphite-vhost.conf

Populate the file with the following configuration.

LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>

    ServerName graphite
    DocumentRoot "/opt/graphite/webapp"
    ErrorLog /opt/graphite/storage/log/webapp/error.log
    CustomLog /opt/graphite/storage/log/webapp/access.log common

    WSGIDaemonProcess graphite-web processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
    WSGIProcessGroup graphite-web
    WSGIApplicationGroup %{GLOBAL}
    WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite-web application-group=%{GLOBAL}

    WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

    Alias /static/ /opt/graphite/static/

    <Directory /opt/graphite/static/>
            <IfVersion < 2.4>
                    Order deny,allow
                    Allow from all
            </IfVersion>
            <IfVersion >= 2.4>
                    Require all granted
            </IfVersion>
    </Directory>

    <Directory /opt/graphite/conf/>
            <IfVersion < 2.4>
                    Order deny,allow
                    Allow from all
            </IfVersion>
            <IfVersion >= 2.4>
                    Require all granted
            </IfVersion>
    </Directory>
</VirtualHost>

Disable default virtual host and enable the virtual host we just created by running.

sudo a2dissite 000-default
sudo a2ensite graphite-vhost

Now restart Apache web server for changes to reflect.

sudo systemctl restart apache2

Now you can access your Graphite web app at http://192.168.0.101/ Make sure to replace 192.168.0.101 with your actual IP address of Snel VPS or Server. You can log in using the credentials we created at step 4 of the tutorial. We still need to start the Carbon server, which we will do at step 7 of the tutorial.

Step 6: Install Statsd

Statsd requires Node.js to work. Configure Node.js 10.x repository by running the command.

curl -L -s https://deb.nodesource.com/setup_10.x | sudo bash

Install Node.js and Git.

sudo apt install -y nodejs git

Clone Statsd git repository.

sudo git clone https://github.com/etsy/statsd.git /opt/statsd

Create new Statsd configuration file by running.

sudo nano /opt/statsd/localConfig.js

Paste the following configuration in it.

{
    graphitePort: 2003,
    graphiteHost: "127.0.0.1",
    port: 8125,
    backends: [ "./backends/graphite" ]
}

Step 7: Set up Supervisor

For running Statsd and Carbon server we need to set up Supervisor which will keep running these servers for us.

Install Supervisor.

sudo apt install -y supervisor

Create a new supervisor config for Statsd server.

sudo nano /etc/supervisor/conf.d/statsd.conf

Put the following configuration in the editor.

[program:statsd]
directory = /opt/statsd/
command = /usr/bin/node stats.js localConfig.js
autostart=true
autorestart=true

Create another supervisor config for Carbon server.

sudo nano /etc/supervisor/conf.d/carbon.conf

Put the following configuration in the editor.

[program:carbon]
directory=/opt/graphite
command=/usr/bin/python bin/carbon-cache.py --debug start
user=www-data
autostart=true
autorestart=true

Restart supervisor and enable it to start automatically at boot time by running the commands.

sudo systemctl restart supervisor
sudo systemctl enable supervisor

Conclusion

Your perfect monitoring server is now ready. Statsd is listening on port 8125/udp. You can use any statsd client to feed data into the server. Data will be stored in Whisper database in Graphite. Add Graphite as a data source in Grafana and start creating beautiful graphs. You can also add alerts on the graphs so that you will immediately know if any service or machine is behaving abnormally.