Graphite and Gentoo

I like to graph things, well not by hand anyway. All the router and switchports are graphed and saved. I was using Cacti for this and to be honest it has served me well for graphing interface traffic.

I have two issues with Cacti :

The last release (0.8) was 04/29/12
Graphing non SNMP things is very annoying

With the above two issues I set out to find a better for flexible graphing system. My good friend pointed me towards Graphite, given that he also likes graphs a lot I have taken his advice. As expected its good advise Graphite is incredibly flexible. In short its just a TCP port that accepts data with a time stamp and stores it. Once stored you can create pretty graphs.

Since I am a Gentoo user I wanted to as much as possible use Gentoo's portage system. Graphite is not part of Portage yet but can be found in theover lays. Sadly the Gentoo ebiuld is still a work in progress, thats ok by following my simple guide you will have Graphite up in no time.

1) .Adding the Overlay.

First up add the overlay. The overlay you will use is kormoc

If you don't currently use overlays do the following two steps first :

# emerge layman

Then add the following to your /etc/make.conf

source /var/lib/layman/make.conf

Once thats done you can add the overlay :

# layman -a kormoc

2) Adding Packages.

Now that you have the overlay added its time to emerge the required software. Firstly make sure you have the following USE flags in your /etc/make.conf file :

USE="apache2 php vhosts sqlite"

Then emerge the following :

# emerge dev-python/django
# emerge www-servers/apache
# emerge graphite-web
# emerge www-apache/mod_wsgi
# emerge dev-python/django-tagging

You may need to unmask some of the above packages.

3) Fixing UTF-8 issues.

You may get a failed emerge from "dev-python/django-tagging" if you do not currently have a UTF-8 locale installed. If you do this ensure you have a UTF-8 locale installed.

Ensure the file "/etc/locale.gen" has at least one UTF-8 locale set, mine looks like this:

en_US ISO-8859-1
en_US.UTF-8 UTF-8
#ja_JP.EUC-JP EUC-JP
#ja_JP.UTF-8 UTF-8
#ja_JP EUC-JP
#en_HK ISO-8859-1
#en_PH ISO-8859-1
#de_DE ISO-8859-1
#de_DE@euro ISO-8859-15
#es_MX ISO-8859-1
#fa_IR UTF-8
#fr_FR ISO-8859-1
#fr_FR@euro ISO-8859-15
#it_IT ISO-8859-1

Also ensure your "/etc/env.d/02locale" file has a UTF-8 locale. Mine looks like this :

LANG="en_US.UTF-8"

Once you have set the locale ensure that you re-generate the locale's with :

# locale-gen

You should see something like this :

# locale-gen 
 * Generating 2 locales (this might take a while) with 1 jobs
 *  (1/2) Generating en_US.ISO-8859-1 ... [ ok ]
 *  (2/2) Generating en_US.UTF-8 ... [ ok ]
 * Generation complete

4) Configuring Carbon.

Now its time to configure the carbon data store application.

Edit the files in the "/etc/carbon/" You will need to copy the file named "carbon.conf.example" to "carbon.conf"

Hear is my "/etc/carbon/carbon.conf" file for reference :

[cache]
STORAGE_DIR    = /opt/graphite/carbon/
CONF_DIR       = /etc/carbon/
LOG_DIR        = /var/log/carbon/
LOCAL_DATA_DIR = /opt/graphite/storage/whisper/
USER = apache
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500
MAX_CREATES_PER_MINUTE = 50
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
USE_FLOW_CONTROL = True
LOG_UPDATES = False
WHISPER_AUTOFLUSH = False


[relay]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
RELAY_METHOD = rules
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True


[aggregator]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5

You can see I have set carbon to run as the apache user and change data and log directory.

Ensure you set the correct permissions on the log directory :

# chown apache:apache /var/log/carbon/

The next file you need to edit is the "/etc/carbon/storage-schemas.conf" file. Make a copy of the storage-schemas.conf.example file and call it "storage-schemas.conf".

This file sets the data retention periods. The my file I am keeping 1min data for 35days and 15mins of data for 10 years. All values are expressed as seconds.

My /etc/carbon/storage-schemas.conf file

[carbon]
pattern = ^carbon\.
retentions = 60:90d

pattern = .*
retentions = 60:43200,900:350400

5) Apache and WSGI.

Next up you need to set up apache and the WSGI run directory.

Create the following directory :

# mkdir /run/wsgi

Ensure its owned by the apache user :

chown apache:apache /run/wsgi/

Next add the wsgi module to the apache start up config :

Edit the file :

/etc/conf.d/apache2

Look for the liline that starts with "APACHE2_OPTS=, add in "-D WSGI". for example mine is :

APACHE2_OPTS="-D PHP5 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D WSGI"

Create a default wsgi config file, one has been supplied you just need to copy it.

# cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi

Now you need to create a vhosts file for apache. Graphite provides a sample at :

/opt/graphite/examples/example-graphite-vhost.conf

I have edited mine a little bit, but its based off the example. My vhosts file :

<IfDefine DEFAULT_VHOST>
Listen 80
NameVirtualHost *:80

<Directory /opt/graphite/webapp>
Order deny,allow
Allow from all
</Directory>


WSGISocketPrefix /run/wsgi

<VirtualHost *:80>
        ServerName monitor
        DocumentRoot "/opt/graphite/webapp"
        ErrorLog /var/log/apache/error.log
        CustomLog /var/log/access.log common

        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi 

        Alias /content/ /opt/graphite/webapp/content/
        <Location "/content/">
                SetHandler None
        </Location>

        Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        <Directory /opt/graphite/conf/>
                Order deny,allow
                Allow from all
        </Directory>

</VirtualHost>

Create the above file at the following location :

/etc/apache2/vhosts.d/graphite-vhost.conf

Lastly ensure the "/opt/graphite" directory is owned by apache :

# chown -R apache:apache /opt/graphite/

6) Carbon Database.

Time to create the database.

Change into the following directory :

# cd /opt/graphite/webapp/graphite/

Next up run the following command :

# python ./manage.py syncdb

This script will ask a few questions, but its quite simple, the output should be like this :

# python ./manage.py syncdb
Creating tables ...
Creating table account_profile
Creating table account_variable
Creating table account_view
Creating table account_window
Creating table account_mygraph
Creating table dashboard_dashboard_owners
Creating table dashboard_dashboard
Creating table events_event
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_session
Creating table django_admin_log
Creating table django_content_type
Creating table tagging_tag
Creating table tagging_taggeditem

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): 
E-mail address: 
Error: That e-mail address is invalid.
E-mail address: YOUR_EMAIL@EXAMPLE.COM
Password: 
Password (again): 
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
#

One thing to note is that the "superuser" you create will also be the user you log into the web site with. You can create more users later on.

7) Setting up the local settings file.

A there is an example "local_setting.py" settings file provided in "/opt/graphite/webapp/graphite/local_settings.py.example". I have edited this and saved it as "/opt/graphite/webapp/graphite/local_settings.py"

The contests of my local_settings.py are :

TIME_ZONE = 'Asia/Hong_Kong'
LOG_RENDERING_PERFORMANCE = True
LOG_CACHE_PERFORMANCE = True
LOG_METRIC_ACCESS = True

DATABASES = {
    'default': {
        'NAME': '/opt/graphite/storage/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

Save that and move on to the next step.

8) Starting up Graphite.

How do you start it all up? Apache is simple :

# /etc/init.d/apache2 start

Ensure its set to run on next boot :

# rc update add apache2 default

What about Carbon-cache.... There seems to be no init script for this yet. I have written a very simple one.

Create the file :

/etc/init.d/carbon-cache

Enter the following into the above file ;

#!/sbin/runscript
# Writen by Brendan Horan
# blog.horan.hk


depend() {
        need net
        use logger
}


checkconfig() {
        if [ ! -f /etc/carbon/carbon.conf ] ; then
                eerror "Please create /etc/carbon/carbon.conf"
                eerror "Sample conf: /etc/carbon/carbon.conf.example"
                return 1
        fi
        return 0
}

start() {
        checkconfig || return $?

        ebegin "Starting carbon-cache"
        start-stop-daemon --start --  /usr/bin/carbon-cache.py --config=/etc/carbon/carbon.conf start \
            --pidfile /var/run/carbon-cache.pid \
            -- -p /var/run/carbon-cache.pid
        eend $? "Failed to start carbon-cache"
}

stop() {
        ebegin "Stopping carbon-cache"
        start-stop-daemon --stop \
                --pidfile /var/run/carbon-cache.pid \
                --exec /usr/bin/carbon-cache.py stop
        eend $? "Failed to stop carbon-cache"
}

Ensure the file has execuite bit set :

# chmod +x/etc/init.d/carbon-cache

Start it with :

# /etc/init.d/carbon-cache start

Add it to run on next boot :

# rc-update add carbon-cache default

Thats it! Just browse to your host on port 80 and you will find a blank Graphite install waiting to serve you.

9) Testing it out.

I am sure you want to test it...

Its very simple to test Graphite, recall it just wants a metric and a time stamp? You can use three tools that will be on every Genoo install to test. They are "echo", "date" and busybox's "nc". Run the following command on the host running Graphite/Carbon :

# echo "local.random.diceroll 900 `date +%s`" | busybox nc localhost 2003

This will create a metric called "local.random.diceroll" with a value of "900" at the current date and time.

You can see where I have created to test data points at 900 and 600. Example

Your all done and tested. I will write more on how to get some real data into Graphite soon.