I like to graph things, well not by hand anyway. All the router and switchports are graphed and saved. I was using Cacti for this and to be honest it has served me well for graphing interface traffic.
I have two issues with Cacti :
- The last release (0.8) was 04/29/12
- Graphing non SNMP things is very annoying
With the above two issues I set out to find a better for flexible graphing system. My good friend pointed me towards Graphite, given that he also likes graphs a lot I have taken his advice. As expected its good advise Graphite is incredibly flexible. In short its just a TCP port that accepts data with a time stamp and stores it. Once stored you can create pretty graphs.
Since I am a Gentoo user I wanted to as much as possible use Gentoo's portage system. Graphite is not part of Portage yet but can be found in theover lays. Sadly the Gentoo ebiuld is still a work in progress, thats ok by following my simple guide you will have Graphite up in no time.
1) .Adding the Overlay.
First up add the overlay. The overlay you will use is kormoc
If you don't currently use overlays do the following two steps first :
# emerge layman
Then add the following to your /etc/make.conf
source /var/lib/layman/make.conf
Once thats done you can add the overlay :
# layman -a kormoc
2) Adding Packages.
Now that you have the overlay added its time to emerge the required software. Firstly make sure you have the following USE flags in your /etc/make.conf file :
USE="apache2 php vhosts sqlite"
Then emerge the following :
# emerge dev-python/django # emerge www-servers/apache # emerge graphite-web # emerge www-apache/mod_wsgi # emerge dev-python/django-tagging
You may need to unmask some of the above packages.
3) Fixing UTF-8 issues.
You may get a failed emerge from "dev-python/django-tagging" if you do not currently have a UTF-8 locale installed. If you do this ensure you have a UTF-8 locale installed.
Ensure the file "/etc/locale.gen" has at least one UTF-8 locale set, mine looks like this:
en_US ISO-8859-1 en_US.UTF-8 UTF-8 #ja_JP.EUC-JP EUC-JP #ja_JP.UTF-8 UTF-8 #ja_JP EUC-JP #en_HK ISO-8859-1 #en_PH ISO-8859-1 #de_DE ISO-8859-1 #de_DE@euro ISO-8859-15 #es_MX ISO-8859-1 #fa_IR UTF-8 #fr_FR ISO-8859-1 #fr_FR@euro ISO-8859-15 #it_IT ISO-8859-1
Also ensure your "/etc/env.d/02locale" file has a UTF-8 locale. Mine looks like this :
LANG="en_US.UTF-8"
Once you have set the locale ensure that you re-generate the locale's with :
# locale-gen
You should see something like this :
# locale-gen * Generating 2 locales (this might take a while) with 1 jobs * (1/2) Generating en_US.ISO-8859-1 ... [ ok ] * (2/2) Generating en_US.UTF-8 ... [ ok ] * Generation complete
4) Configuring Carbon.
Now its time to configure the carbon data store application.
Edit the files in the "/etc/carbon/" You will need to copy the file named "carbon.conf.example" to "carbon.conf"
Hear is my "/etc/carbon/carbon.conf" file for reference :
[cache] STORAGE_DIR = /opt/graphite/carbon/ CONF_DIR = /etc/carbon/ LOG_DIR = /var/log/carbon/ LOCAL_DATA_DIR = /opt/graphite/storage/whisper/ USER = apache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = 500 MAX_CREATES_PER_MINUTE = 50 LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2003 ENABLE_UDP_LISTENER = False UDP_RECEIVER_INTERFACE = 0.0.0.0 UDP_RECEIVER_PORT = 2003 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2004 USE_INSECURE_UNPICKLER = False CACHE_QUERY_INTERFACE = 0.0.0.0 CACHE_QUERY_PORT = 7002 USE_FLOW_CONTROL = True LOG_UPDATES = False WHISPER_AUTOFLUSH = False [relay] LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2013 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2014 RELAY_METHOD = rules REPLICATION_FACTOR = 1 DESTINATIONS = 127.0.0.1:2004 MAX_DATAPOINTS_PER_MESSAGE = 500 MAX_QUEUE_SIZE = 10000 USE_FLOW_CONTROL = True [aggregator] LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2023 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2024 DESTINATIONS = 127.0.0.1:2004 REPLICATION_FACTOR = 1 MAX_QUEUE_SIZE = 10000 USE_FLOW_CONTROL = True MAX_DATAPOINTS_PER_MESSAGE = 500 MAX_AGGREGATION_INTERVALS = 5
You can see I have set carbon to run as the apache user and change data and log directory.
Ensure you set the correct permissions on the log directory :
# chown apache:apache /var/log/carbon/
The next file you need to edit is the "/etc/carbon/storage-schemas.conf" file. Make a copy of the storage-schemas.conf.example file and call it "storage-schemas.conf".
This file sets the data retention periods. The my file I am keeping 1min data for 35days and 15mins of data for 10 years. All values are expressed as seconds.
My /etc/carbon/storage-schemas.conf file
[carbon] pattern = ^carbon\. retentions = 60:90d pattern = .* retentions = 60:43200,900:350400
5) Apache and WSGI.
Next up you need to set up apache and the WSGI run directory.
Create the following directory :
# mkdir /run/wsgi
Ensure its owned by the apache user :
chown apache:apache /run/wsgi/
Next add the wsgi module to the apache start up config :
Edit the file :
/etc/conf.d/apache2
Look for the liline that starts with "APACHE2_OPTS=, add in "-D WSGI". for example mine is :
APACHE2_OPTS="-D PHP5 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D WSGI"
Create a default wsgi config file, one has been supplied you just need to copy it.
# cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
Now you need to create a vhosts file for apache. Graphite provides a sample at :
/opt/graphite/examples/example-graphite-vhost.conf
I have edited mine a little bit, but its based off the example. My vhosts file :
<IfDefine DEFAULT_VHOST> Listen 80 NameVirtualHost *:80 <Directory /opt/graphite/webapp> Order deny,allow Allow from all </Directory> WSGISocketPrefix /run/wsgi <VirtualHost *:80> ServerName monitor DocumentRoot "/opt/graphite/webapp" ErrorLog /var/log/apache/error.log CustomLog /var/log/access.log common WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 WSGIProcessGroup graphite WSGIApplicationGroup %{GLOBAL} WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL} WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi Alias /content/ /opt/graphite/webapp/content/ <Location "/content/"> SetHandler None </Location> Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/" <Location "/media/"> SetHandler None </Location> <Directory /opt/graphite/conf/> Order deny,allow Allow from all </Directory> </VirtualHost>
Create the above file at the following location :
/etc/apache2/vhosts.d/graphite-vhost.conf
Lastly ensure the "/opt/graphite" directory is owned by apache :
# chown -R apache:apache /opt/graphite/
6) Carbon Database.
Time to create the database.
Change into the following directory :
# cd /opt/graphite/webapp/graphite/
Next up run the following command :
# python ./manage.py syncdb
This script will ask a few questions, but its quite simple, the output should be like this :
# python ./manage.py syncdb Creating tables ... Creating table account_profile Creating table account_variable Creating table account_view Creating table account_window Creating table account_mygraph Creating table dashboard_dashboard_owners Creating table dashboard_dashboard Creating table events_event Creating table auth_permission Creating table auth_group_permissions Creating table auth_group Creating table auth_user_user_permissions Creating table auth_user_groups Creating table auth_user Creating table django_session Creating table django_admin_log Creating table django_content_type Creating table tagging_tag Creating table tagging_taggeditem You just installed Django's auth system, which means you don't have any superusers defined. Would you like to create one now? (yes/no): yes Username (leave blank to use 'root'): E-mail address: Error: That e-mail address is invalid. E-mail address: YOUR_EMAIL@EXAMPLE.COM Password: Password (again): Superuser created successfully. Installing custom SQL ... Installing indexes ... Installed 0 object(s) from 0 fixture(s) #
One thing to note is that the "superuser" you create will also be the user you log into the web site with. You can create more users later on.
7) Setting up the local settings file.
A there is an example "local_setting.py" settings file provided in "/opt/graphite/webapp/graphite/local_settings.py.example". I have edited this and saved it as "/opt/graphite/webapp/graphite/local_settings.py"
The contests of my local_settings.py are :
TIME_ZONE = 'Asia/Hong_Kong' LOG_RENDERING_PERFORMANCE = True LOG_CACHE_PERFORMANCE = True LOG_METRIC_ACCESS = True DATABASES = { 'default': { 'NAME': '/opt/graphite/storage/graphite.db', 'ENGINE': 'django.db.backends.sqlite3', 'USER': '', 'PASSWORD': '', 'HOST': '', 'PORT': '' } }
Save that and move on to the next step.
8) Starting up Graphite.
How do you start it all up? Apache is simple :
# /etc/init.d/apache2 start
Ensure its set to run on next boot :
# rc update add apache2 default
What about Carbon-cache.... There seems to be no init script for this yet. I have written a very simple one.
Create the file :
/etc/init.d/carbon-cache
Enter the following into the above file ;
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | #!/sbin/runscript # Writen by Brendan Horan # blog.horan.hk depend() { need net use logger } checkconfig() { if [ ! -f /etc/carbon/carbon.conf ] ; then eerror "Please create /etc/carbon/carbon.conf" eerror "Sample conf: /etc/carbon/carbon.conf.example" return 1 fi return 0 } start() { checkconfig || return $? ebegin "Starting carbon-cache" start-stop-daemon --start -- /usr/bin/carbon-cache.py --config=/etc/carbon/carbon.conf start \ --pidfile /var/run/carbon-cache.pid \ -- -p /var/run/carbon-cache.pid eend $? "Failed to start carbon-cache" } stop() { ebegin "Stopping carbon-cache" start-stop-daemon --stop \ --pidfile /var/run/carbon-cache.pid \ --exec /usr/bin/carbon-cache.py stop eend $? "Failed to stop carbon-cache" } |
Ensure the file has execuite bit set :
# chmod +x/etc/init.d/carbon-cache
Start it with :
# /etc/init.d/carbon-cache start
Add it to run on next boot :
# rc-update add carbon-cache default
Thats it! Just browse to your host on port 80 and you will find a blank Graphite install waiting to serve you.
9) Testing it out.
I am sure you want to test it...
Its very simple to test Graphite, recall it just wants a metric and a time stamp? You can use three tools that will be on every Genoo install to test. They are "echo", "date" and busybox's "nc". Run the following command on the host running Graphite/Carbon :
# echo "local.random.diceroll 900 `date +%s`" | busybox nc localhost 2003
This will create a metric called "local.random.diceroll" with a value of "900" at the current date and time.
You can see where I have created to test data points at 900 and 600.
Your all done and tested. I will write more on how to get some real data into Graphite soon.