Foreman dashboard to Graphite

Daniel Schutterop
19-07-2015

I currently use Foreman as our life cycle management tool, and to be honest, it’s a great tool. Using Foreman (or rather, Foreman’s API) and Puppet/Chef integration you can spin up machines in seconds -or minutes-, provision them and nuke them when you’re done. That is, if you’re prepared to use the API.

API?

Foreman’s API is the most flexible way to use the core functionality of Foreman and it makes your life a lot easier too. For example: using Python I’ve written a wrapper around the Foreman API that creates, builds, and provisions vmWare guests completely based on their VM guest name. Even better, using Puppet you can ensure the existence of the VM’s and build them if they were accidentally deleted. In short: Foreman makes infrastructure as code extremely easy.

Downside

Well, there isn’t much of a downside, really, using the API. The only thing I didn’t like when starting out with Foreman was the status dashboard (the default landing page). I can appreciate the information (it’s always nice to see how many hosts are OK or in Error state) but the dashboard is, well, just another dashboard.

Every additional management tool I need to look at during the day is one too many, what I really like is unifying all management information in one dashboard (which is also why I think OpsTheatre is a promising tool and I really hope it will take off!).

Anyway, one dashboard. OpsTheatre is a little too fresh for me right now, so it’s Graphite and Grafana.

Sending metrics to Graphite

Few things are easier than sending data to Graphite (well, you actually send data to Carbon but I’ll refer to Carbon/Graphite as Graphite for argument’s sake). All you do is send three pieces of information to Graphite:

  • Metric Path
  • Metric Value
  • Metric Timestamp

If you put these three components in a string and fire it in the general direction of your Graphite server (more specifically, to the listener on your Graphite server, usually TCP 2003), you can find the data you just submitted in Graphite.

Sending Foreman metrics to Graphite

With the above in mind, I’ve written a tiny piece of Python code to automate stuff.

What it does

It’s really quite simple. It does an API call to the Foreman machine and gets the Dashboard data (JSON), creates three-piece-strings that Graphite loves, opens a connection to Graphite and drops the performance metrics there.

The script does this in the background, by default every 60 seconds.

What it needs

I didn’t take too much time writing the script, so I it’s not really as neat and tidy as I would normally write stuff.

Parameters

No command line arguments, you’ll need to edit the following data in the script:

  • Foreman Host
  • Foreman User (API authentication)
  • Foreman Password (API authentication)
  • Graphite Host
  • Graphite Port (Default 2003)
  • Graphite Database (foreman by default)

Python Modules

To grab the JSON from Foreman I use the Python Requests module. All other modules should be included in a default Python 2.7 deployment.

#!/usr/bin/python
#
# == Synopsis
# Quick and dirty solution to grab
# dashboard data from Foreman
# and stuff it into Graphite / Carbon
#
# Works with: Foreman             1.7
#             Graphite-web/Carbon 0.9.12
#             Python              2.7
#
# === Workflow
# This script uses the Foreman API to
# download dashboard information, after
# which it puts the data in a Carbon-accepted
# form into carbon
#
# Carbon only needs three things:
# <metric> <value> <timestamp>
#
# So what we do is grab the Foreman dashboard
# key as the metric, grab the dashboard-key value
# as the value and make up our own timestamp. Well,
# actually, that's done through time()
#
# Author  : D. Schutterop
# Email   : daniel@schutterop.nl
# Version : v0.1
#
import json,requests,time,socket,os,sys

runInterval = 60

frmHost     = 'foreman.mydomain.com'
frmUser     = 'foremanUser'
frmPass     = 'changemeToAValidPassword'

grpHost     = 'graphite.mydomain.com'
grpPort     = 2003
grpDatabase = 'foreman'

#Suppress SSL warnings generated when contacting Foreman
requests.packages.urllib3.disable_warnings()

def frmGetData(frmHost,frmUser,frmPass):
  frmUrl     = "https://%s/api/v2/dashboard" % frmHost
  frmHeaders = {'Content-type': 'application/json'}
  frmRequest = requests.get(frmUrl, verify=False, auth=(frmUser,frmPass), headers=frmHeaders)

  return json.loads(frmRequest.text)
def grpPutMessage(grpMetricKey,grpMetricValue):
  metricPrepend = grpDatabase
  metricAppend  = grpMetricKey
  metricKey     = "%s.%s" % (metricPrepend,grpMetricKey)
  metricTime    = int(time.time())

  metricValue   = grpMetricValue

  return "%s %s %s" % (metricKey,metricValue,metricTime)

def run(runInterval):
  while True:
    grpSocket = socket.socket()
    grpSocket.connect((grpHost,grpPort))

    frmData = frmGetData(frmHost,frmUser,frmPass)
    message = ' '
    for listItem in frmData:
      message = "\n %s %s" % (grpPutMessage(listItem,frmData[listItem]),message)
    grpSocket.sendall(message)
    grpSocket.close()

    time.sleep(runInterval)

if __name__ == "__main__":

  procPid = os.fork()

  if procPid != 0:
    sys.exit(0)

  print ("Running %s every %s seconds in the background." % (__file__,runInterval))
  run(runInterval)

When you run this, you’ll likely see a new folder being created in your Graphite tree. If not, something went wrong. The actions performed by this script were simple enough for me not to write any debugging or logging functions. If you think this is a grave mistake, just let me know, OK? 🙂

In the end, it’s easy as Pi.

LEAVE A REPLY

1 Comment

  • Lauxtermann says:

    Really liked the article. I used Graphite with PL/SQL, elasticSearch and Java. Have you ever worked with Spark/HDFS/Python? The possibilities for DevOps are remarkable!

you might also like