Using Python to Log Data to Loggly

One of my readers suggested that I should try logging my data to a web service called Loggly. As I understand it, Loggly is a way to share log data with everyone in a business so that you no longer need to log in to individual machines. They also provide graphs, filters and searches of the logs. They don’t have a Python API, but it’s still pretty easy to send data to Loggly via Pythons urllib2 module and simplejson. Also note that you can use Loggly for 30-day trial period.

Let’s take a look at some code. This code is based on the code from my article aboud logging currently running processes. You will need the following modules installed for this example to work:

• psutil
• simplejson

I just used pip to install them both. Now that we have those, let’s see how to use them to connect to Loggly

import psutil
import simplejson
import time
import urllib2

#----------------------------------------------------------------------
def log():
    """"""
    token = "YOUR-LOGGLY-TOKEN"
    url = "https://logs-01.loggly.com/inputs/%s/tag/python/" % token
    
    while True:
        proc_dict = {}
        procs = psutil.get_process_list()
        procs = sorted(procs, key=lambda proc: proc.name)
        for proc in procs:
            cpu_percent = proc.get_cpu_percent()
            mem_percent = proc.get_memory_percent()
            try:
                name = proc.name()
            except:
                # this is a process with no name, so skip it
                continue
            
            data = {"cpu_percent": cpu_percent,
                    "mem_percent": mem_percent,
                    }
            proc_dict[name] = data
        
        log_data = simplejson.dumps(proc_dict)
        urllib2.urlopen(url, log_data)
        time.sleep(60)
            
if __name__ == "__main__":
    log()

This is a pretty simple function, but let’s break it down anyway. First off we set out Loggly token and create a Loggly URL to send our data to. Then we create an infinite loop that will grab a list of currently running processes every 60 seconds. Next we extract out the bits that we want to log and then we put those pieces into a dictionary of dictionaries. Finally we use simplejson’s dumps method to turn our nested dictionary into a JSON formatted string and we pass that to our url. This send the log data to Loggly where it can get parsed.

Once you’ve sent enough data to Loggly for it to analyze, you can login to your account and see some bar charts that are created automatically based on your data. My data in this example didn’t translate very well into graphs or trends, so those ended up looking pretty boring. I would recommend sending a system log or something else that contains more variety in it to get a better understanding of how useful this service would be for you.

UPDATE: The people at Loggly mentioned trying to send the data as floats instead of strings, so I edited the code above appropriately. Feel free to play around with the code and see what you come up with.

Related Code

• One of my readers came up with a modified version of my example