Parsing XML with Python using lxml.objectify

Posted by Mike on June 6th, 2012 filed in Cross-Platform, Python, Web

A couple years ago I started a series of articles on XML parsing. I covered lxml’s etree and Python’s included minidom XML parsing library. For whatever reason I didn’t notice lxml’s objectify sub-package, but I saw it recently and decided I should check it out. To my mind, the objectify module seems to be even more “Pythonic” than etree is. Let’s take a some time and go over my old XML examples using objectify and see how it’s different!

Let’s Get This Party Started!

If you haven’t already, go out and download lxml or you won’t be able to follow along very well. Once you have it, we can continue. We’ll be using the following piece of XML for our parsing pleasure:

<?xml version="1.0" ?>
<zAppointments reminder="15">
    <appointment>
        <begin>1181251680</begin>
        <uid>040000008200E000</uid>
        <alarmTime>1181572063</alarmTime>
        <state></state>
        <location></location>
        <duration>1800</duration>
        <subject>Bring pizza home</subject>
    </appointment>
    <appointment>
        <begin>1234360800</begin>
        <duration>1800</duration>
        <subject>Check MS Office website for updates</subject>
        <location></location>
        <uid>604f4792-eb89-478b-a14f-dd34d3cc6c21-1234360800</uid>
        <state>dismissed</state>
  </appointment>
</zAppointments>

Now we need to write some code that can parse and modify the XML. Let’s take a look at this little demo that shows a bunch of the neat abilities that objectify provides.

from lxml import etree, objectify
 
#----------------------------------------------------------------------
def parseXML(xmlFile):
    """"""
    with open(xmlFile) as f:
        xml = f.read()
 
    root = objectify.fromstring(xml)
 
    # returns attributes in element node as dict
    attrib = root.attrib
 
    # how to extract element data
    begin = root.appointment.begin
    uid = root.appointment.uid
 
    # loop over elements and print their tags and text
    for appt in root.getchildren():
        for e in appt.getchildren():
            print "%s => %s" % (e.tag, e.text)
        print
 
    # how to change an element's text
    root.appointment.begin = "something else"
    print root.appointment.begin
 
    # how to add a new element
    root.appointment.new_element = "new data"
 
    # print the xml
    obj_xml = etree.tostring(root, pretty_print=True)
    print obj_xml
 
    # remove the py:pytype stuff
    #objectify.deannotate(root)
    etree.cleanup_namespaces(root)
    obj_xml = etree.tostring(root, pretty_print=True)
    print obj_xml
 
    # save your xml
    with open("new.xml", "w") as f:
        f.write(obj_xml)
 
#----------------------------------------------------------------------
if __name__ == "__main__":
    f = r'path\to\sample.xml'
    parseXML(f)

The code is pretty well commented, but we’ll spend a little time going over it anyway. First we pass it our sample XML file and objectify it. If you want to get access to a tag’s attributes, use the attrib property. It will return a dictionary of the attribute’s of the tag. To get to sub-tag elements, you just use dot notation. As you can see, to get to the begin tag’s value, we can just do something like this:

begin = root.appointment.begin

If you need to iterate over the children elements, you can use iterchildren. You may have to use a nested for loop structure to get everything. Changing an element’s value is as simple as just assigning it a new value. And if you need to create a new element, just add a period and the name of the new element (see below):

root.appointment.new_element = "new data"

When we add or change items using objectify, it will add some annotations to the XML, such as xmlns:py=”http://codespeak.net/lxml/objectify/pytype” py:pytype=”str”. You may not want that included, so you’ll have to call the following method to remove that stuff:

etree.cleanup_namespaces(root)

You can also use “objectify.deannotate(root)” to do some deannotation chores, but I wasn’t able to get it to work for this example. To save the new XML, you actually seem to need lxml’s etree module to convert it to a string so you can save it.

At this point, you should be able to parse most XML documents and edit them effectively with lxml’s objectify. I thought it was very intuitive and easy to pick up. Hopefully you will find it useful in your endeavors as well.

Further Reading

Print Friendly

  • http://tshepang.net/ Tshepang Lekhonkhobe

    Maybe rather avoid using the word “builtin” for a stdlib module, since there is also a concept of “built-in” functions and since it’s a bit inaccurate/misleading.

  • http://www.pythonlibrary.org Mike

    I suppose. I fixed it anyway. Thanks!

  • ivan_kleshnin

    It seems like PyQuery is already abandoned. BugTracking list looks horrible. I found 2 bugs on myself today. I would not recommend it to anybody…

  • Pingback: Python: Creating XML with lxml.objectify | Hello Linux()

  • kutpaste123

    I tried this example and I am not entirely sure what I am missing here. The for loop rather than parsing the entire file and printing all the tags prints the first section only. Can you let me know why this is :

    My code :

    ————————————-

    from lxml import etree, objectify

    #http://python.dzone.com/articles/using-lxmlobjectify-parse-xml

    def parseXML(xmlFile):

    with open(xmlFile) as f:

    xml = f.read()

    root = objectify.fromstring(xml)

    attrib = root.attrib

    #How to get the first begin & uid in the XML file

    begin = root.appointment.begin

    uid = root.appointment.uid

    print (“BeginID : %s & UID: %s ” % (begin, uid))

    print

    #how to iterate over all the tags & their text

    for e in root.appointment.iterchildren():

    print (“%s => %s ” % (e.tag, e.text))

    print

    #how to change an elements text

    root.appointment.begin = “something else”

    print (“Changed tag is : %s” % (root.appointment.begin))

    print

    #how to add a new element(tag)

    root.appointment.new_element = “new data”

    objectify.deannotate(root)

    etree.cleanup_namespaces(root)

    obj_xml = etree.tostring(root, pretty_print=True)

    print (obj_xml)

    with open(“example4.xml”, “w”) as f:

    f.write(obj_xml)

    #——————————————–

    if __name__ == “__main__”:

    f = r”example3.xml”

    parseXML(f)

    —————————————————————————-

    Output :

    >>>

    BeginID : 1181251680 & UID: 40000008200.0

    begin => 1181251680

    uid => 040000008200E000

    alarmTime => 1181572063

    state => None

    location => None

    duration => 1800

    subject => Bring pizza home

    Changes tag is : something else

    something else

    040000008200E000

    1181572063

    1800

    Bring pizza home

    new data

    1234360800

    1800

    Check MS Office website for updates

    604f4792-eb89-478b-a14f-dd34d3cc6c21-1234360800

    dismissed

  • http://www.blog.pythonlibrary.org/ Mike Driscoll

    I’m not sure why I did it that way, but basically what that code says is that it is going to loop over just the first appointment tag, as you’ve found out. I actually need a nested loop to make it right. I have updated the code in this article to show how to get the data from both appointments.

  • kutpaste123

    Hi Mike,
    Thanks, I admire your to the point explanation. Could you let me know when you would use lxml & objectify module rather than others out there.
    Thanks.

  • http://www.blog.pythonlibrary.org/ Mike Driscoll

    I use lxml’s objectify most of the time because it makes working with nested / complex XML very easy to use and create. There’s nothing wrong with ElementTree, which is included with Python. I just don’t think it’s ease of use is as good as lxml’s