The Mouse Vs. The Python » SqlAlchemy

Improving MediaLocker: wxPython, SQLAlchemy, and MVC

Mike — Thu, 01 Dec 2011 02:26:54 +0000

This blog ran an article about wxPython, SQLAlchemy, CRUD and MVC earlier this month. The program that we created in that post was dubbed “MediaLocker”, whether or not it was explicitly stated as such. Anyway, since then, I have received a couple comments about improving the program. One came from Michael Bayer, one of the creative minds behind SQLAlchemy itself and the other comments came from Werner Bruhin, a nice guy who haunts the wxPython mailing list, helping new users. So I went about creating an improved version of the code following their advice. Werner then improved it a bit more. So in this article, we will be looking at improving the code, first with my example and then with his. Enough talk though; let’s get to the meat of story!

Making MediaLocker Better

Michael Bayer and Werner Bruhin both thought that I should only connect to the database once as that’s a fairly “expensive” operation. This could be an issue if there were multiple sessions existing at the same time too, but even in my original code, I made sure to close the session so that wouldn’t happen. When I wrote my original version, I thought about separating out the session creation, but ended up going with what I thought was more straightforward. To fix this niggling issue, I changed the code so that I passed the session object around instead of constantly calling my controller’s connectToDatabase function. You can read more about Sessions here. See the code snippet from mediaLocker.py:

class BookPanel(wx.Panel):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self, parent):
        """Constructor"""
        wx.Panel.__init__(self, parent)
 
        if not os.path.exists("devdata.db"):
            controller.setupDatabase()
 
        self.session = controller.connectToDatabase()
        try:
            self.bookResults = controller.getAllRecords(self.session)
        except:
            self.bookResults = []

Note that we have a little conditional right up front that will create the database if it doesn’t already exist. Next I create the session object in the main GUI as a property of the panel sub-class. Then I pass it where ever I need to. One example can be seen above where I pass the session object to the controller’s getAllRecords method.

Another big change was to remove the ObjectListView model from model.py and just use the SQLAlchemy table class instead:

########################################################################
class Book(DeclarativeBase):
    """"""
    __tablename__ = "book"
 
    id = Column(Integer, primary_key=True)
    author_id = Column(Integer, ForeignKey("person.id"))
    title = Column(Unicode)
    isbn = Column(Unicode)
    publisher = Column(Unicode)
    person = relation("Person", backref="books", cascade_backrefs=False)
 
    @property
    def author(self):
        return "%s %s" % (self.person.first_name, self.person.last_name)

This is actually mostly the same as the original class except that it uses SQLAlchemy constructs. I also needed to add a special property to return the author’s full name for display in our widget, so we used Python’s built-in function: property which returns a property attribute. It’s easier to understand if you just look at the code. As you can see, we applied property as a decorator to the author method.

Werner’s Additions

Werner’s additions are mostly adding more explicit imports in the model. The biggest change in the model is as follows:

import sys
if not hasattr(sys, 'frozen'):
    # needed when having multiple versions of SA installed
    import pkg_resources
    pkg_resources.require("sqlalchemy") # get latest version
 
import sqlalchemy as sa
import sqlalchemy.orm as sao
import sqlalchemy.ext.declarative as sad
from sqlalchemy.ext.hybrid import hybrid_property
 
maker = sao.sessionmaker(autoflush=True, autocommit=False)
DBSession = sao.scoped_session(maker)
 
class Base(object):
    """Extend the base class
 
    - Provides a nicer representation when a class instance is printed.
        Found on the SA wiki, not included with TG
    """
    def __repr__(self):
        return "%s(%s)" % (
                 (self.__class__.__name__),
                 ', '.join(["%s=%r" % (key, getattr(self, key))
                            for key in sorted(self.__dict__.keys())
                            if not key.startswith('_')]))
 
DeclarativeBase = sad.declarative_base(cls=Base)
metadata = DeclarativeBase.metadata
 
def init_model(engine):
    """Call me before using any of the tables or classes in the model."""
    DBSession.configure(bind=engine)

The first few lines are for people with SetupTools / easy_install on their machine. If the user has multiple versions of SQLALchemy installed, it will force it to use the latest. Most of the other imports are shortened to make it very obvious where various classes and attributes come from. I am honestly not familiar with the hybrid_property, so here’s what its docstring had to say:

A decorator which allows definition of a Python descriptor with both instance-level and class-level behavior.

Werner also added a little __repr__ method to the Base class to make it return a better representation of the class instance when it’s printed, which is handy for debugging. Finally, he added a function called init_model to initialize the model.

Wrapping Up

Now you should know that Werner and I have decided to make MediaLocker into an example of a wxPython database-enabled application. He’s been doing a bunch of work on it since the simple edits I mentioned above. We’ll be making an official announcement about that soon. In the mean time, I hope that this has helped open your eyes to some fun ways to enhance a project and clean it up a bit. It is my plan to add lots of new features to this program and chronicle those on this blog in addition to all my other articles.

Source Code

wxPython and SqlAlchemy: An Intro to MVC and CRUD

Mike — Fri, 11 Nov 2011 04:15:35 +0000

In this article, we will be creating a program to store our personal books or perhaps just the books you’ve read. It will demonstrate how to combine wxPython and SqlAlchemy, a SQL Object Relational Mapper (ORM). This program will also give you an introduction to the model-view-controller (MVC) and “create, read, update and destroy” (CRUD) methodologies. The aim is to show you how to create an application that can do the following:

Create a database (create)
Add records to that database (sort of create)
Display the records (read and view)
Modify records (update)
Delete records (destroy)

The order of this tutorial will follow MVC, so we will start with the model. The model is the basis for the GUI (the View) anyway, so it’s a good place to start.

Prerequisites

Before we jump into the model section, you may need to download some extras to follow along. Here’s what you’ll need:

Python 2.5 or better, but not 3.x
wxPython >= 2.8.10.1
SqlAlchemy >= 0.5.0
ObjectListView

Once you have all those installed, you’re ready to move on to the next section!

The Model

Our model contains our SqlAlchemy database classes and a model for our ObjectListView widget, which we’ll be using to display our data later. The SqlAlchemy classes are using the Declarative system, which combines the table creation and the class creation. It’s a little easier to understand if you just take a look at the code.

# model.py
 
from sqlalchemy import Table, Column, create_engine
from sqlalchemy import Integer, ForeignKey, String, Unicode
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import backref, relation
 
engine = create_engine("sqlite:///devdata.db", echo=True)
DeclarativeBase = declarative_base(engine)
metadata = DeclarativeBase.metadata
 
########################################################################
class OlvBook(object):
    """
    Book model for ObjectListView
    """
 
    #----------------------------------------------------------------------
    def __init__(self, id, title, author, isbn, publisher, last_name, first_name):
        self.id = id  # unique row id from database
        self.title = title
        self.author = author
        self.isbn = isbn
        self.publisher = publisher
        self.last_name = last_name
        self.first_name = first_name
 
########################################################################
class Person(DeclarativeBase):
    """"""
    __tablename__ = "people"
 
    id = Column(Integer, primary_key=True)
    first_name = Column("first_name", String(50))
    last_name = Column("last_name", String(50))
 
    #----------------------------------------------------------------------
    def __repr__(self):
        """"""
        return "" % (self.first_name, self.last_name)
 
 
########################################################################
class Book(DeclarativeBase):
    """"""
    __tablename__ = "books"
 
    id = Column(Integer, primary_key=True)
    author_id = Column(Integer, ForeignKey("people.id"))
    title = Column("title", Unicode)
    isbn = Column("isbn", Unicode)
    publisher = Column("publisher", Unicode)
    person = relation("Person", backref="books", cascade_backrefs=False)
 
metadata.create_all()

We’re not going to spend a lot of time parsing this code as there are several SqlAlchemy tutorials out there and the SqlAlchemy documentation is really good. Instead, we’ll just give a quick overview. The beginning of the code imports all the SqlAlchemy stuff we need. One of the most important bits is where create the engine:

engine = create_engine("sqlite:///devdata.db", echo=True)

This means that we’ll be creating a SQLite database in the same directory as the model is in. You can pass a path in too, like this: create_engine(“sqlite:///c:/path/to/devdata.db”) and you can name the database whatever you want, including the extension. The echo parameter is turned on for debugging purposes. It will output all the SQL that SqlAlchemy issues so we can use it to figure out our queries or issues that may crop up when we create the database.

The first class we encounter is OlvBook, which defines a book class. We’ll be using that later to help us populate the previously mentioned ObjectListView widget that we will use to display our records. The next two classes are called Person and Book, respectively. The tablename is defined by the magic method: __tablename__. Person is a ForeignKey in Book, which we use for our Author. Originally, I thought I could use Person to add multiple authors or other people related to the book, like the artist. Or maybe for the Songwriter for one of my CDs, but I decided to just keep it simple instead. You can enhance this model a little to make it more polymorphic so it can behave in that way though.

Anyway, if you run this script, it will execute the last line: metadata.create_all() This causes the database file to be created with those two tables defined. Now we’re ready to add data to it. But first, we need to take a look at the View!

The View

The View is our wxPython interface that we will use to display the records as well as filter, add, modify and delete them. The code for our GUI is just over 150 lines long. Let’s take a look!

import addModRecord
import commonDlgs
import controller
import wx
from ObjectListView import ObjectListView, ColumnDefn
 
########################################################################
class BookPanel(wx.Panel):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self, parent):
        """Constructor"""
        wx.Panel.__init__(self, parent)
        try:
            self.bookResults = controller.getAllRecords()
        except:
            self.bookResults = []
 
        mainSizer = wx.BoxSizer(wx.VERTICAL)
        searchSizer = wx.BoxSizer(wx.HORIZONTAL)
        btnSizer = wx.BoxSizer(wx.HORIZONTAL)
        font = wx.Font(10, wx.SWISS, wx.NORMAL, wx.BOLD)
 
        # create the search related widgets
        cat = ["Author", "Title", "ISBN", "Publisher"]
        searchByLbl = wx.StaticText(self, label="Search By:")
        searchByLbl.SetFont(font)
        searchSizer.Add(searchByLbl, 0, wx.ALL, 5)
 
        self.categories = wx.ComboBox(self, value="Author", choices=cat)
        searchSizer.Add(self.categories, 0, wx.ALL, 5)
 
        self.search = wx.SearchCtrl(self, style=wx.TE_PROCESS_ENTER)
        self.search.Bind(wx.EVT_TEXT_ENTER, self.onSearch)
        searchSizer.Add(self.search, 0, wx.ALL, 5)
 
        self.bookResultsOlv = ObjectListView(self, style=wx.LC_REPORT
                                                        |wx.SUNKEN_BORDER)
        self.bookResultsOlv.SetEmptyListMsg("No Records Found")
        self.setBooks()
 
        # create the button row
        addRecordBtn = wx.Button(self, label="Add")
        addRecordBtn.Bind(wx.EVT_BUTTON, self.onAddRecord)
        btnSizer.Add(addRecordBtn, 0, wx.ALL, 5)
 
        editRecordBtn = wx.Button(self, label="Edit")
        editRecordBtn.Bind(wx.EVT_BUTTON, self.onEditRecord)
        btnSizer.Add(editRecordBtn, 0, wx.ALL, 5)
 
        deleteRecordBtn = wx.Button(self, label="Delete")
        deleteRecordBtn.Bind(wx.EVT_BUTTON, self.onDelete)
        btnSizer.Add(deleteRecordBtn, 0, wx.ALL, 5)
 
        showAllBtn = wx.Button(self, label="Show All")
        showAllBtn.Bind(wx.EVT_BUTTON, self.onShowAllRecord)
        btnSizer.Add(showAllBtn, 0, wx.ALL, 5)
 
        mainSizer.Add(searchSizer)
        mainSizer.Add(self.bookResultsOlv, 1, wx.ALL|wx.EXPAND, 5)
        mainSizer.Add(btnSizer, 0, wx.CENTER)
        self.SetSizer(mainSizer)
 
    #----------------------------------------------------------------------
    def onAddRecord(self, event):
        """
        Add a record to the database
        """
        dlg = addModRecord.AddModRecDialog()
        dlg.ShowModal()
        dlg.Destroy()
        self.showAllRecords()
 
    #----------------------------------------------------------------------
    def onEditRecord(self, event):
        """
        Edit a record
        """
        selectedRow = self.bookResultsOlv.GetSelectedObject()
        if selectedRow == None:
            commonDlgs.showMessageDlg("No row selected!", "Error")
            return
        dlg = addModRecord.AddModRecDialog(selectedRow, title="Modify",
                                           addRecord=False)
        dlg.ShowModal()
        dlg.Destroy()
        self.showAllRecords()
 
    #----------------------------------------------------------------------
    def onDelete(self, event):
        """
        Delete a record
        """
        selectedRow = self.bookResultsOlv.GetSelectedObject()
        if selectedRow == None:
            commonDlgs.showMessageDlg("No row selected!", "Error")
            return
        controller.deleteRecord(selectedRow.id)
        self.showAllRecords()
 
    #----------------------------------------------------------------------
    def onSearch(self, event):
        """
        Searches database based on the user's filter choice and keyword
        """
        filterChoice = self.categories.GetValue()
        keyword = self.search.GetValue()
        print "%s %s" % (filterChoice, keyword)
        self.bookResults = controller.searchRecords(filterChoice, keyword)
        self.setBooks()
 
    #----------------------------------------------------------------------
    def onShowAllRecord(self, event):
        """
        Updates the record list to show all of them
        """
        self.showAllRecords()
 
    #----------------------------------------------------------------------
    def setBooks(self):
        self.bookResultsOlv.SetColumns([
            ColumnDefn("Title", "left", 350, "title"),
            ColumnDefn("Author", "left", 150, "author"),
            ColumnDefn("ISBN", "right", 150, "isbn"),
            ColumnDefn("Publisher", "left", 150, "publisher")
        ])
        self.bookResultsOlv.SetObjects(self.bookResults)
 
    #----------------------------------------------------------------------
    def showAllRecords(self):
        """
        Show all records in the object list view control
        """
        self.bookResults = controller.getAllRecords()
        self.setBooks()
 
########################################################################
class BookFrame(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="MvP Media Organizer",
                          size=(800, 600))
        panel = BookPanel(self)
 
        self.Show()
 
#----------------------------------------------------------------------
if __name__ == "__main__":
    app = wx.App(False)
    frame = BookFrame()
    app.MainLoop()

You’ll notice that we import a few custom items, like addModRecord, commonDlgs and controller. The addModRecord is a dialog that can be used both to add a record and modify one. The commonDlgs module is used just to simplify creating message dialogs. If we add some new features to this program, we would add other small dialog code in that module. The controller module is where all the SqlAlchemy code is executed. Most programs have some configuration options, so this would be a good place to store that code.

Anyway, let’s go over the highlights of the GUI. The search bar is set to search when the user presses enter. That’s what the style flag wx.TE_PROCESS_ENTER means. That and the ObjectListView’s SetEmptyListMsg method are probably the only really unfamiliar things here. That method just shows a message to the user when there’s no data in the control.

The onAddRecord and onEditRecord both call our addModRecord module to display the Add / Modify dialog. It should look something like this:

The onEditRecord method sets a couple parameters that affect the look of the widget and passes in some information to populate the fields. The onDelete method just takes the selected item and calls the controller to delete it, then it updates the display.

Our onSearch method does what you’d expect. It searches for records in our database based on what filter the user chooses and their keyword. They can choose “Author”, “Title”, “ISBN” or “Publisher” as their filter.

The onShowAllRecord method just shows all the records on the display. A future enhancement would be to limit the number of currently showed records so we don’t cause major slow down of the application. The setBooks method just updates the displayed records.

Now we’ve reached the last piece of our puzzle: the Controller.

The Controller and CRUD

The controller is the glue which holds the model and the view together. At least, that’s what it seems like to me. It uses the model for queries and for adding, updating and deleting records. It also reports the updates to the database to the view (our GUI) so it can update appropriately. Let’s take a look at our controller code:

# controller.py
from model import Book, Person, OlvBook
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
 
#----------------------------------------------------------------------
def addRecord(data):
    """
    Data should be a tuple of two dictionaries in the following format:
 
    ("author":{"first_name":"John", "last_name":"Doe"},
     "book":{"title":"Some book", "isbn":"1234567890",
             "publisher":"Packt"}
    )
    """
    book = Book()
    book.title = data["book"]["title"]
    book.isbn = data["book"]["isbn"]
    book.publisher = data["book"]["publisher"]
    author = Person()
    author.first_name = data["author"]["first_name"]
    author.last_name = data["author"]["last_name"]
    book.person = author
 
    # connect to session and commit data to database
    session = connectToDatabase()
    session.add(book)
    session.commit()
    session.close()
 
#----------------------------------------------------------------------
def connectToDatabase():
    """
    Connect to our SQLite database and return a Session object
    """
    engine = create_engine("sqlite:///devdata.db", echo=True)
    Session = sessionmaker(bind=engine)
    session = Session()
    return session
 
#----------------------------------------------------------------------
def convertResults(results):
    """
    Convert results to OlvBook objects
    """
    print
    books = []
    for record in results:
        author = "%s %s" % (record.person.first_name,
                            record.person.last_name)
        book = OlvBook(record.id, record.title, author,
                       record.isbn, record.publisher,
                       record.person.last_name,
                       record.person.first_name
                       )
        books.append(book)
    return books
 
#----------------------------------------------------------------------
def deleteRecord(idNum):
    """
    Delete a record from the database
    """
    session = connectToDatabase()
    record = session.query(Book).filter_by(id=idNum).one()
    session.delete(record)
    session.commit()
    session.close()
 
#----------------------------------------------------------------------
def editRecord(idNum, row):
    """
    Edit a record
    """
    session = connectToDatabase()
    record = session.query(Book).filter_by(id=idNum).one()
    print
    record.title = row["title"]
    record.person.first_name = row["first_name"]
    record.person.last_name = row["last_name"]
    record.isbn = row["isbn"]
    record.publisher = row["publisher"]
    session.add(record)
    session.commit()
    session.close()
 
#----------------------------------------------------------------------
def getAllRecords():
    """
    Get all records and return them
    """
    session = connectToDatabase()
    result = session.query(Book).all()
    books = convertResults(result)
    session.close()
    return books
 
#----------------------------------------------------------------------
def searchRecords(filterChoice, keyword):
    """
    Searches the database based on the filter chosen and the keyword
    given by the user
    """
    session = connectToDatabase()
    if filterChoice == "Author":
        qry = session.query(Person)
        result = qry.filter(Person.first_name.contains('%s' % keyword)).all()
        records = []
        for record in result:
            for book in record.books:
                records.append(book)
        result = records
        print result
    elif filterChoice == "Title":
        qry = session.query(Book)
        result = qry.filter(Book.title.contains('%s' % keyword)).all()
    elif filterChoice == "ISBN":
        qry = session.query(Book)
        result = qry.filter(Book.isbn.contains('%s' % keyword)).all()
    else:
        qry = session.query(Book)
        result = qry.filter(Book.publisher.contains('%s' % keyword)).all()
    books = convertResults(result)
    print
    return books

Our controller has 7 functions defined:

addRecord
connectToDatabase
convertResults
deleteRecord
editRecord
getAllRecords
searchRecords

These are pretty self explanatory. However, we’ll spend a little time explaining what convertResults does and how searchRecords works.

The convertResults method is for converting the SqlAlchemy book results that we get from querying the database, into the ObjectListView Book model objects. This is necessary so we can display them when there are updates to the database or when displaying the results of a query. As you can see, we just iterate over the results and massage the data appropriately.

In the searchRecords method, we pass in our filter and keyword and do queries on the database using our session object via a conditional “if” statement. Depending on the type of filter, we query differently. For the most part, the query is the same though, so this could use some refactoring. I’ll leave that for the reader to figure out as an assignment. Once we have the results converted, we then return them to the calling function, which in this case happens to be our GUI’s search event handler.

The CRUD part is covered in the other methods where we create, read, update, and delete records.

Wrapping Up

Now you should know the basics behind creating a wxPython program using the MVC framework. This isn’t a perfect example and I don’t claim that it is, but it should give you an idea of how to start making your own. It also shows how to interact with a SQLite database using SqlAlchemy. I hope you’ve found this helpful and look forward to hearing your comments.

Source Code

mediaLocker.zip
mediaLocker.tar
You can also check out the code from Bitbucket

SqlAlchemy ProgrammingError 42000 and MS SQL

Mike — Sat, 15 Jan 2011 22:19:40 +0000

I have been working on a software inventory script lately using SqlAlchemy on Windows XP to connect to a newly created table in a Microsoft SQL Server 2005 database. I created the table using Aqua Data Studio, logging in as the SQL Administrator (sa) and thought all was well until I tried to commit some data to the table. Here’s the screwy error that I received:

pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Cannot find the object "dbo.software" because it does not exist or you do not have permissions. (1088) (SQLExecDirectW)')

Now, what does that mean? I know the database exists because I just created it with Aqua Data Studio (a database management suite) and I gave myself the following permissions: Select, Insert, Update and Delete. If you Google this error, you’ll end up finding four references to one thread. They don’t really have a solution in the thread, although they mention that adding permission to alter the schema might work. Note that this problem only happens when you use the declarative syntax. Anyway, at this point, you probably want to see my code, so take a look:

from sqlalchemy import create_engine, Column, DateTime, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import datetime
 
uri = "mssql://username:pw@mssqlServerPath/Inventory"
engine = create_engine(uri)
engine.echo = True
Base = declarative_base(engine)
 
########################################################################
class Software(Base):
    """
    SqlAlchemy table representation of "software" login
    """
    __tablename__ = "software"
    __table_args__ = {"schema":"dbo"}
 
    id = Column(Integer, primary_key=True, autoincrement=True)
    name = Column(String(100))
    date_added = Column(DateTime)
    date_checked = Column(DateTime)
    machine_name = Column(String(25))
    version = Column(String(25))
 
    #----------------------------------------------------------------------
    def __init__(self, name, date_added, date_checked, machine_name, version):
        """"""
        self.name = name
        self.date_added = date_added
        self.date_checked = date_checked
        self.machine_name = machine_name
        self.version = version
 
Session = sessionmaker(bind=engine)
session = Session()
now = datetime.datetime.now()
new_record = Software("Adobe Acrobat", now, now, "MCIS0467", "9.0")
session.add(new_record)
session.commit()

From what my boss could figure out, the declarative syntax using an identity-based query system and if the identity isn’t set, then SqlAlchemy cannot find the database. If you use SqlAlchemy to create the table, then you won’t have this problem. Anyway, it turns out that you have to use Microsoft SQL Server Management Studio to rectify this particular situation. Load it up and navigate to the correct database and table, then open the columns tree. Right-click the primary key field and choose “Modify”. There’s a “Column Properties” tab at the bottom of the screen. Go there and scroll down to Identity Specification and expand it. Finally, make sure that the “(Is Identity)” field is set to “Yes”. Save it and you’re done!

This is a pretty weird and rare issue so you probably won’t have it, but I thought it was an interesting problem and I wanted to document the solution. Note: I was using Python 2.5 with SqlAlchemy 0.6.6 on Windows XP.

SqlAlchemy: A Simple Tutorial for Elixir

Mike — Fri, 26 Nov 2010 01:55:10 +0000

Elixir is a lightweight declarative layer on top of SqlAlchemy that’s been around since 2006, well before SqlAlchemy released their own built-in Declarative syntax. Elixir was created by a collaboration between Jonathan LaCour, Daniel Haus and Gaëtan de Menten who had a passion for making SqlAlchemy even easier to use. In this tutorial, we’ll look at how to to create our own database with Elixir and how to communicate with a pre-existing database.

Note: This is my first time using Elixir so I wouldn’t be surprised if there are better ways to do some of the stuff mentioned herein.

Creating a Database with Elixir

For this section, we’re going to look at creating a super simple bookmark database where each record is made up of a title, a url and a description. It will also have an id column that will be our primary key. Elixir does this for us automatically whenever we create a database without a primary key specified and it will auto-increment the id as well. Let’s get this party started with some code!

from elixir import metadata, Entity, Field
from elixir import Unicode, UnicodeText
 
metadata.bind = "sqlite:///bookmarks.sqlite"
metadata.bind.echo = True
 
########################################################################
class Bookmarks(Entity):
    """
    Custom made bookmark example
    """
    title = Field(Unicode)
    url = Field(Unicode)
    description = Field(UnicodeText)
 
    #----------------------------------------------------------------------
    def __repr__(self):
        """"""
        return " % (self.title, self.url,
                                             self.description)
 
if __name__ == "__main__":
    from elixir import create_all, setup_all, session
    setup_all()
    create_all()
 
    # Adding records
    Bookmarks(title="Google", url="http://www.google.com",
              description="Search engine")
    Bookmarks(title="Digital Photography School",
              url="http://www.digital-photography-school.com",
              description="An Australian Photography blog")
    Bookmarks(title="Mouse Vs. the Python",
              url="http://www.blog.pythonlibrary.org/",
              description="Mike Driscoll's Python blog")
    session.commit()
 
    # --------------------------------------------------
    # Simple Queries
 
    # get all the records
    records = Bookmarks.query.all()
    for record in records:
        print "-" * 20
        print record.title
        print record.url
        print record.description
 
    # find a specific record
    qry = Bookmarks.query.filter_by(title=u'Google')
    record = qry.first()
    print "%s (%s)" % (record.title, record.url)
 
    # delete the record
    record.delete()
    session.commit()

Let’s take a moment to study this code and see if we can figure it out. In this example, we import just the classes we want from the elixir package.

After the imports are two lines dealing with metadata. According to Gaëtan de Menten, the metadata is “a container of python objects representing database tables.” Anyway, what happens in those lines is the creation of the sqlite database file (assuming it doesn’t already exist) and the setting of the echo to True. That means that all the SQL that Elixir emits will be sent to stdout so you can read it too.

Next we subclass the Entity object in our Bookmarks class. Here we create our three fields (or columns). The “__repr__” basically makes the printing of the Bookmarks objects more friendly for humans such that it prints out the title, url and description of the Bookmark rather than just something like “Bookmark object “.

Lastly, in the “if” statement at the end, we setup the mappings and such and create the database. Then we add three rows and commit them to said database. Next, we do a couple of simple queries against the database and at the very end, we delete a record. Hopefully you can follow all that without explicit description.

For our next trick, we will look at how to read an already existing database with Elixir.

Reading a Pre-Existing Database

In this example, we will use a copy of the places.sqlite database that is generated by Mozilla Firefox. It holds bookmarks and (I think) your browsing history. The code in this example will assume that the database file has been copied into the same location as the Python code file. Let’s take a look at that code now:

from elixir import metadata, using_options, Entity, Field
from elixir import Integer, Unicode, UnicodeText
 
metadata.bind = "sqlite:///places.sqlite"
metadata.bind.echo = True
 
########################################################################
class Bookmarks(Entity):
    """
    Firefox Bookmarks class
    """
    using_options(tablename="moz_bookmarks")
 
    _id = Field(Integer, colname="id", primary_key=True)
    _type = Field(Integer, colname="type")
    fk = Field(Integer)
    parent = Field(Integer)
    position = Field(Integer)
    title = Field(Unicode)
    keyword_id = Field(Integer)
    folder_type = Field(UnicodeText)
    dateAdded = Field(Integer)
    lastModified = Field(Integer)
 
    #----------------------------------------------------------------------
    def __repr__(self):
        """Constructor"""
        return ' % (self._id, self.title)
 
 
if __name__ == "__main__":
    from elixir import setup_all
    setup_all()
    result = Bookmarks.query.all()
    x = 0
    print "-" * 10
    for item in result:
        if x > 10:
            break
        print item.title
        x += 1

The first few lines are basically the same as the ones in the previous example. The only real difference is that we specify a database that exists this time and we imported a few more classes. In the Bookmark class, we need to mirror the columns in the “moz_bookmarks” database. There’s a difference right there. We use the “using_options” class from Elixir to specify which table we want. Then we create a matching field name. The first two fields would actually shadow Python builtins, so we give them an underscore to differentiate them and set the column name by passing it in the colname keyword argument of the Field class.

When testing this code, I noticed that some of the titles were nulls, so the __repr__ may not work as expected. Anyway, in the “if” statement at the end, we do a query against the table and print out the first 10 records we find.

As a side note, if you don’t want to declare all the field names, you don’t have to. Elixir supports the same autoload functionality that SqlAlchemy does. So instead of declaring the field names, you would do something like this: using_options(tablename=”moz_bookmarks”, autoload=True)

Wrapping Up

As you can see, using Elixir is pretty simple. SqlAlchemy has since added its own built-in declarative method, but for some, Elixir proves to be a potent help. There are many more query examples and relational examples on their official website (see link below). I encourage you to check it out and see if you find it helpful in learning SqlAlchemy.

SqlAlchemy and Microsoft Access

Mike — Sun, 10 Oct 2010 13:04:23 +0000

Update (10/12/2010) – One of my alert readers told me that SqlAlchemy 0.6.x currently does NOT support the Access dialect. Read here for more info.

A year or two ago, I was asked to transfer some data from some old Microsoft Access files to our Microsoft SQL Server. Since I enjoy using SqlAlchemy, I decided to see if it supported Access. The documentation at the time was pretty unhelpful in this regard, but it did seem to be possible and I found one thread about it on SqlAlchemy’s Google group.

The code to connect to Microsoft Access is pretty simple. It goes something like this:

from sqlalchemy import create_engine
engine = create_engine(r'access:///C:/some/path/database.MDB')

See how easy that was? You just tell SqlAlchemy what kind of database to connect to, add three forward slashes and then the path to the file. Once that’s done, you can do pretty much anything with the Access file that you can do with a normal database:

########################################################################
class TableName(Base):
    """
    MS Access database
    """
    __tablename__ = "ROW"
    __table_args__ = ({"autoload":True})  # load the database
    FILENUM = Column("FILE #", Integer, key="FILENUM")

In the code above, I use SqlAlchemy’s declarative syntax to autoload the database’s structure. I can’t recall is this database had its primary key set, but I’m guessing it didn’t since I had to add that last line.

Anyway, once you have your connection, you can just run queries as you normally would. In my case, I ended up creating a model file to hold all the table definitions for the Access file and the SQL Server database, then I did a SELECT * on the Access file, looped over the result and inserted each row into the SQL Server one. The only thing you have to watch out for is that Access is a lot more forgiving of NULLS than SQL Server is, so I had to write some special handling around that atrocity.

Well, that’s really all there is to it. You can check out my other SqlAlchemy tutorials for more general information about using SqlAlchemy to interact with your database.

SqlAlchemy: Connecting to pre-existing databases

Mike — Fri, 10 Sep 2010 12:59:59 +0000

Accessing databases with Python is a simple process. Python even provides a sqlite database library that’s built into the main distribution (since 2.5). My favorite way to access databases with Python is to use the 3rd party package, SqlAlchemy. SqlAlchemy is an object-relational mapper (ORM), which means that it takes SQL constructs and makes them more like the target language. In this case, you end up using Python syntax to execute SQL rather than straight SQL and you can use the same code to access multiple database backends (if you’re careful).

In this article, we’re going to look at how to use SqlAlchemy to connect to pre-existing databases. If my experience is any indication, you’ll probably be spending more time working with databases that you didn’t create than with ones that you did. This article will show you how to connect to them.

SqlAlchemy’s autoload

SqlAlchemy has two ways to define a databases columns. The first way is the long way, in which you define each and every field along with its type. The easy (or short) way is to use SqlAlchemy’s autoload functionality, which will introspect the table and pull out the field names in a rather magical way. We’ll start off with the autoload method and then show the long way in the next section.

Before we begin though, it needs to be noted that there are two methods for configuring SqlAlchemy: a long-hand way and a declarative (or “shorthand” way. We will go over both ways. Let’s start out with the long version and then do it “declaratively”.

from sqlalchemy import create_engine, MetaData, Table
from sqlalchemy.orm import mapper, sessionmaker
 
class Bookmarks(object):
    pass
 
#----------------------------------------------------------------------
def loadSession():
    """"""
    dbPath = 'places.sqlite'
    engine = create_engine('sqlite:///%s' % dbPath, echo=True)
 
    metadata = MetaData(engine)
    moz_bookmarks = Table('moz_bookmarks', metadata, autoload=True)
    mapper(Bookmarks, moz_bookmarks)
 
    Session = sessionmaker(bind=engine)
    session = Session()
    return session
 
if __name__ == "__main__":
    session = loadSession()
    res = session.query(Bookmarks).all()
    res[1].title

In this snippet, we import a few handy classes and utilities from sqlalchemy that allow us to define such things as an engine (a type of connection/interface to the database), metadata (a table catalog) and a session (a “handle” to the database that allows us to query it). Note that we have a “places.sqlite” file. This database comes from Mozilla Firefox. If you have it installed, go looking for it as it makes an excellent testbed for this sort of thing. On my Windows XP machine, it’s in the following location: “C:\Documents and Settings\Mike\Application Data\Mozilla\Firefox\Profiles\f7csnzvk.default”. Just copy that file into the location that you’ll be using for the scripts in this article. If you try to use it in place and you have Firefox open, you may have issues as your code may interrupt Firefox or vice-versa.

In the create_engine call, we set echo to True. This causes SqlAlchemy to send all the SQL it generates to stdout, which is quite handy for debugging purposes. I would recommend setting it to False when you put your code in production. For our purposes, the most interesting line is as follows:

moz_bookmarks = Table('moz_bookmarks', metadata, autoload=True)

This tells SqlAlchemy to attempt to load the “moz_bookmarks” table automatically. If it’s a proper database with a primary key, this will work great. If the table doesn’t have a primary key, then you can do this to hack it:

from sqlalchemy import Column, Integer
moz_bookmarks = Table('moz_bookmarks', metadata,
                      Column("id", Integer, primary_key=True),
                      autoload=True)

This adds an extra column called “id”. It’s basically monkey-patching the database. I’ve used this method successfully when I have to use poorly designed databases from vendors. SqlAlchemy will automatically increment it too, if the database supports it. The mapper will map the table object to the Bookmarks class. Then we create a session that’s bound to our engine so we can make queries. The res = session.query(Bookmarks).all() basically means SELECT * FROM moz_bookmarks and will return the result as a list of Bookmark objects.

Sometimes the bad database will have a field that should obviously be the primary key. If so, you can just use that name instead of “id”. Other times, you can create a unique key by setting the primary key to two columns (i.e. fields). You do that by creating two columns and setting “primary_key” to True on both of them. This is a whole lot better than having to define an entire table if the table has several dozen fields.

Let’s move on to the declarative autoload method:

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
 
engine = create_engine('sqlite:///places.sqlite', echo=True)
Base = declarative_base(engine)
########################################################################
class Bookmarks(Base):
    """"""
    __tablename__ = 'moz_bookmarks'
    __table_args__ = {'autoload':True}
 
#----------------------------------------------------------------------
def loadSession():
    """"""
    metadata = Base.metadata
    Session = sessionmaker(bind=engine)
    session = Session()
    return session
 
if __name__ == "__main__":
    session = loadSession()
    res = session.query(Bookmarks).all()
    print res[1].title

The declarative method looks a little different, huh? Let’s try to unpack the new stuff. First off, we have a new import: from sqlalchemy.ext.declarative import declarative_base. We use “declarative_base” to create a class using our engine that we then create the Bookmarks subclass from. To specify the table name, we use the magic method, __tablename__ and to tell it to autoload, we create the __table_args__ dict. Then in the loadSession function, we get the metadata object like this: metadata = Base.metadata. The rest of the code is the same. Declarative syntax usually simplifies the code because it puts it all in one class rather than creating a class and a Table object separately. If you like the declarative style, you might also want to look at Elixir, a SqlAlchemy extension that did declarative before they added this style to the main package.

Defining Your Databases Explicitly

There are times when you can’t use autoload or you just want to have complete control over the table definition. SqlAlchemy allows you to do this almost as easily. We’ll look at the longhand version first and then look at the declarative style.

from sqlalchemy import create_engine, Column, MetaData, Table
from sqlalchemy import Integer, String, Text
from sqlalchemy.orm import mapper, sessionmaker
 
class Bookmarks(object):
    pass
 
#----------------------------------------------------------------------
def loadSession():
    """"""
    dbPath = 'places.sqlite'
 
    engine = create_engine('sqlite:///%s' % dbPath, echo=True)
 
    metadata = MetaData(engine)
    moz_bookmarks = Table('moz_bookmarks', metadata,
                          Column('id', Integer, primary_key=True),
                          Column('type', Integer),
                          Column('fk', Integer),
                          Column('parent', Integer),
                          Column('position', Integer),
                          Column('title', String),
                          Column('keyword_id', Integer),
                          Column('folder_type', Text),
                          Column('dateAdded', Integer),
                          Column('lastModified', Integer)
                          )
 
    mapper(Bookmarks, moz_bookmarks)
 
    Session = sessionmaker(bind=engine)
    session = Session()
 
if __name__ == "__main__":
    session = loadSession()
    res = session.query(Bookmarks).all()
    print res[1].title

This code really isn’t all that different from the code we’ve seen. The part we care about most is the Table definition. Here we use the keyword, Column, to define each column. The Column class accepts a name of the column, the type, whether to set it as the primary key and whether or not the column is nullable. It may accept a few other arguments, but those are the ones you’ll see the most. This example doesn’t show the nullable bit because I couldn’t figure out if the Firefox table had those constraints.

Anyway, once you’ve got the columns defined, the rest of the code is the same as before. Let’s move on to the declarative style then:

from sqlalchemy import Column, Integer, String, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
 
engine = create_engine('sqlite:///places.sqlite', echo=True)
Base = declarative_base(engine)
########################################################################
class Places(Base):
    """"""
    __tablename__ = 'moz_places'
 
    id = Column(Integer, primary_key=True)
    url = Column(String)
    title = Column(String)
    rev_host = Column(String)
    visit_count = Column(Integer)
    hidden = Column(Integer)
    typed = Column(Integer)
    favicon_id = Column(Integer)
    frecency = Column(Integer)
    last_visit_date = Column(Integer)
 
    #----------------------------------------------------------------------
    def __init__(self, id, url, title, rev_host, visit_count,
                 hidden, typed, favicon_id, frecency, last_visit_date):
        """"""
        self.id = id
        self.url = url
        self.title = title
        self.rev_host = rev_host
        self.visit_count = visit_count
        self.hidden = hidden
        self.typed = typed
        self.favicon_id = favicon_id
        self.frecency = frecency
        self.last_visit_date = last_visit_date
 
    #----------------------------------------------------------------------
    def __repr__(self):
        """"""
        return "" % (self.id, self.title,
                                                 self.url)
 
#----------------------------------------------------------------------
def loadSession():
    """"""
    metadata = Base.metadata
    Session = sessionmaker(bind=engine)
    session = Session()
    return session
 
if __name__ == "__main__":
    session = loadSession()
    res = session.query(Places).all()
    print res[1].title

We switched things up a bit with this example by setting it up for a different table: “moz_places”. You’ll notice that you set up the columns here by creating class variables. If you want to access the instance’s variables, you need to redefine them in the __init__. If you don’t do this, you’ll end up with some pretty confusing issues. The __repr__ is there as a “pretty print” method. When you print one of the “Places” objects, you’ll get whatever __repr__ returns. Other than that, the code is pretty similar to the other pieces that we’ve seen.

Wrapping Up

As you can see, using SqlAlchemy to connect to databases is a breeze. Most of the time, you will know ahead of time what the database’s design is, either from documentation, because you built it yourself, or because you have some utility that can tell you. Having that information is helpful when it comes to defining the tables in SqlAlchemy, but now you know what to try even if you don’t know the table’s configuration beforehand! Isn’t learning Python tricks fun? See you next time!

Downloads

Another Step-by-Step SqlAlchemy Tutorial (part 2 of 2)

Mike — Thu, 04 Feb 2010 01:32:06 +0000

In the first part of this series, we went over what some might call the “SQL Expression” method of using SqlAlchemy to interact with your database. The theory behind this is that we should learn the less abstract way of doing things before we get to the higher level (and more abstract) methods. This is true in many math classes, like Calculus where you learn the long way to find the standard deviation of some calulation before you learn about the shortcut.

For the second half, we will be doing what some might say is the easy way to use SqlAlchemy. It is known as the “Object Relational” method and the official documentation actually starts with it. This methodology takes a little longer to set up initially, but in many ways, it is also much easier to follow.

Getting Used to Data Mapping

Robin Munn’s old school SqlAlchemy tutorial called this section “data mapping” because we’ll be mapping the data in a database to Python classes. Let’s get started!

from sqlalchemy import create_engine
from sqlalchemy import Column, MetaData, Table
from sqlalchemy import Integer, String, ForeignKey
from sqlalchemy.orm import mapper, sessionmaker
 
####################################################
class User(object):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self, name, fullname, password):
        """Constructor"""
        self.name = name
        self.fullname = fullname
        self.password = password
 
    def __repr__(self):
        return "" % (self.name, self.fullname, self.password)
 
# create a connection to a sqlite database
# turn echo on to see the auto-generated SQL
engine = create_engine("sqlite:///tutorial.db", echo=True)
 
# this is used to keep track of tables and their attributes
metadata = MetaData()
users_table = Table('users', metadata,
                    Column('user_id', Integer, primary_key=True),
                    Column('name', String),
                    Column('fullname', String),
                    Column('password', String)
                    )
email_table = Table('email', metadata,
                    Column('email_id', Integer, primary_key=True),
                    Column('email_address', String),
                    Column('user_id', Integer, ForeignKey('users.user_id'))
                    )
 
# create the table and tell it to create it in the 
# database engine that is passed
metadata.create_all(engine)
 
# create a mapping between the users_table and the User class
mapper(User, users_table)

The first difference to take note of compared to our previous examples is the User class. We have changed our original example (see part one) a little to match what’s on the official documentation, namely the parameters are now name, full name and password. The rest should look the same until we get to the mapper statement. This handy method allows SqlAlchemy to map the User class to the users_table. This might not seem like a big deal, but this method makes adding users to the database much more straightforward.

However, before we get to that, we need to discuss the Declarative configurational style. While the style above gives us granular control over the table, mapper and class, for the most part we don’t need it to be that complicated. That is where the Declarative style comes in. It makes configuring everything even easier. The first declarative style that I knew of was an add-on for SqlAlchemy called Elixir. This built-in Declarative style isn’t as full-featured as Elixir, but it’s handier because you don’t have the extra dependency. Let’s see how Declarative is different:

from sqlalchemy import Column, Integer, String, ForeignKey
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import backref, mapper, relation, sessionmaker
 
Base = declarative_base()
 
########################################################################
class User(Base):
    """"""
    __tablename__ = "users"
 
    id = Column(Integer, primary_key=True)
    name = Column(String)
    fullname = Column(String)
    password = Column(String)
 
    #----------------------------------------------------------------------
    def __init__(self, name, fullname, password):
        """Constructor"""
        self.name = name
        self.fullname = fullname
        self.password = password
 
    def __repr__(self):
        return "" % (self.name, self.fullname, self.password)
 
########################################################################
class Address(Base):
    """
    Address Class
 
    Create some class properties before initilization
    """
    __tablename__ = "addresses"
    id = Column(Integer, primary_key=True)
    email_address = Column(String, nullable=False)
    user_id = Column(Integer, ForeignKey('users.id'))
 
    # creates a bidirectional relationship
    # from Address to User it's Many-to-One
    # from User to Address it's One-to-Many
    user = relation(User, backref=backref('addresses', order_by=id))
 
    #----------------------------------------------------------------------
    def __init__(self, email_address):
        """Constructor"""
        self.email_address = email_address
 
    def __repr__(self):
        return "" % self.email_address
 
 
# create a connection to a sqlite database
# turn echo on to see the auto-generated SQL
engine = create_engine("sqlite:///tutorial.db", echo=True)
 
# get a handle on the table object
users_table = User.__table__
# get a handle on the metadata
metadata = Base.metadata
metadata.create_all(engine)

As you can see, almost everything is now created in classes. We create class attributes (which are like global variables for the class) that identify the columns of the table. Then we create the same __init__ that we had in the original class example above. Also, we sub-class declarative_base rather than the basic object. If we need a table object, we have to call the following magic method User.__table__; and to get the metadata, we need to call Base.metadata. That covers the differences that we care about. Now we can look at how to add data to our database.

Class is now in Session

The beauty of using the Object Relational method for interacting with our database can be shown in a few quick code snippets. Let’s see how we can create a row:

mike_user = User("mike", "Mike Driscoll", "password")
print "User name: %s, fullname: %s, password: %s" % (mike_user.name,
                                                     mike_user.fullname,
                                                     mike_user.password)

As you can see, we can create users with the User class. We can use dot-notation to access the attributes, just as we would in any other Python class. We can even use them to update the rows. For example, if we need to change the user object above, we would do the following:

# this is how you would change the name field
mike_user.fullname = "Mike Dryskull"

Note that none of this will add the row to the database automatically like those insert methods that we looked at in our first article. Instead, we need a Session object to do that. Let’s run through some of the basics of using a Session:

from sqlalchemy.orm import sessionmaker
 
Session = sessionmaker(bind=engine)
session = Session()
 
mike_user = User("mike", "Mike Driscoll", "password")
session.add(mike_user)

We’ll pause here to explain what’s going on. First we need to import the sessionmaker from sqlalchemy.orm and bind it to an engine (technically, you can create the session without binding, but to do anything useful, you’ll need to bind it eventually). Next we create a session instance. Then we instantiate a user object and add it to the session. At this point, no SQL code has been run and the transaction is just pending. To persist this row, we would need to call session.commit() or run a query.

If you need to add multiple users, you do this:

session.add_all([
     User('Mary', 'Mary Wonka', 'foobar'),
     User('Sue', 'Sue Lawhead', 'xxg527'),
     User('Fay', 'Fay Ray', 'blah')])

If you happen to change one of the user’s attributes after committing it to the database, you can use session.dirty to check which one was modified. If you just need to know what’s rows are pending, call session.new. Finally, we can use session.rollback() to rollback a transaction.

Now let’s take a look at some sample queries:

# do a Select all
all_users = session.query(User).all()
 
# Select just one user by the name of "mike"
our_user = session.query(User).filter_by(name='mike').first()
print our_user
 
# select users that match "Mary" or "Fay"
users = session.query(User).filter(User.name.in_(['Mary', 'Fay'])).all()
print users
 
# select all and print out all the results sorted by id
for instance in session.query(User).order_by(User.id):
    print instance.name, instance.fullname

We don’t need to go over each of these as they all have explanations in the comments. Instead, we’ll move on to the topic of joins.

Join in the Fun

There are joins using the SQL Expression syntax that I won’t be covering here. Rather, we will use the Object Relational methodology. If you look back at either of the beginning examples for creating the tables, you will notice that we have already set up the join with the ForeignKey object. The Declarative format looked like this:

user_id = Column(Integer, ForeignKey('users.id'))
 
# creates a bidirectional relationship
# from Address to User it's Many-to-One
# from User to Address it's One-to-Many
user = relation(User, backref=backref('addresses', order_by=id))

Let’s see how this works by creating a new user:

prof = User("Prof", "Prof. Xavier", "fudge")
prof.addresses

Because of the ForeignKey and the backref command, the User object has an addresses attribute. If you run that code, you will see that it is empty. Let’s add some addresses! (Note: Be sure to add the prof user to the session: session.add(prof) )

prof.addresses = [Address(email_address='profx@dc.com'),
                        Address(email_address='xavier@yahoo.com')]

See how easy that was? It’s even easy to get the information back out. For example, if you wanted to access just the first address, you would just call prof.addresses[0]. Now, say that you need to change one of the addresses (i.e. do an UPDATE). It’s as easy as pie:

# change the first address
prof.addresses[0].email_address = "profx@marvel.com"

Now let’s move on to doing queries on joins:

for u, a in session.query(User, Address).filter(User.id==Address.user_id).filter(Address.email_address=='xavier@yahoo.com').all():
    print u, a

That is one long query! I find those hard to follow myself, so I usually do the following to make it easier on my brain:

sql = session.query(User, Address)
sql = sql.filter(User.id==Address.user_id)
sql = sql.filter(Address.email_address=='xavier@yahoo.com')
 
for u, a in sql.all():
    print u, a

Now, for those of you that like the one-liners, there is nothing wrong with the first example. It will produce the exact same results. I just happen to find the longer version easier to debug. Finally, we can also use a real join:

from sqlalchemy.orm import join
session.query(User).select_from(join(User, Address)).filter(Address.email_address=='xavier@yahoo.com').all()

This also does the same as the previous two examples, but in a more explicit manner. For more information on joins using the Object Relational syntax, I recommend the official documentation.

Wrapping Up

At this point, you should be able to create your database with tables, populate the tables with data as well as select, update and commit transactions to your database using SqlAlchemy. I hope you found this tutorial helpful in understanding this amazing technology.

Note: This tutorial was tested on Windows with Python 2.5 and SqlAlchemy 0.5.8.

Further Reading

Another Step-by-Step SqlAlchemy Tutorial (part 1 of 2)

Mike — Thu, 04 Feb 2010 01:31:59 +0000

A long time ago (circa 2007 if Google serves me right), there was a Python programmer named Robin Munn who wrote a really nice tutorial on SqlAlchemy. It was originally based on the 0.1 release, but updated for the newer 0.2. Then, Mr. Munn just disappeared and the tutorial was never updated. I have been kicking around the idea of releasing my own version of this tutorial for quite some time and finally decided to just do it. I hope you will find this article helpful as I found the original to be.

Getting Started

SqlAlchemy is usually referred to as an Object Relational Mapper (ORM), although it is much more full featured than any of the other Python ORMs that I’ve used, such as SqlObject or the one that’s built into Django. SqlAlchemy was founded by a fellow named Michael Bayer. I usually see Jonathan Ellis’ name floating around the project a lot too, especially at PyCon.

This tutorial will be based on the latest released version of SqlAlchemy: 0.5.8. You can check your version by doing the following:

import sqlalchemy
print sqlalchemy.__version__

Note: I’ll also be using Python 2.5 on Windows for testing. However, this code should work equally well on Mac and Linux. If you need SqlAlchemy to work on Python 3, then you’ll want the SVN version of 0.6. The website gives instructions on how to do acquire the code.

If you don’t happen to have SqlAlchemy, you can download it from their website or use easy_install if you have setuptools installed. Let’s see how:

In the case of downloading the source, you’ll need to extract it and then open a console window (on Windows, go to Start, Run and type “cmd”, without the quotes). Then change directories until you are in the unzipped folder. To install, do this:

python setup.py install

You may need to specify the path to your python if it’s not on the system path. For easy_install, it’s even easier:

easy_install sqlalchemy

This also assumes that you have easy_install on your path. If you do not, then use the full path to use it (i.e. c:\python25\scripts\easy_install or whatever).

Creating the First Script

Now we get to create our first example using SqlAlchemy. We will create a simple table to store a user’s name, their age and their password.

from sqlalchemy import create_engine
from sqlalchemy import MetaData, Column, Table, ForeignKey
from sqlalchemy import Integer, String
 
engine = create_engine('sqlite:///tutorial.db',
                       echo=True)
 
metadata = MetaData(bind=engine)
 
users_table = Table('users', metadata,
                    Column('id', Integer, primary_key=True),
                    Column('name', String(40)),
                    Column('age', Integer),
                    Column('password', String),
                    )
 
addresses_table = Table('addresses', metadata,
                        Column('id', Integer, primary_key=True),
                        Column('user_id', None, ForeignKey('users.id')),
                        Column('email_address', String, nullable=False)
                        )
 
# create tables in database
metadata.create_all()

Looking Deeper

As you can see, we needed to import various bits and pieces from the sqlalchemy package, namely create_engine, MetaData, Column, Table, Integer, and String. Then we create an “engine” which is basically an object that knows how to communicate with the provided database using the credentials you supply. In this case, we are using a Sqlite database that doesn’t need credentials. There are in-depth docs on this subject alone where you can read up on for your favorite database flavor. Also notice that we set echo to True. This means that SqlAlchemy will output all the SQL command it is executing to stdout. This is handy for debugging, but should be set to False when you’re ready to put the code into production.

Next, we create a MetaData object. This cool creation from the SqlAlchemy team holds all the database metadata. It consists of Python objects that hold descriptions of the tables and other schema-level objects of the database. We can bind the metadata object to our database here or in the create_all statement near the end of the code.

The last section is how we create the tables programmatically. This is accomplished by using SqlAlchemy’s Table and Column objects. Notice that we have various field types available to us, like String and Integer. There are many others too. For this example, we create a database and name it “users”, then pass in our metadata object. Next, we put it in the Columns. The “id” column is set as our primary key. SqlAlchemy will magically increment this for us as we add users to the database. The “name” column is a String type and capped at 40 characters long. The “age” column is just a simple Integer and the “password” column is just set to String. We didn’t set its length, but we probably should. The only major difference in the addresses_table is how we set up the Foreign key attribute that connects the two tables. Basically, we point it at the other table by passing the correct field name in a string to the ForeignKey object.

The final line of this snippet actually creates the database and the table. You can call it as often as you like as it will always check for the existence of the specified table before trying to create it. That means you can create additional tables and call create_all and SqlAlchemy will only create the new table.

SqlAlchemy also provides a way to load tables that were previously created:

someTable = Table("users", metadata, autoload=True, schema="schemaName")

I’ve noticed that with this release, SqlAlchemy has gotten pretty picky about having the database schema specified when you autoload a database. Should you have that issue, you’ll need to add the following to your Table definition: schema=”some schema”. For more information, see the documentation.

Inserting

There are several ways to add and extract information from the database. We’ll look at the low-level ways first and then in one of the other parts of this series, we’ll get into Sessions and the Declarative styles which tend to be slightly more abstract. Let’s take a look at the different ways to Insert data into our database:

# create an Insert object
ins = users_table.insert()
# add values to the Insert object
new_user = ins.values(name="Joe", age=20, password="pass")
 
# create a database connection
conn = engine.connect()
# add user to database by executing SQL
conn.execute(new_user)

The code above shows how to use a Connection object to do the insert. First, you need to create the Insert object by calling the table’s insert method. Then you can use the Insert’s values method to add the required values for the row. Next, we create the Connection object via the engine’s connect method. Finally, we call the Connection object’s execute method on the Insert object. It sounds kind of complicated, but it’s actually pretty easy.

This next snippet shows a couple ways to do an insert without the Connection object:

# a connectionless way to Insert a user
ins = users_table.insert()
result = engine.execute(ins, name="Shinji", age=15, password="nihongo")
 
# another connectionless Insert
result = users_table.insert().execute(name="Martha", age=45, password="dingbat")

In both cases, you will need to call the table object insert method. Basically, you just take the engine out of the picture in the second instance. The last insert method we’ll look at is how to insert multiple rows:

conn.execute(users_table.insert(), [
    {"name": "Ted", "age":10, "password":"dink"},
    {"name": "Asahina", "age":25, "password":"nippon"},
    {"name": "Evan", "age":40, "password":"macaca"}
])

This is pretty self-explanatory, but the gist is that you need to use the Connection object from earlier and pass it two arguments: the table’s Insert object and a list of dictionaries which contain column name and value pairs. Please note that by using the execute method, the data is committed to the database in these examples.

Now let’s move on to doing selects.

Selecting

SqlAlchemy provides a robust set of methods to accomplish Selects. We will focus on the simple methods here. For the advanced stuff, I recommend their official documentation and mailing list. One of the most common examples is doing a select all, so let’s start with that:

from sqlalchemy.sql import select
 
s = select([users_table])
result = s.execute()
 
for row in result:
    print row

First we have to import the select method from sqlalchemy.sql. Then we pass it the table as a one element list. Finally we call the select object’s execute method and store the returned data in the result variable. Now that we have all the results, we should probably see if we got what we expected. Thus, we create a for loop to iterate over the result.

If you need all the results in a list of tuples rather than Row object, you can do the following:

# get all the results in a list of tuples
conn = engine.connect()
res = conn.execute(s)
rows = res.fetchall()

And if you just need the first result back, then you use fetchone() instead of fetchall():

res = conn.execute(s)
row = res.fetchone()

Now let’s pretend that we need to get a little bit more granular in our results. In the next example, we just want to return the user’s name and their age while omitting their password.

s = select([users_table.c.name, users_table.c.age])
result = conn.execute(s)
for row in result:
    print row

Well, that was pretty easy. All we had to do was specify the column names in our select statement. The little “c” basically means “column”, so we do a select on column name and column age. If you had multiple tables, then the select statement would be something like this:

select([tableOne, tableTwo])

Of course, this will probably return duplicate results, so you’ll want to do something like this to mitigate the issue:

s = select([tableOne, tableTwo], tableOne.c.id==tableTwo.c.user_id)

The SqlAlchemy documentation called the first result a Cartesian product since it caused each row from the first table to be produced against each row of the second table. The second statement above eliminates that annoyance. How? Well, this is the way you do a WHERE clause using this form of a select. In the next part of the series, I’ll show a different way to do a select and where with sessions.

Here are a couple more examples though with explanations in the comments:

from sqlalchemy.sql import and_
 
# The following is the equivalent to 
# SELECT * FROM users WHERE id > 3
s = select([users_table], users_table.c.id > 3)
 
# You can use the "and_" module to AND multiple fields together
s = select(and_(users_table.c.name=="Martha", users_table.c.age < 25))

The code above illustrates that SqlAlchemy can use operators and conjunctions in their queries as well. I recommend reading their documentation for full details here.

Wrapping Up

I think this is a good place for us to stop. We have now learned how to create a database, add rows and select data from our database. In the next part of our series, we will learn the more popular way of doing this using the Object Relation methods. We will also learn about a few other key topics, such as the SqlAlchemy session. We will also look at how joins work in SqlAlchemy. See you then!

Further Reading

Downloads

The Mouse Vs. The Python » SqlAlchemy

Improving MediaLocker: wxPython, SQLAlchemy, and MVC

Making MediaLocker Better

Werner’s Additions

Wrapping Up

Source Code

wxPython and SqlAlchemy: An Intro to MVC and CRUD

Prerequisites

The Model

The View

The Controller and CRUD

Wrapping Up

Source Code

SqlAlchemy ProgrammingError 42000 and MS SQL

Top Ten Articles of 2010

SqlAlchemy: A Simple Tutorial for Elixir

Creating a Database with Elixir

Reading a Pre-Existing Database

Wrapping Up

Further Reading

SqlAlchemy and Microsoft Access

SqlAlchemy: Connecting to pre-existing databases

SqlAlchemy’s autoload

Defining Your Databases Explicitly

Wrapping Up

Further Reading

Downloads

Another Step-by-Step SqlAlchemy Tutorial (part 2 of 2)

Getting Used to Data Mapping

Class is now in Session

Join in the Fun

Wrapping Up

Another Step-by-Step SqlAlchemy Tutorial (part 1 of 2)

Getting Started

Creating the First Script

Looking Deeper

Inserting

Selecting

Wrapping Up