Dev-Picayune

I was working on a section of my Checkers game code a few days ago. It looked like this:

class Checkerboard(object):
   def legal_moves(self, state):
	return state.captures or state.moves

The behavior above mirrors the standard Checkers rule that if you have a capture, you must take it — otherwise, you can make a regular move. In my code, both state.captures and state.moves are properties that return Python lists. Since an empty list evaluates to False in an or statement, if state.captures is empty, then the return statement automatically uses state.moves list instead.

Now I was trying to make the legal_moves method work a bit faster and use less memory, so I switched to generators instead. Oops … now the code above fails since state.captures and state.moves now return generator objects. The legal_moves method above doesn’t work because state.captures is always a valid object instance and it always evaluates to True in the or statement — even when there are no actual captures available.

It took some thought to come up with an equivalent behavior for generator objects. In the new version, legal_moves becomes a wrapper method that returns one generator object or the other based on whether the first generator object yields at least one item.

class Checkerboard(object):
   def legal_moves(self, state):
      return state.captures if state.captures.next() else state.moves

It’s not as elegant as the version using lists, but it works.

Lately, I’ve found myself switching from Wing IDE to Komodo IDE for my Python work. One of the major reasons for this has been my switch to using a Mac at work. While Wing runs on the Mac, it’s really an X11 app and doesn’t have the full Mac’y feel that Komodo has. Another reason, is that while in the past, Komodo’s code completion was “less” it’s now gotten a lot better and while still not up to the excellent standard of Wing, it’s pretty good now.

So for work at home, I figured I’d fully make the switch as well. That’s when I found an issue. While I had managed to get my Mac code completing well for my django app’s, my 9.04 ubuntu box at home wasn’t running so well. Just trying to code complete the django models object didn’t work.

After trying everything I’d seen via googling, I found that adding the following directories to the additional directories import seemed to work:

/usr/local/lib/python2.6/dist-packages

/usr/local/lib/python2.6/site-packages

I’m sure I’ll end up adding more directories to make it all work right… but these fixed the basic django code completion issues. Just thought I’d post this in case I forgot what I did.

This post discusses an object pool design pattern that is applicable in the following scenario:

• You have a list of objects whose state is continually updated.
• The objects are frequently added or deleted from the list, i.e. have limited lifetimes.
• You want to reuse the objects instead, to improve performance or simplify your code.

Introduction

When writing my game Asteroid Smash using Python, I used lists to keep track of on-screen objects like laser shots and asteroids. These objects all have limited lifetimes (for example, a laser shot that only goes a predetermined distance, or an asteroid that drifts across the screen until it gets destroyed). When it’s no longer needed, the object then gets deleted from its associated list. Here is a simplified example:

import random
 
MAX_SHOTS = 5
 
class Shot(object):
    def __init__(self, lifetime):
        self.lifetime = lifetime
 
    def update(self):
        self.lifetime -= 1
        return self.lifetime > 0
 
# create list of Shots with random lifetimes
lst = [Shot(random.randint(1,100)) for x in range(MAX_SHOTS)]
 
# update loop
while lst:
    delete_lst = []
    for i, x in enumerate(lst):
        if x.update(): # decrement object lifetime
            delete_lst.append(i)
 
    # delete any objects whose time count is 0
    for i in reversed(delete_lst):
        del lst[i]
print "Done!"

To summarize what’s going on, a list of 5 Shot objects gets created, each with random lifetime values, and added to a list. After that, a loop is started that calls an update method on each Shot that decrements its lifetime variable by 1. (This roughly simulates what would happen during each pass of a typical game loop.) Once the object’s lifetime value reaches zero, then the object is added to a delete list. Any objects in the delete list get destroyed after each pass of the update loop.

This code works, but it’s clumsy. Note all the bookkeeping that’s necessary. I can’t delete items while I’m iterating through the list, or I’ll invalidate the iterator. This means I have to keep track of the index of which items need to be deleted, and then actually delete them in a second loop. Also, I have to delete the items in reverse order, or I’ll throw off the indexes and delete the wrong items.

To improve the code, I decided to try list comprehensions to shorten the amount of code in the update loop.

# new update loop with list comp
while lst:
    lst = [x for x in lst if x.update()]

This is much more appealing, single-pass approach. However, the list comprehension, while tidy looking, has a big disadvantage: it constantly creates and destroys a list during each iteration of the loop. This makes it a big problem if we pass around a reference to the list within our program, as the reference is immediately invalidated when the list is recreated. If we need to keep the reference to our list constant, then we have to try another approach.

Rethinking the problem

Perhaps if we want to make a better design, then we need to recast the problem slightly. Notice that once all the objects have a lifetime count of 0, then they can all be deleted at one time, which is a straightforward problem. This means filtering the list by deleting the objects turns out to be unnecessary; what we want is a way to separate objects that still need to be updated in the list from those that do not. Then, once all of the objects fail the update call, the entire list can be deleted.

A little-known algorithm in the C++ Standard Template Library (STL) called partition does the separation of elements mentioned above. Given a sequence, partition will reorder items in the sequence so that all items that satisfy a certain condition (like an update call!) precede those that fail the condition. One other great thing about partition is that it works in linear time; assuming a list of N items, then each use of partition will apply the condition check N times and swap items in the list at most N/2 times. [1]

Here’s what partition looks like in Python (translated from SGI’s version of the STL algorithm [2]; comments added by me based on the SGI docs.)

def partition(pred, seq, first, last):
    """ Partition reorders the elements in seq in the range [first, last)
        based on the function pred, such that the elements
        that satisfy pred precede the elements that fail to satisfy it.
        The postcondition is that, for some index value middle in the
        range [first, last), pred(x) is true for every item x in the
        range [first, middle) and false for every item x in the range
        [middle, last). The return value of partition is middle. """
 
    # quick error check on the list indices
    if first > last:
        return 0
 
    # find first element that fails to satisfy the predicate
    for i in range(first, last):
        if not pred(seq[i]):
            break
    else:
        return last
 
    # for all remaining elements in the sequence, if we find one
    # that satisfies the predicate, swap it with one that is known
    # to have (possibly) failed the test. All swaps are necessary
    # to preserve the correct ordering upon exit.
    for j in range(i+1, last):
        if pred(seq[j]):
            seq[i], seq[j] = seq[j], seq[i]
            i += 1
    return i

We also need a predicate function to use with partition, since the update method is actually part of our Shot class and can’t be called directly. We could use a lambda function instead, but that is somewhat slower than a named function.

def update_object(x):
    return x.update()

Using our new partition and update_object functions, our update loop changes to:

end = len(lst)
while end:
    end = partition(update_object, lst, 0, end)

This is one line longer than our list comprehension version, but we also don’t invalidate our reference to the list on each pass since partition performs in-place modification to our list. This makes it better on both counts.

One other item of interest is the variable end. After each pass of the loop, the end index marks the separation between “live” and “dead” objects in the list. As we noted earlier, once end reaches 0, all the items in the list can then be deleted. Look closely though … we were only deleting objects from the list before in order to filter those that didn’t need updating, but partition already does the filtering for us. Since all the objects in the list are, in fact, still good, to reuse one of them we just need to reset the object pointed to by the end index and then increment end by one. If we try to reuse one of the objects in the list and end goes beyond the last item in the list, then we can either expand the list or just ignore the request, depending on what behavior we want.

Managing details

Here’s the interesting thing: by switching our code to use partition, what we’ve created is a simple object pool. At this point though, there are too many bookkeeping details, and it would be better to encapsulate the pool inside a class.

class ObjectPool:
    def __init__(self, **kwargs):
        self._clsname = kwargs['classname']
        self._args = kwargs.get('args') or []
        self._num_objects = max(kwargs['num'], 0)
        self._pred = kwargs['update_func']
        self._max_objects = kwargs.get('max', self._num_objects)
 
        # Create the objects
        self._objs = [apply(self._clsname, self._args)
                      for x in range(self._num_objects)]
        self._end = len(self._objs)
 
    def _extend_list(self, args):
        """ default policy is to add one object, up to max_objects ...
        override with different behavior if needed. """
        self._objs.append(apply(self._clsname, args))
        self._num_objects += 1
 
    def add(self, *args):
        """ Add one new (live) object to the pool. """
        newend = self._end + 1
        # if we reach our predefined maximum number of objects,
        # don't create any more.
        if newend > self._max_objects:
            return None
        # if we bump up against the end of our list but are below the
        # max number of objects, extend the list
        if newend > len(self._objs):
            self._extend_list(args)
        else:
            self._objs[self._end].reset(*args)
        self._end += 1
        return self._end - 1
 
    def update(self, *args):
        """ Update each object in the pool, using the pred function. """
        self._end = partition(self._pred, self._objs, 0, self._end, args)
        return self._end

That’s perhaps a lot to digest. Let’s look at a short code sample of how to use the new class that will make it easier to understand.

shots = ObjectPool(classname=Shot, update_func=update_object, num=5)
while shots.update():
    pass
print "Done!"

You can see here that the ObjectPool class takes keyword parameters. The first parameter, classname, specifies the name of each object’s class in the pool. The second parameter, update_func, specifies the predicate function used to update the objects. The third parameter, num, specifies the initial number of objects created. (There is also an optional fourth parameter, max, that specifies the maximum number of objects within the pool; if max isn’t explicitly specified, then it will default to num.)

Besides the update method we’ve already discussed, the add method will attempt to either insert a new object in the pool or reset an existing object whose lifetime has expired. (The add method assumes that each object in the pool has implemented a reset method that reinitializes the object.) If neither option is possible (the pool has reached its max or no objects have expired), then add will simply return None to indicate a failure. (You can override the _extend_list method if you wish to change this default behavior.)

A short demo of object pooling

To demonstrate the utility of this code, I’ve modified Alex Holkner’s Noisy demo in the pyglet 1.1.2 distribution to use my ObjectPool class. His example program displays balls bouncing around the edges of a windowed display. Normally the balls bounce indefinitely until the user presses a Delete key, but I changed the demo to have the balls exist for only a short time before they are deleted from the display. The sample also demonstrates how the ObjectPool prevents you from adding more than the max number of objects defined in its constructor. This demo, along with the provided unit tests, should give you a good starting point for using ObjectPool in your own games or applications.

Download op.zip (17 Kb)

References:
[1] http://www.sgi.com/tech/stl/partition.html
[2] http://www.sgi.com/tech/stl/stl_algo.h

Okay, very weird problem today. I am not sure what to make of it… when using the latest version of MySQLdb (1.2.2.final.0) which is the version in Ubuntu 7.10 Gutsy, I am getting an “incomplete format” error when performing an executemany. My variables are presented as a list of dictionaries as I’m using named/mapped variables in the SQL statement. Using the exact same code with MySQLdb (1.2.1.final.2) is fine (the version in Ubuntu Feisty 7.04).

So for completeness sake, I have documented the scenario like this:

On a MySQL server, you just need a simple little table:

CREATE TABLE `test_table` (
`id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`test_col_1` varchar(20) NOT NULL,
`test_col_2` varchar(20) NOT NULL,
PRIMARY KEY  (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=latin1

Then the Python code (obviously you’ll need to set the connection stuff appropriately):

import MySQLdb
 
print "-----------------------------------------------------"
print MySQLdb.version_info
print "-----------------------------------------------------"
 
sqlStringItems = """INSERT INTO test_table
(test_col_1,test_col_2)
VALUES
(%(test1)s, %(test2)s);"""
 
dtlDict = {'test1':'test value 1','test2':'test value 2'}
 
con = MySQLdb.connect(host="localhost",user="myuser",passwd="mypassword",db="test")
cur = con.cursor()
exitval = cur.executemany(sqlStringItems,[dtlDict])
 
print "exit value from executemany= " + str(exitval)
cur.close()

version (1, 2, 2, ‘final’, 0) yields:

ValueError: incomplete format

Whereas version (1, 2, 1, ‘final’, 2) works fine.

Also, doing the same thing, but with execute instead of executemany works fine (and of course taking the dict out of the list). So this seems to be purely an issue with executemany. Looking at the MySQLdb code, it’s obvious that between the two versions it was pretty radically changed.

I guess I need to report this as a bug, but I’m just finding it hard to believe that I didn’t do something wrong.

Update (2008-06-03)
Back in January, I reported this issue (1874176). Then in late April, Raphael Guillet submitted a possible fix. I haven’t spent much time looking at the right vs. wrong of the fix, but I have tried it and it appears to work. So for those looking, here’s the possible fix which as near as I can tell has still not been cut into the official version.

The change is in the cursors.py file. So you have to hunt that down wherever it is stored in your particular OS version and/or install.

Comment out lines 201 and 202:

        #e = m.end(1)
        #qv = m.group(1)

And then add right after that:

        e=len(query)
        qv = query[p:e]

I’ll try to continue to monitor this.

(now updated with some pointless Python code)

This is perfect for this little blog concerned with trivial and small things. I noticed that my Peanuts calendar on my desk is essentially a reproduction of the 1990 calendar. Obviously, they all have to be reproductions because Charles Schulz died in 2000. So looking more closely, I realized they’d chosen 1990 primarily because the dates and days of 1990 match 2007. So a quick hack with the ‘cal’ program under linux and I was off and searching for what will most likely be next year’s calendar. Since 2008 is a leap year and obviously 1991 is not, I knew that wasn’t going to work.

First I generated target calendar with:

cal 2008 > cal2008.txt

Then, I just manually (although I could have written a script) looped through a few known leap years and found that 1980 seems to match perfectly as well.

cal 1980|diff - cal2008.txt

Obviously, what we’re looking for is a diff of only the first line, the year. Side note: the selection of 1952 (also a match), would seem to be fairly unlikely.

For 2009, there are a bunch of choices including: 1953, 1959, 1970, 1981, 1987 and 1998.

Why am I posted this? I don’t know other than it was fun to use ‘cal’ and ‘diff’ to anticipate which Peanuts calendar would be next.

UPDATE

I couldn’t resist doing a Python solution to this little problem. I know this isn’t as efficient as someone else could make it, but it was entertaining for me to do it. It took about 5 or 6 minutes to write and ran correctly the first time (matching my previous results):

import sys 
import datetime
def matchCals(inputYear,minYear,maxYear):
    sDayOfWeek = datetime.date(inputYear,1,1).weekday()
    eDayOfWeek = datetime.date(inputYear,12,31).weekday()
    for testYear in range(minYear,maxYear+1):
        if (sDayOfWeek == datetime.date(testYear,1,1).weekday() and eDayOfWeek == datetime.date(testYear,12,31).weekday()):
            print "Match Found... %s = %s" % (str(testYear) , str(inputYear))
 
if __name__ == "__main__":
    matchCals(int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]))

Here I just check the day of the week on the first day of the year, and the last day of the year. If both of those fall on the same day, I assume the calendars for those years are the same. And since we’re dealing with a limited set of dates (valid complete Peanuts calendars running roughly 1951 to 1999, it handles a min and max year for input as well. So to run it for 2008, you just type in:

python ./caltest.py 2008 1951 1999

Of course, there is no error checking… so if you enter a string or an invalid year… that’s your own problem. I just wanted to see how quick I could do it. Obviously, I’ve waisted more of my time… and more of your time.