Nose is a fantastic testing framework. What surprises me though, is that there’s no out of the box plugin to time tests to see which tests are the slowest, and most likely, problematic. After all, unit tests are supposed to be wicked fast. I googled, but nothing really came up except an insightful google-groups post.

I figured, what the hey, might as well just write a simple nose plugin to time the tests. I modeled it slightly off the xunit nose plugin. With the xunit plugin as a guiding example, I thought it was pretty trivial to write a nose plugin, definitely a testament to the great design chosen by nose.

Here’s the plugin below:

"""This plugin provides test timings to identify which tests might be
taking the most. From this information, it might be useful to couple
individual tests nose's `--with-profile` option to profile problematic
tests.

This plugin is heavily influenced by nose's `xunit` plugin.

Add this command to the way you execute nose::

    --with-test-timer

After all tests are executed, they will be sorted in ascending order.

(c) 2011 - Mahmoud Abdelkader (http://github.com/mahmoudimus)

LICENSE:
            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
                    Version 2, December 2004

 Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>

 Everyone is permitted to copy and distribute verbatim or modified
 copies of this license document, and changing it is allowed as long
 as the name is changed.

            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

  0. You just DO WHAT THE FUCK YOU WANT TO.

"""

import operator
from time import time

import nose
from nose.plugins.base import Plugin


class TestTimer(Plugin):
    """This plugin provides test timings

    """

    name = 'test-timer'
    score = 1

    def _timeTaken(self):
        if hasattr(self, '_timer'):
            taken = time() - self._timer
        else:
            # test died before it ran (probably error in setup())
            # or success/failure added before test started probably
            # due to custom TestResult munging
            taken = 0.0
        return taken

    def options(self, parser, env):
        """Sets additional command line options."""
        super(TestTimer, self).options(parser, env)

    def configure(self, options, config):
        """Configures the test timer plugin."""
        super(TestTimer, self).configure(options, config)
        self.config = config
        self._timed_tests = {}

    def startTest(self, test):
        """Initializes a timer before starting a test."""
        self._timer = time()

    def report(self, stream):
        """Report the test times"""
        if not self.enabled:
            return
        d = sorted(self._timed_tests.iteritems(), key=operator.itemgetter(1))
        for test, time_taken in d:
            stream.writeln("%s: %0.4f" % (test, time_taken))

    def _register_time(self, test):
        self._timed_tests[test.id()] = self._timeTaken()

    def addError(self, test, err, capt=None):
        self._register_time(test)

    def addFailure(self, test, err, capt=None, tb_info=None):
        self._register_time(test)

    def addSuccess(self, test, capt=None):
        self._register_time(test)


if __name__ == '__main__':
    nose.main(addplugins=[TestTimer()])

Something cool about nose was that you don’t have to install a plugin package-wide using setuptools, but you can actually just dynamically add it during run-time. All you have to do is:

import nose
from yourplugin import YourPlugin

if __name__ == '__main__':
    nose.main(addplugins=[YourPlugin()])

This way, you can just execute your tests as normal, like so:

python nose-testtimers.py --with-test-timer -sv --debug=sqlalchemy.engine

and you get a nice little output of test times

I had a need to trace a function’s call stack to identify call chain paths in some difficult-to-follow Python code that was laced with lots of magic and abstractions.

I tried stepping through ipdb, but it took forever. So, I thought to myself, why can’t I just take a stack trace and I’ll be able to get the call stack. It didn’t help that this function was apparently called multiple times from different places so when I tried raising an unhandled exception, it only helped display the stack trace for the first call to the function, not the successive ones. I also tried throwing and catching the exception, assuming that a stack trace can share some information about its call chain. That didn’t work too well because the exception masked the stack[1].

So I wrote some simple code that will do exactly that, take a stack trace and format it as if it looked like an exception traceback. Without further ado, here is, as far as I know, the only way to get an arbitrary stack trace in Python from any line of code.

    import inspect
    import traceback

    # get the currently frames' stack
    # this returns the frameobject, the filename,
    # the line number of the current line, the
    # function name, a list of lines of context from
    # the source code, and the index of the current
    # line within that list.
    stack = inspect.stack()
    # reverse the stack trace so the most recent is at the bottom of the stack
    stack.reverse()
    try:
        stack_list = []
        for s in stack:
            _, filename, line_no, func_name, code_list, index_in_code_list = s
            stack_list.append(
                (filename, line_no, func_name, code_list[index_in_code_list]))
        print ''.join(traceback.format_list(stack_list))
    finally:
        # avoid memory leak issues
        del stack

Hope this helps in your debugging adventures.

I’d love to hear if there are any other ways of doing the same thing.

[1] I should probably revisit this later to see why it didn’t work. In theory it should do what the stack trace snippet above does.

EDIT:

My original assumption was right. Using a stack trace to get the call chain is one way of doing it.

Turns out Python already does this in the traceback.print_stack function.

Thanks to @teepark for the comment!

I’ve recently had to do some work that required sorting a very large CSV file, containing fields with embedded newlines, quickly. As it turns out, Linux comes with a sort implementation that has a “–zero-terminated” option, which sorts on null-terminated delimited strings instead of the default newline separator.

Writing null-terminated CSV files

Since I was writing a process to generate these CSV files, I figured I can just use Python’s CSV module, which has support for different types of dialects. Inheriting from csv.Dialect, we can write a simple dialect that will allow us to terminate all lines with a null byte.

import csv
import struct

class null_terminated(csv.excel):
    lineterminator = struct.pack('B', 0)

csv.register_dialect("null-terminated", null_terminated)

Essentially, we’ve registered a global csv dialect called "null-terminated" that inherits from the excel dialect, which has sensible standard defaults.

Here’s a simple snippet that shows the usage of the new "null-terminated" dialect that I created above.

from csv import DictWriter

with open("/tmp/file.csv", "w") as f:
	dwriter = DictWriter(f, fieldnames=["id","field"], dialect="null-terminated")

	for i, field in enumerate(("foo", "bar", "baz", "bif")):
    		dwriter.writerow({"id": i, "field": field})

Now, /tmp/file.csv will contain a file with four rows that are separated by a null-terminator. As you can see, it’s pretty easy to write a null-terminated CSV file, but unfortunately, it’s a bit tricky to read a null-terminated csv file due to some inflexible hardcoded defaults.

Reading null-terminated CSV files

The CSV module’s unintuitive restriction for Dialect.lineterminator is hard-coded to recognize '\r' or '\n' as the end of line terminator, which unfortunately, means we will need to handle null-termination and implement reading ourselves.

There are many ways of writing a procedure to read null-terminated strings, but I figured the simplest algorithm is to read character-by-character, concatenating everything into a string until we reach a null byte, then we can just return the string. I’d figure an implementation might go something like this:

def read(fobj):
    current_string = ""
    while True:
        char = fobj.read(1)
        if char and char != nullbyte:
            current_string += char
        elif char == nullbyte:
            yield current_string
            current_string = ""
        elif not char:
            if current_string:
                yield current_string
            raise StopIteration

Looks awesome, but, how can we integrate this into the CSV module? We would want to just plug and play with the existing CSV module. A simple solution is to wrap the function above to iterate over each line, like so:

 # we use StringIO since cStringIO has poor unicode support
from StringIO import StringIO
from csv import reader

class NullTerminatedDelimiterReader(object):
    """
    A CSV reader which will iterate over lines in the CSV file 'f',
    which are line terminated by a null byte

    """

    def __init__(self, f,  dialect, *args, **kwds):
        # satisfying DictReader instance
        self._line_num = 0
        self.fobj = f
        self.dialect = dialect
        self.reader = self._read()
        self.string_io = StringIO()

    def _properly_parse_row(self, current_string):
        self.string_io.write(current_string)
        # seek to the first byte
        self.string_io.seek(0)
        # we instantiate a reader here to properly parse the row
        # taking into account escaping, and various edge cases
        return next(reader(self.string_io, dialect=self.dialect))

    def _read(self):
        current_string = ""
        while True:
            char = self.fobj.read(1)  # read one byte
            if char and char != null_byte:
                # keep appending to the current string
                current_string += char
            elif char == null_byte:
                yield self._properly_parse_row(current_string)
                # increment instrumentation
                self._line_num += 1
                # clear internal reading buffer
                self.string_io.seek(0)
                self.string_io.truncate()
                # clear row
                current_string = ""
            elif not char:
                if current_string:
                    yield self._properly_parse_row(current_string)
                raise StopIteration

    @property
    def line_num(self):
        return self._line_num

    def next(self):
        return next(self.reader)

    def __iter__(self):
        return self

To use the DictReader class, we’ll inherit from the DictReader class and override the reader object. It’s the cleanest and simplest way of doing it.

class NullByteDictReader(csv.DictReader):
    def __init__(self, f, *args, **kwds):
        csv.DictReader.__init__(self, f, *args, **kwds)
        self.reader = NullTerminatedDelimiterReader(f, *args, **kwds)

with open("/tmp/file.csv", "r") as f:
    for line in NullByteDictReader(f, dialect="null-terminated"):
        print line["id"], line["field"]

Voila

Conclusions and Future Work

Something that might be interesting to pursue further is the possibility of writing, or wrapping a python interface around, a C library as a substitute for the current CSV module. It should be able to support different line terminators, multi-byte delimiters, and have unicode detection outside the box, which happen to be my main three gripes with the CSV module.

You can find working source code and implementation of this here and you should follow me on twitter here.

Over the weekend, I was working on upgrading python-ngrams because I had discovered a bug where the tokenization was incorrect. I was reading a research paper that was describing q-grams and while following their examples, I realized I was getting incorrect results for a fundamental n-gram result.

The tokenization that’s required here is quite simple, given some iterable consisting of values [x0, x1, x2, x3...,xi], produce an exhaustive iterator of n-tuples such that it satisfies (x0,...xn), (x1,...xn+1), (xm,...xi).

To make this easier to understand, if given a list of [0, 1, 2, 3, 4, 5, 6], I want to be able to return an iterator such that:

for first, second, third in n_wise([0, 1, 2, 3, 4, 5, 6], 3):
    print first, second, third

Will have a result of:

0, 1, 2
1, 2, 3
2, 3, 4
3, 4, 5
4, 5, 6

Fortunately, this wasn’t too difficult as the trivial implementation of this is already done in itertools, under pairwise. Below I’ve implemented a n-wise iterator implementation that can take any iterator and return n-iterators where each iterator is advanced by a step ahead of the other.

from itertools import tee, izip

def n_wise(iterable, n):
    """Returns n iterators for an iterable that are sequentially
    n-wise

    """
    n_iterators = tee(iterable, n)
    zippables = [n_iterators[0]]

    for advance, iteratee in enumerate(n_iterators[1:]):
        advance += 1  # since enumerate is 0 indexed.
        while advance > 0:
            # we advance the iterator `advance+1` steps
            next(iteratee, None)
            advance -= 1
        # append everything to the zippables
        zippables.append(iteratee)
    # return the izip expansion of each iterator
    return izip(*zippables)

I find that I sometimes need to open an iterator for a file and I need to read n-wise lines each step, this iterator will do just that. For sake of completeness, look at how concise, powerful, and easy to use this combination is:

# assuming that n_wise is imported into this namespace
from functools import partial

triple_wise_line_reader = partial(n_wise, n=3)
for line1, line2, line3 in triple_wise_line_reader(open("some_file.log", "r")):
    # do some computation with line1, line2 and line3
    do_something(line1, line2, line3)
    # next step:
    # line1 <- line2
    # line2 <- line3
    # line3 <- line4

Imagine doing this in C It will be just ugly!

Please let me know if you find this useful or you have a better implementation to the problem!

I just recently purchased a MacBook Pro, which comes with Snow Leopard installed, and I noticed that it comes with python 2.6.1 installed. I wanted to upgrade to the latest python release of 2.6.4, so I tried installing the official python Mac OS distribution from python.org. After installation, I wanted to install Twisted and I kept getting this error below:

creating build/temp.macosx-10.3-fat-2.6
creating build/temp.macosx-10.3-fat-2.6/twisted
creating build/temp.macosx-10.3-fat-2.6/twisted/runner
gcc-4.0 -arch ppc -arch i386 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c twisted/runner/portmap.c -o build/temp.macosx-10.3-fat-2.6/twisted/runner/portmap.o
In file included from /usr/include/architecture/i386/math.h:626,
 from /usr/include/math.h:28,
 from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/pyport.h:235,
 from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:58,
 from twisted/runner/portmap.c:10:
/usr/include/AvailabilityMacros.h:108:14: warning: #warning Building for Intel with Mac OS X Deployment Target < 10.4 is invalid.
Compiling with an SDK that doesn't seem to exist: /Developer/SDKs/MacOSX10.4u.sdk
Please check your Xcode installation
gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup build/temp.macosx-10.3-fat-2.6/twisted/runner/portmap.o -o build/lib.macosx-10.3-fat-2.6/twisted/runner/portmap.so
ld: library not found for -lbundle1.o
ld: library not found for -lbundle1.o
collect2: ld returned 1 exit status
collect2: ld returned 1 exit status
lipo: can't open input file: /var/folders/T6/T6diKRiFGJSwsabKP4864E+++TI/-Tmp-//ccIK1c3K.out (No such file or directory)
error: command 'gcc-4.0' failed with exit status 1

Something’s not right — setuptools is detecting that I’m using macosx-10.3, but I’m using Mac OS X 10.6.  Why is it that setuptools also wants to use /Developer/SDKs/MacOS10.4u.sdk to build python extensions? I’m currently using the SDK for Mac OS X 10.6, and I don’t want to install another SDK.

Well, I did a little bit of research and I learned that PSF’s 2.6.4 python package for a Mac is built with an option called –enable-universalsdk which, according to the readme, defaults to /Developer/SDKs/MacOSX.10.4u.sdk. This is why building third-party extensions tries to reference the 10.4 SDK.

I was able to build python successfully using the following:

./configure --enable-framework --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk/ --with-universal-archs=intel
make && make test
sudo make install

You’ll notice that the following 3 tests failed when attempting to run the unit tests:

• asyncore
• test_platform
• test_macostools

The test that should really put you on alert is asyncore. After doing some research, it turns out the asyncore module is using some variant of select.poll(), which isn’t supported by the FreeBSD kernel. FreeBSD uses something called kqueue, which is what the test doesn’t take into account. To fix this, I pulled the asyncore.py module from the trunk and overwrote /Lib/asyncore.py. The tests passed then.

You don’t need to fix the other two, as they only pertain to fixing the actual tests themselves instead of having to actually change a module. If you’re interested though, fixing “test_platform” follows the same pattern as asyncore. Brett Cannon actually filed a bug for this test and submitted a patch, but you will still need to replace the entire test_platform.py module to get it working. Apparently, this patch is for python v2.7, v3.1, and v3.2. To fix it, download the patched test_platform.py module and replace it with the one in /Lib/test/test_platform.py. Make sure you also delete /Lib/test/test_platform.pyc.

The last test, test_macostools, is actually quite interesting. Apparently, Apple does not supply 64-bit versions of the Carbon frameworks used by these modules. This is why this test is failing. Looks like there might not be a way to fix this test until Apple upgrades the Carbon frameworks to 64-bit usage.

After fixing these bugs, make sure you run the following command:

sudo make install

Needless to say, after I installed python with the options above, and after I fixed all these modules, the Twisted 9.0 installation was successful.

I hope this helps some of you in case you run into this problem!

At work, I’m migrating over python to our 64bit machines and one thing that I’ve noticed was that there really was no standard python 64bit verification method to ensure the build was really 64bit or not. I’ve read somewhere previously, especially for the Mac OS X crowd, that the LDFLAGS=”-arch x86_64″ flag had to be passed in before building on a 64bit machine.

It looks like python2.6 changed the way it was required to build respective 64bit binaries. To build on standard linux x86_64 architecture, the following standard steps to installing on a 64bit machine worked for me:

./configure
make && make test
make install

Surprisingly, I received a segmentation fault when building as well as testing. I’ve never seen this before, but for those of you who are interested, the error message was:

Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
make: *** [Include/graminit.h] Segmentation fault
Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
make: *** [Python/graminit.c] Segmentation fault

The verification step is actually pretty intuitive. An easy test to verify that you’re on a 64bit machine is to find the size of the MAX_INT. Luckily for us, python makes this a very easy verification.

To verify the build, I went on a regular python 32bit machine and I did:

h[1] >>> import sys
h[1] >>> sys.maxint
2147483647

On a 64bit machine, I did:

h[2] >>> import sys
h[2] >>> sys.maxint
9223372036854775807

Clearly, my 64bit installation worked:)

Hope this helps some of you.

And so, we meet again, world. I’ve finally gotten around to registering a home online, installing WordPress, and ready to share my ideas with the world. I’ve given a lot of topics some thought, and I think I might be able to influence and/or help others with my various migrations.

First, I’d like to thank Canonical for Ubuntu and making my migration from Windows to Linux desktop. Some kinks here, and there, but overall I think that it was pretty flawless. I’ve also started a large push towards using vim as my primary editor, instead of constantly switching between IDEs. I believe Netbeans 6.7 came out, I haven’t had the opportunity to play around with it, but if it was anything like Netbeans 6.5, then hey, that’s +1 for them! What a great IDE, especially with the fantastic jVI extension.

I’ll update various posts here and there with some musings about python (what a language), some software releases, mathematical musings, natural language processing tidbits (including really cool algorithms to generate domain names), and various interesting ideas that I’ve had some time to play around with.

A primary reason for starting up this blog is to share with the world some of my thoughts, improve my writing, and try to contribute to the open source world. I think there’s a lot of work to do and I can NOT wait to start. Well world, I’ll hope to speak to you soon.