I’ve recently had to do some work that required sorting a very large CSV file, containing fields with embedded newlines, quickly. As it turns out, Linux comes with a sort implementation that has a “–zero-terminated” option, which sorts on null-terminated delimited strings instead of the default newline separator.

Writing null-terminated CSV files

Since I was writing a process to generate these CSV files, I figured I can just use Python’s CSV module, which has support for different types of dialects. Inheriting from csv.Dialect, we can write a simple dialect that will allow us to terminate all lines with a null byte.

import csv
import struct

class null_terminated(csv.excel):
    lineterminator = struct.pack('B', 0)

csv.register_dialect("null-terminated", null_terminated)

Essentially, we’ve registered a global csv dialect called "null-terminated" that inherits from the excel dialect, which has sensible standard defaults.

Here’s a simple snippet that shows the usage of the new "null-terminated" dialect that I created above.

from csv import DictWriter

with open("/tmp/file.csv", "w") as f:
	dwriter = DictWriter(f, fieldnames=["id","field"], dialect="null-terminated")

	for i, field in enumerate(("foo", "bar", "baz", "bif")):
    		dwriter.writerow({"id": i, "field": field})

Now, /tmp/file.csv will contain a file with four rows that are separated by a null-terminator. As you can see, it’s pretty easy to write a null-terminated CSV file, but unfortunately, it’s a bit tricky to read a null-terminated csv file due to some inflexible hardcoded defaults.

Reading null-terminated CSV files

The CSV module’s unintuitive restriction for Dialect.lineterminator is hard-coded to recognize '\r' or '\n' as the end of line terminator, which unfortunately, means we will need to handle null-termination and implement reading ourselves.

There are many ways of writing a procedure to read null-terminated strings, but I figured the simplest algorithm is to read character-by-character, concatenating everything into a string until we reach a null byte, then we can just return the string. I’d figure an implementation might go something like this:

def read(fobj):
    current_string = ""
    while True:
        char = fobj.read(1)
        if char and char != nullbyte:
            current_string += char
        elif char == nullbyte:
            yield current_string
            current_string = ""
        elif not char:
            if current_string:
                yield current_string
            raise StopIteration

Looks awesome, but, how can we integrate this into the CSV module? We would want to just plug and play with the existing CSV module. A simple solution is to wrap the function above to iterate over each line, like so:

 # we use StringIO since cStringIO has poor unicode support
from StringIO import StringIO
from csv import reader

class NullTerminatedDelimiterReader(object):
    """
    A CSV reader which will iterate over lines in the CSV file 'f',
    which are line terminated by a null byte

    """

    def __init__(self, f,  dialect, *args, **kwds):
        # satisfying DictReader instance
        self._line_num = 0
        self.fobj = f
        self.dialect = dialect
        self.reader = self._read()
        self.string_io = StringIO()

    def _properly_parse_row(self, current_string):
        self.string_io.write(current_string)
        # seek to the first byte
        self.string_io.seek(0)
        # we instantiate a reader here to properly parse the row
        # taking into account escaping, and various edge cases
        return next(reader(self.string_io, dialect=self.dialect))

    def _read(self):
        current_string = ""
        while True:
            char = self.fobj.read(1)  # read one byte
            if char and char != null_byte:
                # keep appending to the current string
                current_string += char
            elif char == null_byte:
                yield self._properly_parse_row(current_string)
                # increment instrumentation
                self._line_num += 1
                # clear internal reading buffer
                self.string_io.seek(0)
                self.string_io.truncate()
                # clear row
                current_string = ""
            elif not char:
                if current_string:
                    yield self._properly_parse_row(current_string)
                raise StopIteration

    @property
    def line_num(self):
        return self._line_num

    def next(self):
        return next(self.reader)

    def __iter__(self):
        return self

To use the DictReader class, we’ll inherit from the DictReader class and override the reader object. It’s the cleanest and simplest way of doing it.

class NullByteDictReader(csv.DictReader):
    def __init__(self, f, *args, **kwds):
        csv.DictReader.__init__(self, f, *args, **kwds)
        self.reader = NullTerminatedDelimiterReader(f, *args, **kwds)

with open("/tmp/file.csv", "r") as f:
    for line in NullByteDictReader(f, dialect="null-terminated"):
        print line["id"], line["field"]

Voila

Conclusions and Future Work

Something that might be interesting to pursue further is the possibility of writing, or wrapping a python interface around, a C library as a substitute for the current CSV module. It should be able to support different line terminators, multi-byte delimiters, and have unicode detection outside the box, which happen to be my main three gripes with the CSV module.

You can find working source code and implementation of this here and you should follow me on twitter here.

At work, I’m migrating over python to our 64bit machines and one thing that I’ve noticed was that there really was no standard python 64bit verification method to ensure the build was really 64bit or not. I’ve read somewhere previously, especially for the Mac OS X crowd, that the LDFLAGS=”-arch x86_64″ flag had to be passed in before building on a 64bit machine.

It looks like python2.6 changed the way it was required to build respective 64bit binaries. To build on standard linux x86_64 architecture, the following standard steps to installing on a 64bit machine worked for me:

./configure
make && make test
make install

Surprisingly, I received a segmentation fault when building as well as testing. I’ve never seen this before, but for those of you who are interested, the error message was:

Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
make: *** [Include/graminit.h] Segmentation fault
Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
make: *** [Python/graminit.c] Segmentation fault

The verification step is actually pretty intuitive. An easy test to verify that you’re on a 64bit machine is to find the size of the MAX_INT. Luckily for us, python makes this a very easy verification.

To verify the build, I went on a regular python 32bit machine and I did:

h[1] >>> import sys
h[1] >>> sys.maxint
2147483647

On a 64bit machine, I did:

h[2] >>> import sys
h[2] >>> sys.maxint
9223372036854775807

Clearly, my 64bit installation worked:)

Hope this helps some of you.

And so, we meet again, world. I’ve finally gotten around to registering a home online, installing WordPress, and ready to share my ideas with the world. I’ve given a lot of topics some thought, and I think I might be able to influence and/or help others with my various migrations.

First, I’d like to thank Canonical for Ubuntu and making my migration from Windows to Linux desktop. Some kinks here, and there, but overall I think that it was pretty flawless. I’ve also started a large push towards using vim as my primary editor, instead of constantly switching between IDEs. I believe Netbeans 6.7 came out, I haven’t had the opportunity to play around with it, but if it was anything like Netbeans 6.5, then hey, that’s +1 for them! What a great IDE, especially with the fantastic jVI extension.

I’ll update various posts here and there with some musings about python (what a language), some software releases, mathematical musings, natural language processing tidbits (including really cool algorithms to generate domain names), and various interesting ideas that I’ve had some time to play around with.

A primary reason for starting up this blog is to share with the world some of my thoughts, improve my writing, and try to contribute to the open source world. I think there’s a lot of work to do and I can NOT wait to start. Well world, I’ll hope to speak to you soon.