In the past, people have hassled me to make it easier for them to install my applications and libraries on OS X. I’m not very skilled with OS X, and searching for how to do this task took some time. I’ve made a script that does this in a modular way for Python. Specifically, this article describes the process of making a Python module or extension installer for OS X, similar to the Python installer for Windows.

I use a Python module called bdist_mpkg which does most of the work. It simply creates a “mpkg” file out of your distribution by reading your standard distutils setup.py file. You can run it using either the command “bdist_mpkg” or as part of running setup.py, i.e. “python setup.py bdist_mpkg”. They both seem to do the same thing. The generated file is an OS X metapackage used by the OS X installer, “Installer.app”.

Once you have the mpkg file, which is generated in the directory “dist”, the script puts it in a “dmg” file. Putting a mpkg into a dmg file seems to be the standard way of distributing applications on OS X, yet I haven’t found out exactly why–someone point me in the right direction!

The script uses hdiutil to create a new, empty disk image then attaches (mounts) it to your system as a “device”. The attach command outputs the location of the path and device in the file system, which the script saves for detaching. Now that the image is mounted, the mpkg file is put into the image. The script then detaches (unmounts) then compresses the dmg file.

Unfortunately I didn’t save the references I used to find out the code to do this so I haven’t given credit, even though this is probably straightforward to an experienced OS X Python developer. The script, which I’ve called “python-osx-dist.sh”, accepts a package name and a version number as arguments. The script source is here and downloadable here, and am happy to get comments or patches for it.

from pygooglechart import QRChart

# Create a 125x125 QR code chart
chart = QRChart(125, 125)

# Add the text
chart.add_data('Hello, World!')

# "Level H" error correction with a 0 pixel margin
chart.set_ec('H', 0)

# Download
chart.download('qr-hello.png')

This will download a png from the URL:

http://chart.apis.google.com/chart?cht=qr&chs=125×125&chl=Hello%2C%20World%21&chld=H|0

The image downloaded looks like this:

QR codes are new to me. I’ve only seen a QR code on a bus stop advertisement once in Sydney but I’m sure more will pop up.

This heat map has about 100 samples in it, which are shown as white dots. There is a balcony at the top of the image and the common office area on the right, which was drawn by hand (hence the waviness). As you can see it has pinpointed the location of the access point in the red area.

Below is another heat map with the same points but on a different access point.

Even though I don’t have points of data on the neighbour’s office area, the algorithm has worked out the approximate area of the access point.

The application (which is tentatively named “wirelessheatmap”) is written in Python with the OpenGL library, pyglet. It relies on a wireless packet sniffer called airodump-ng from the Aircrack-ng suite of wireless tools. I also had to patch airodump-ng so that it writes the (later explained) CSV (comma-separated values) file more often.

After setting my Wi-Fi device to “monitor” mode, I run airodump-ng, listening on all channels. It writes a CSV file containing the signal strength of all access points from where my laptop is. With the patched version it updates this file every second. While airodump-ng is running, I also run the Python application with an image overlay containing the floor plan for reference.

To map out the points, I simply go to a location, wait a second or two for the signal strength to stabilize and for the CSV file to be updated. In the application, I click on my location on the floor plan map, then the application reads the CSV file along with all of the access point information, then saves the data for the location. I continue collecting points until I have a nice supply of sample points. This creates a mapping of 2D points to signal strength data for every access point.

The way the application generates the heat map is fairly straightforward. It allocates a 2D grid which overlaps the sample area. For each grid position it works out the nearest 10 sample points (or another tweakable number depending on your data), then works out the average signal strength of all the points. I went through several iterations of different methods and this turned out to be the most accurate and best for projection access point locations.

If you want to run this application yourself, the source code is available but is not user friendly, buggy and may require you to know some Python to work out. To start you off run “./heat –help” to see the options. “–image” is the image file you want as the overlay reference map. “–dump” is the airodump-ng CSV file. “–store” is the file that contains the mapping between sample points and access point information.

I may continue this project in the future if there is enough demand for it, but for now it will stay as is.

Prerequisites

You’ll need to be able to compile C files for this tutorial to work. You also need Cython installed and the Python development files installed. This is what I did on my “Gutsy Ubuntu” to get it going:

sudo apt-get install python-dev
sudo easy_install Cython

I can attempt to help you if you can’t get things to compile, as long as you check Google for your errors first!

Python module

We’ll create a Python module to work out prime numbers up to a certain number. It’s not the most efficient method, but that’s not important in this tutorial. Lets call it prime.py.

def calculate(limit):
    primes = []
    divisor = 0
    for current in xrange(limit):
        previous = []
        for divisor in xrange(2, current):
            if current % divisor == 0:
                break
        if divisor == current - 1:
            primes.append(current)
    return primes

Add another program to import this module and use it. Let us say test_prime.py.

import prime

numbers = prime.calculate(10000)
print len(numbers)

Running “python test_prime.py” will return the number of prime numbers from 0 to 10,000, which is 1229. Lets add some timing code to see how fast this runs.

from timeit import Timer

t = Timer('prime.calculate(10000)', 'import prime')
reps = 5
print sum(t.repeat(repeat=reps, number=1)) / reps

This code above runs “prime.calculate” 5 times and averages out the time, so we get a more accurate result. On this old computer, it runs at 1.46 seconds. For the curious, this code under Psyco runs at 0.12 seconds.

Converting to a Cython extension

So far we have test_prime.py which calls the prime.py module. To convert it to Cython we need to create a setup.py file and make a copy of prime.py to c_prime.pyx. The pyx extension is specific to Pyrex and Cython. Here is our setup.py file:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
  name = 'PrimeTest',
  ext_modules=[
    Extension('c_prime', ['c_prime.pyx'])
    ],
  cmdclass = {'build_ext': build_ext}
)

The ext_modules list tells distutils that we’re building an extension module called c_prime from c_prime.pyx. The cmdclass argument mentions that we’re using a third-party tool (Cython) to build the extension.

Running the following will hopefully build your extension. “build_ext” tells distutils that we’re building an extension and the “–inplace” argument means it will put the built extension in the current directory.

python setup.py build_ext --inplace

This is the step that fails the most since it depends on gcc, Cython, Python development code and possibly other things. Try to find your problem on Google, otherwise ask here if you have no luck.

Now let us modify test_prime.py so that it is using our extension. Change the use of prime to c_prime on line 3:

t = Timer('c_prime.calculate(10000)', 'import c_prime')

Right now we have a C extension generated from our Python module. The time it takes to run test_prime.py now takes 0.79 seconds–almost double in speed. And that’s without any modification to our Python code!

More speed

If you know a bit of C, you can check out the generated C code in c_prime.c. It is quite ugly and has about 670 lines. Looking closer at the C code, you can see it demonstrating (within comment blocks) which line of code it has converted from with the translated code below. This is handy for optimisation to work out ways of increasing speed.

Looking at line 4 of our c_prime.pyx file, you can see this is simple a for-loop to count from 0 to a variable. In the C code it is about 20 lines as you can see from this excerpt!

/* "/home/gak/src/muckaround/cython-getting-started-tutorial/c_prime.pyx":4
 *     primes = []
 *     divisor = 0
 *     for current in xrange(limit):             # <<<<<<<<<<<<<<
 *         previous = []
 *         for divisor in xrange(2, current):
 */
  __pyx_1 = PyTuple_New(1); if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;}
  Py_INCREF(__pyx_v_limit);
  PyTuple_SET_ITEM(__pyx_1, 0, __pyx_v_limit);
  __pyx_3 = PyObject_Call(__pyx_builtin_xrange, ((PyObject *)__pyx_1), NULL); if (unlikely(!__pyx_3)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;}
  Py_DECREF(((PyObject *)__pyx_1)); __pyx_1 = 0;
  if (PyList_CheckExact(__pyx_3)) { __pyx_2 = 0; __pyx_1 = __pyx_3; Py_INCREF(__pyx_1); }
  else { __pyx_1 = PyObject_GetIter(__pyx_3); if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} }
  Py_DECREF(__pyx_3); __pyx_3 = 0;
  for (;;) {
    if (PyList_CheckExact(__pyx_1)) { if (__pyx_2 >= PyList_GET_SIZE(__pyx_1)) break; __pyx_3 = PyList_GET_ITEM(__pyx_1, __pyx_2++); Py_INCREF(__pyx_3); }
    else {
      __pyx_3 = PyIter_Next(__pyx_1);
      if (!__pyx_3) {
        break;
      }
    }
    Py_DECREF(__pyx_v_current);
    __pyx_v_current = __pyx_3;
    __pyx_3 = 0;

The reason why this is 20 lines is that it is basically doing the loop via Python. You might be able to make out that the C code is calling xrange in Python, then accessing its iterator until there are no more items in the xrange iterator. It is also converting Python objects to long integers. This is ridiculous for C code since it can be accomplished in one line.

We can tell Cython to make it only generate one line of C code by using its extended syntax. The syntax is:

for i from 0 <= i < n:

We have two of these for-loops, so lets change them both:

def calculate(limit):
    primes = []
    divisor = 0
    for current from 0 <= current < limit:
        previous = []
        for divisor from 2 <= divisor < current:
            if current % divisor == 0:
                break
        if divisor == current - 1:
            primes.append(current)
    return primes

This is only slightly faster and the generated code for that for-loop is 5 lines now. We still want a one line C style for-loop. The reason it is 5 lines in the is that the variable “current” is still a Python object and the C code has to convert between a Python object and a C long integer. We need to tell Cython to use C variable types instead of Python objects. You do this using the “cdef” keyword. Lets do this and while we’re here, we’ll change all the numbers to long integers:

def calculate(long limit):
    cdef long current
    cdef long divisor
    primes = []
    divisor = 0
    for current from 0 <= current < limit:
        previous = []
        for divisor from 2 <= divisor < current:
            if current % divisor == 0:
                break
        if divisor == current - 1:
            primes.append(current)
    return primes

As you can see I have changed the “limit” argument to say “long limit” which tells Cython to convert the limit argument from a Python object to a long integer when it is called. The “current” and “divisor” variables are now defined at the start of the function preceded with “cdef long”, telling Cython that they are long integers in the C code.

Now looking at that for-loop in the generated C code, it is now one line as we wanted:

for (__pyx_v_current = 0; __pyx_v_current < __pyx_v_limit; __pyx_v_current++) {

This ends up running at an incredible 0.05 seconds. That is 15x faster than our original Python module.

I hope this tutorial has been useful to you!

UPDATE: It turns out that changing all the variables in the xrange to cdefs will cause the generated C code to be one-liner for-loop. This means the alternate syntax “for i from 0 <= i < n” is not necessary when you’re only working with cdef’ed variables.

The plugin uses the reCAPTCHA service and is serving me well so far. Here’s how it looks within the template:

This is my first publicly released Trac plugin so I’m unsure how it’ll perform out there, so feedback is welcome.

chart = SimpleLineChart(width, height, x_range=(0, 100), y_range=(0, 100))

Other features are the new API chart types and options. There are 3 new chart types: Map, Google-o-Meter and Radar. Here is an example of using the Map chart type:

import pygooglechart as gc

chart = gc.MapChart(440, 220)
chart.set_colours(('AAAAAA', '30A030', 'A0C030'))
chart.set_codes(['AU', 'US', 'CA', 'BR', 'NZ'])
chart.add_data([1, 0.5, 0.7, 0.3, 0.1])
chart.download('map.png')

Produces the URL:
http://chart.apis.google.com/chart?cht=t&chs=440×220&chd=s:9frSG&chco=AAAAAA,30A030,A0C030&chtm=world&chld=AUUSCABRNZ

And the image:

The Google-o-meter has a funny name. Here is an example of its usage:

import pygooglechart as gc

chart = gc.GoogleOMeterChart(440, 220, y_range=(0, 10))
chart.set_pie_labels(['Awesome'])
chart.add_data([8])
chart.download('gom.png')

It produces this URL:
http://chart.apis.google.com/chart?cht=gom&chs=440×220&chd=s:x&chl=Awesome

And the image:

Well, have fun. Please write a ticket for any bugs, patches or feature requests you have.

• PyGoogleChart 0.1.0 - Initial release of a Python wrapper for the Google Chart API.
• Python Call Graph 0.4.1 - Speed improvements for the visual call graph Python library.