Python Code Optimization Tips

The other day I came across an interesting article, written by the creator of Python, Guido van Rossum. A friend had asked him what would be the best way to convert a list of integers (representing ASCII characters) into a string. In the article, Rossum analysed several different ways to tackle this simple problem. I learned several general tips to keep in mind for when speed and performance is important. This entry demonstrates what I learned from the article, and illustrates some test results as well.

One: Focus on bottlenecks.

Code optimization is fun and interesting. It can be tempting to nitpick every line of code in an effort to gain performance increases. It is important, however, to keep in mind that real performance improvement is found only at the bottlenecks. This is obviously not Python-specific. In real-world situations, your efficiency as a developer is often just as valuable as the efficient code that you produce. Only optimize code where a proven speed bottleneck exists. Doing so will save a lot of work and leave you free to tackle more problems. That being said, this tip is not an excuse to be lazy or to write poor code.

Two: Local variables are faster than globals.

In Python, function names are considered global constants. That means if you are going to use a built-in function inside a loop, for example, it’s possible to save some overhead by assigning it to a local variable before the loop. Rossum points out that local variables are faster than global variables. Here’s an example to demonstrate:

def option1(l):
    string = ""
    _chr = chr
    for i in l:
        string = string + _chr(i)
    return string

def option2(l):
    string = ""
    for i in l:
        string = string + chr(i)
    return string

Option1 stores the global built-in function chr in a local variable, which saves about one second of execution time. Below are the results of executing these pieces of code 100,000 times with a list of 1024 digits.

Option 1: 11.59 seconds Option 2: 12.74 seconds

These results may not seem like much, but applied to some situations, the use of this simple optimization could help drastically improve performance.

Three: Use functions implemented in C when possible.

Rossum points out that you can’t really get any faster than C. Choose functions that are implemented in C whenever possible for massive speed-up’s. This means that it’s well worth your time to check the Python documentation for a built-in function that does what you want. Rossum demonstrated the use of a string concatenation function from the string module, which is written in C. I have written the example below to show just how much faster using C code can be.

import string

def option4(l):
    return string.joinfields(map(chr, l), "")

Notice how the code executed around 4 seconds faster than the other two versions:

Option 1: 11.49 seconds Option 2: 12.64 seconds Option 4: 7.79 seconds

What’s even more impressive is this one-line solution using the array module, which is also implemented in C. The array module is for handling sequences of fixed-type numerical data efficiently. A normal Python list is handy because it does not care what type of data you put inside of it, but it’s convenience comes at a performance cost. The Array module only lets you create a sequence of data that is all the same type, which saves lots of overhead and space. The following code completes execution in only 3.76 seconds:

import array

def option5(l):
    return array.array('B', l).tostring()

Conclusion

It’s amazing how small details have potential to make a big impact on code performance. Taking the time to profile your code will reveal performance bottlenecks. It’s important to be aware of what’s going on behind the scenes in the code that you write.

Here are the results of each function:

Option 1: 11.52 seconds Option 2: 12.65 seconds

Option 4: 7.79 seconds Option 5: 3.76 seconds

Check out the article that I have been referencing for more tips and performance results. You can find a link in the introduction, or the URL in the references section below.