Gary Robinson's Rants

Calculating the product of a Python list

I came across a stackoverflow question about multiplying the numbers in a Python list. Suggestions there included:

def product(list):
    p = 1
    for i in list:
        p *= i
    return p

and

from operator import mul
reduce(mul, list)

You can also do:

import math
math.exp(sum(map(math.log, list)))

It's not as readable as the reduce-based solution, though if you're a mathematician who isn't familiar with reduce() the opposite might be true!

I wouldn't advise using the log-based solution under normal circumstances. But if you're ever in a situation where you risk overflow or overflow, such as in

>>> reduce(mul, [10.]*309)
inf

and your purpose is to compare the products of different sequences rather than to know what the products are, then

>>> sum(map(math.log, [10.]*309))
711.49879373515785

is the way to go because it's virtually impossible to have a real-world problem in which you would overflow or underflow with this approach. (The larger the result of that calculation is, the larger the product would be if you could calculate it.)

Selenium's WebDriver & Python: Dealing with script timeouts

Selenium's WebDriver is a great way to interact with web sites in an automated way. It's primarily for testing, though I've also had occasion to use it for other purposes (with the permission of the relevant site owners).

In order to interact with sites that use a lot of Javascript, it's helpful to use Selenium's Firefox driver. It works great, but I did have one problem, where scripts were timing out. You can modify Firefox's timeouts via changing preferences, but Selenium's Firefox driver uses its own profile, so it ignores Firefox's usual means for changing preferences.

I searched the web for solutions and found various suggestions that didn't work. Perhaps many of them worked with pre-WebDriver versions of Selenium, I don't know.

In any case, the following worked for me to change the timeouts to 10 minutes:

from selenium import webdriver

profile = webdriver.firefox \

.firefox_profile.FirefoxProfile()

set_pref = profile.set_preference

set_pref('dom.max_script_run_time', 600)

set_pref('dom.max_chrome_script_run_time', 600)

driver = webdriver.Firefox(fx_profile=profile)

Note that the dom.max_chrome_script_run_time preference has nothing to do with Google Chrome, it's related to Firefox's internal Chrome URL's. In fact, for my purposes, I only needed to change dom.max_chrome_script_run_time.

Note: if you're not familiar with WebDriver, do help(webdriver.Firefox) for a description of the methods you have available for interacting with sites. Hint: the oddest thing is that to fill in a text field, you get an object representing the element with the text field, and then call myElement.send_keys("the text").

Update: Docs on the Python bindings may be found here.

 

Non-blocking raw_input for Python

[Edited Aug. 30, 2010 to fix a typo in the function name and generally improve formatting]

I needed a way to allow a raw_input() call to time out. In case it's useful to anyone, I wrote this solution which works under Unix-like OS's.

import signal

class AlarmException(Exception):
    pass

def alarmHandler(signum, frame):
    raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20):
    signal.signal(signal.SIGALRM, alarmHandler)
    signal.alarm(timeout)
    try:
        text = raw_input(prompt)
        signal.alarm(0)
        return text
    except AlarmException:
        print '\nPrompt timeout. Continuing...'
    signal.signal(signal.SIGALRM, signal.SIG_IGN)
    return ''

 

 

Splitting a Python list into sublists

[Edited Dec 6, 2010 to mention another solution based on zip and iter.]

Suppose you want to divide a Python list into sublists of approximately equal size. Since the number of desired sublists may not evenly divide the length of the original list, this task is (just) a tad more complicated than one might at first assume.

One Python Cookbook entry is:

def slice_it(li, cols=2):
    start = 0
    for i in xrange(cols):
        stop = start + len(li[i::cols])
        yield li[start:stop]
        start = stop

which gives the exact number of subsequences, while varying the length of the subsequences a bit if necessary. It uses Python's slicing feature to get the lengths.

That was written in response to an earlier cookbook entry which had the following one-liner:

[seq[i:i+size] for i  in range(0, len(seq), size)]

I like that it's a one-liner but don't like a couple of things about it. If your goal isn't a particular sublist length but rather to divide the list up into pieces, you need another line to compute the size. And then it doesn't turn out too well. Suppose you want to divide a string of length 10 into 4 substrings:

>>> size=10/4
>>> size
2
>>> seq = [1,2,3,4,5,6,7,8,9,10]
>>> [seq[i:i+size] for i  in range(0, len(seq), size)]
[[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]

This leaves us with one substring more than desired.

Try setting size to 3 to get fewer substrings:

>>> size=3
>>> [seq[i:i+size] for i  in range(0, len(seq), size)]
[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]] 

This leaves us with dissimilar lengths.

Here's a briefer one-liner using the slice idea, which doesn't require you to compute the length in advance, and does give the exact number of subsequences you want and with lengths that are more appropriately divided:

[seq[i::num] for i in range(num)]

The drawback here is that the subsequences are not actually subsequences of seq; seq is sliced and diced. But, in many situations that doesn't matter. In any case, all the elements are in the output and the subsequences are as close as possible to the same length:

>>> seq = [1,2,3,4,5,6,7,8,9,10]
>>> [seq[i::num] for i in range(num)]
[[1, 5, 9], [2, 6, 10], [3, 7], [4, 8]]

 

Update: I just read about a clever and interesting solution involving zip and tier that works in Python 2.7:

>>> items, chunk = [1,2,3,4,5,6,7,8,9], 3
>>> zip(*[iter(items)]*chunk)
[(1, 2, 3), (4, 5, 6), (7, 8, 9)]

Read a full explanation at Go deh! The disadvantage from a practical point of view is that the if the list of items is not evenly divisible into chunks, some items get left out. But I still like it because it's illuminating about the nuances of iterators.

More cleaning up Python __init__() methods

A while ago I posted a suggestion for how to eliminate some mindless code in python __init__() methods when you want to assign instance attributes from __init__() args.

Henry Crutcher postedhis own solution on Thursday in the Python Cookbook.

I've combined the two approaches in a comment to that post (on the same page). I think it's getting to be a good solution.

Cleaning up Python __init__() methods

The ASPN site yesterday presented a technique to eliminate a lot of verbosity in Python __init__() methods where a bunch of arguments to __init__() are assigned to instance attributes. Such assignments are a frequent source of error due to forgetting one or more assignments and/or making spelling errors for the names.

They presented a function for doing the assignments automatically, but required that the __init__() that uses it has no logic with temporary variables -- an unworkable restriction IMHO.

So I added a comment with a version that works even when there is other logic with temporary variables. If you're a Python coder you might want to check it out.

One line Python Web server

[Most recently updated: Aug 11, 2010.]

if you have python installed, go to your terminal. Go to the directory you want to serve HTML files from, then type one of the following, depending on your python version:

Python 2.x:

python -m SimpleHTTPServer

Python 3.x:

python -m http.server 8000

Either one should start up a web server on port 8000 to serve your files.

[I haven't verified the Python 3 version; it was contributed by duffy in one of the comments. The previous version I had for Python 3 was "python3.0 -c ‘import http.server as h; h.test(HandlerClass=h.SimpleHTTPRequestHandler)’"]

This post has been updated a few times to include improved one-liners pointed out by readers. I've deleted the original text since it's quite obsolete now. That might make some of the comments indecipherable because of the lack of context, but it's all for the greater good. If you have any comments/suggests/observations, please speak up! 

Python Singleton mixin class

Here's a quite complete Singleton mixin class for Python, including such niceties as thread safety for singleton creation. Public domain. Based in part on ideas I found on this wiki page, followed by a substantial number of enhancements based on comments posted below and based on my own additional needs. If you find it useful and think of ways to improve it, please post a comment here.

Lately I've noticed some discussion of the evilness of singletons. One poster says:

It's almost impossible to subclass a Singleton, and if you manage it, then you shouldn't have been using a Singleton in the first place. You don't *even* want to go there. I've walked roads that I dare not recount. Just pretend you can't do it, and you'll save yourself amazing amounts of pain.

My singleton mixin source includes unit tests showing that at least rudimentary subclassing does work with the approach used here. I haven't run into any problems in real life either, although I use singletons sparingly. 

If anyone has examples or thoughts about problems with subclassing with this mixin, please post a comment.

Update, July 28, 2009: Now optionally allows multiple calls to getInstance() which all include arguments. See the docstring for more. Update, July 27, 2009: I try to keep the Singleton version here in sync with our svn version, so that any enhancements or fixes will be included. The code, still at the original link (above), has been modified today to add thread safety for singleton creation, and also incorporates ideas from some of the recent comments people have posted here. In particular, singleton construction can now handle keyword arguments.