Gary Robinson's Rants

Python one-liner for counting occurrences of each value in a sequence

Suppose we have a sequence [2,1,1,3,4,4,4,4,4,5,5,2,2,2] and we want to know how many of each value there are, in an easy-to-use dictionary form: {1: 2, 2: 4, 3: 1, 4: 5, 5: 2}.

The simplest way is to use Counter from the collections standard library:

>>> from collections import Counter
>>> Counter([2,1,1,3,4,4,4,4,4,5,5,2,2,2])
Counter({4: 5, 2: 4, 1: 2, 5: 2, 3: 1})

A Counter is dict subclass so, once it's created, you can use it like a dictionary. You can also get a regular dictionary from it with

dict(Counter([2,1,1,3,4,4,4,4,4,5,5,2,2,2]))

[Hat tip to reader qsantos.]

I like the itertools and more-itertools[1] libraries because they contain many quick solutions for working with iterables. itertools is part of the Python standard library; more-itertools isn't, but it's good to look both of them over to know what's available there. For example, the task of counting values can be done using the run_length() function in more-itertools[1], although that method is a little more verbose than the one using Counter:

>>> from more_itertools import run_length
>>> dict(run_length.encode(sorted([2,1,1,3,4,4,4,4,4,5,5,2,2,2])))
{1: 2, 2: 4, 3: 1, 4: 5, 5: 2}

[1] https://pypi.org/project/more-itertools/

A thought on Rust and LLMs.

I wrote a terminal-based utility in Python that I'm thinking of releasing publicly (I've been using it myself for years). But the footprint seems very big for a terminal utility. So I've been looking at converting it to a compiled language. One with memory management. I like F#, but its executable footprint is surprisingly large.

I played with OCaml, and I basically like the language, but its tooling is not great and I'm finding that ChatGPT isn't good at writing OCaml code particularly when using libraries, because there just isn't much code out there using most OCaml ecosystem libraries.

Which finally led me to look at Rust. It has features I want like match statements, and even has macros that do pipelining. Obviously, it's usually considered a systems programming language, although there are things like web frameworks.

One might ask, however: For applications where efficiency isn't a major issue, why would you use a language where you have to manually deal with things like memory management, meaning that it will take significantly more code to accomplish the same thing as you could in a non-systems language?

An answer I'm considering: There is a lot of Rust code out there for training LLMs. My guess is that to the degree that a developer is willing to invoke an LLM to write Rust code, it can handle the memory management pretty much painlessly.

I'm not advocating turning programming tasks entirely over to LLMs. We should still read and understand any LLM-generated code we use, carefully consider the structure and tools it uses, etc..

But what I'm suspecting is that a language like Rust may get more use in the future. The time expenditure that goes with its programming overhead is greatly reduced when LLMs are used to generate code. Maybe there will be less traction going to languages whose main advantage is that they require fewer lines of code for the same results when those lines are often written by an LLM.

So I make a tentative prediction: LLM use will cause the Rust community to grow even more quickly than has already been the case.

Anyway, I'm going to be moving from Python to Rust, making substantial use of LLMs, and see how it feels.

Javascript and SPA's, as seen by a long-time Python developer

I love Python and have posted a number of Python tips on this blog. And I've been involved in the creation of sizable web sites using Django.

But I've come to feel that for many types of web sites, doing all the rendering on the server and shipping the rendered page to the browser is probably not be the best way to go. Certainly, I'm not the first to come to that conclusion. Google Docs is a phenomenal example of what can be done in a "single page app" (SPA) rendered in the browser. There are many others.

In the course of investigating the various technologies for creating SPA's, one thing I've come to appreciate about that strategy is that you may not need much of a server at all. If you use something like Firebase for your database, you may need nothing more than a means of serving static html and JavaScript files on your server. And then you can use a CDN like CloudFlare to keep you online even if that server is temporarily down. All these factors together can eliminate an enormous amount of overhead in server administration.

Eliminating such overhead seems like it could be very helpful for my goal of creating my next project entirely by myself.

Unfortunately, JavaScript is (IMO) not nearly as nice a language as Python. But if used according to certain patterns, such as described in the famous JavaScript: The Good Parts, a lot of its deficiencies are mitigated, and then it's really not so bad. And most modern JavaScript libraries use the language in that way, so the ecosystem as a whole supports you in that.

Javascript even has Python-like constructions such as list comprehensions. And there are other languages that compile to JavaScript and can be fairly easily integrated into JavaScript projects, such as CoffeeScript -- which itself is Python-like in a number of ways, including semantic indentation. 

You can put together tools such as Firebase, Knockout and KnockoutFire to cause changes in your database to automatically and near-instantaneously show up on-screen in your SPA with trivially little code through data-binding. Of course, there are ways of doing that with a framework like Django as well, but data-binding is integral to the way some SPA frameworks operate.

Lately I've been experimenting with Durandal as my SPA framework. It incorporates Knockout, and I'm using KnockoutFire to connect it to a Firebase database. Using a framework like Durandal provides organization for your code, and provides facilities like routing.

So far, I'm very impressed. The main drawback (other than already-noted inherent weaknesses in Javascript itself) is that the documentation and ecosystems of long-existing, high-profile projects like Django are much more evolved. But those things will get better in time, and I'm looking forward to continuing my explorations with Durandal and Firebase.

(Not) calculating the product of a Python list of for comparison purposes

I came across a stackoverflow question about multiplying the numbers in a Python list. Suggestions there included:

def product(list):
    p =1
    for i in list:
        p *= i
    return p

and

from operator import mul
reduce(mul, list)

Adding my own $.02, you can also do:

import math
math.exp(sum(map(math.log, list)))

which gets the same numerical results (at least to as way more decimal places than you're ever likely to care about, the difference being due to computational floating point errors). It's probably not as readable as the reduce-based solution, though if you're a mathematician who isn't familiar with reduce(), the opposite would be true!

(The underlying principle is that a * b * c * … * z == e ^ (ln(a) + ln(b) + ln( c) + … + ln(z)).)

I wouldn't necessarily advise using the log-based solution under normal circumstances. But if you're ever in a situation where you risk overflow or overflow, such as in

>>> reduce(mul, [10.] * 309)
inf

and your purpose is to compare the products of different sequences rather than to know specifically what the products are, then, you can accomplish your purposes without actually computing the entire product. Rather, you can compare the sums of the logs of the terms, without bothering to compute the product by taking the exponential. Whichever list has a larger sum-of-logs also has the larger product. So, following up on our previous example,

>>> sum(map(math.log, [10.] * 309))
711.49879373515785

is the way to go because it's virtually impossible to have a real-world problem in which you would overflow or underflow with this approach. (The larger the result of that calculation is, the larger the product would be if you could calculate it.)

Accordingly, if you wanted to compare 10. ^ 309 and 11. ^ 309, you could do

>>> sum(map(math.log, [10.] * 309)) > sum(map(math.log, [11.] * 309))
False

[Originally posted as "Calculating the product of a Python list. Updated for clarity and completeness.]

 

 

Selenium's WebDriver & Python: Dealing with script timeouts

Selenium's WebDriver is a great way to interact with web sites in an automated way. It's primarily for testing, though I've also had occasion to use it for other purposes (with the permission of the relevant site owners).

In order to interact with sites that use a lot of Javascript, it's helpful to use Selenium's Firefox driver. It works great, but I did have one problem, where scripts were timing out. You can modify Firefox's timeouts via changing preferences, but Selenium's Firefox driver uses its own profile, so it ignores Firefox's usual means for changing preferences.

I searched the web for solutions and found various suggestions that didn't work. Perhaps many of them worked with pre-WebDriver versions of Selenium, I don't know.

In any case, the following worked for me to change the timeouts to 10 minutes:

from selenium import webdriver

profile = webdriver.firefox \

.firefox_profile.FirefoxProfile()

set_pref = profile.set_preference

set_pref('dom.max_script_run_time', 600)

set_pref('dom.max_chrome_script_run_time', 600)

driver = webdriver.Firefox(fx_profile=profile)

Note that the dom.max_chrome_script_run_time preference has nothing to do with Google Chrome, it's related to Firefox's internal Chrome URL's. In fact, for my purposes, I only needed to change dom.max_chrome_script_run_time.

Note: if you're not familiar with WebDriver, do help(webdriver.Firefox) for a description of the methods you have available for interacting with sites. Hint: the oddest thing is that to fill in a text field, you get an object representing the element with the text field, and then call myElement.send_keys("the text").

Update: Docs on the Python bindings may be found here.

 

Non-blocking raw_input for Python

I needed a way to allow a raw_input() call to time out. In case it's useful to anyone, I wrote this solution which works under Unix-like OS's.

Simply call nonBlockingRawInput("your prompt text", n) where n is the number of seconds you would like the prompt to wait until it times out. If it does time out, it will return with an empty string.

import signal

class AlarmException(Exception):
    pass

def alarmHandler(signum, frame):
    raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20):
    signal.signal(signal.SIGALRM, alarmHandler)
    signal.alarm(timeout)
    try:
        text = raw_input(prompt)
        signal.alarm(0)
        return text
    except AlarmException:
        print '\nPrompt timeout. Continuing...'
    signal.signal(signal.SIGALRM, signal.SIG_IGN)
    return ''

[Edited Aug. 30, 2010 to fix a typo in the function name and generally improve formatting]

[Edited Feb 1, 2022 to add a tad more explanation, since the comments indicate people are still finding this!]

Splitting a Python list into sublists

[Edited Dec 6, 2010 to mention another solution based on zip and iter.]

Suppose you want to divide a Python list into sublists of approximately equal size. Since the number of desired sublists may not evenly divide the length of the original list, this task is (just) a tad more complicated than one might at first assume.

One Python Cookbook entry is:

def slice_it(li, cols=2):
    start = 0
    for i in xrange(cols):
        stop = start + len(li[i::cols])
        yield li[start:stop]
        start = stop

which gives the exact number of subsequences, while varying the length of the subsequences a bit if necessary. It uses Python's slicing feature to get the lengths.

That was written in response to an earlier cookbook entry which had the following one-liner:

[seq[i:i+size] for i  in range(0, len(seq), size)]

I like that it's a one-liner but don't like a couple of things about it. If your goal isn't a particular sublist length but rather to divide the list up into pieces, you need another line to compute the size. And then it doesn't turn out too well. Suppose you want to divide a string of length 10 into 4 substrings:

>>> size=10/4
>>> size
2
>>> seq = [1,2,3,4,5,6,7,8,9,10]
>>> [seq[i:i+size] for i  in range(0, len(seq), size)]
[[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]

This leaves us with one substring more than desired.

Try setting size to 3 to get fewer substrings:

>>> size=3
>>> [seq[i:i+size] for i  in range(0, len(seq), size)]
[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]] 

This leaves us with dissimilar lengths.

Here's a briefer one-liner using the slice idea, which doesn't require you to compute the length in advance, and does give the exact number of subsequences you want and with lengths that are more appropriately divided:

[seq[i::num] for i in range(num)]

The drawback here is that the subsequences are not actually subsequences of seq; seq is sliced and diced. But, in many situations that doesn't matter. In any case, all the elements are in the output and the subsequences are as close as possible to the same length:

>>> seq = [1,2,3,4,5,6,7,8,9,10]
>>> [seq[i::num] for i in range(num)]
[[1, 5, 9], [2, 6, 10], [3, 7], [4, 8]]

 

Update: I just read about a clever and interesting solution involving zip and tier that works in Python 2.7:

>>> items, chunk = [1,2,3,4,5,6,7,8,9], 3
>>> zip(*[iter(items)]*chunk)
[(1, 2, 3), (4, 5, 6), (7, 8, 9)]

Read a full explanation at Go deh! The disadvantage from a practical point of view is that the if the list of items is not evenly divisible into chunks, some items get left out. But I still like it because it's illuminating about the nuances of iterators.

More cleaning up Python __init__() methods

A while ago I posted a suggestion for how to eliminate some mindless code in python __init__() methods when you want to assign instance attributes from __init__() args.

Henry Crutcher postedhis own solution on Thursday in the Python Cookbook.

I've combined the two approaches in a comment to that post (on the same page). I think it's getting to be a good solution.

Cleaning up Python __init__() methods

The ASPN site yesterday presented a technique to eliminate a lot of verbosity in Python __init__() methods where a bunch of arguments to __init__() are assigned to instance attributes. Such assignments are a frequent source of error due to forgetting one or more assignments and/or making spelling errors for the names.

They presented a function for doing the assignments automatically, but required that the __init__() that uses it has no logic with temporary variables -- an unworkable restriction IMHO.

So I added a comment with a version that works even when there is other logic with temporary variables. If you're a Python coder you might want to check it out.

One line Python HTTP server

[Most recently updated: Oct 29, 2013.]

One of the sweet things that comes with a Python installation is the ability to start up an http server in just one line. Since just about every modern computer (not counting tablets and phones!) has Python, you almost certainly can start up an http server to serve some files in a chosen directory with one quick line of code.

Go to your terminal. Go to the directory you want to serve HTML files from, then type one of the following, depending on your python version:

Python 2.x:

python -m SimpleHTTPServer

Python 3.x:

python -m http.server

Note that the output on your terminal will say:

Serving HTTP on 0.0.0.0 port 8000 ...

The mention of 0.0.0.0 does not mean the web server is listening at that IP address. Rather, it will be listening at 127.0.0.1:8000. The 0.0.0.0 signifies that the web server will respond to requests that end up on that machine no matter the hostname or IP that was requested.

Note that you can start your web server at port nnnn with: 

Python 2.x:

python -m SimpleHTTPServer nnnn

Python 3.x:

python -m http.server nnnn

Of course, starting an http server can be made even more convenient by an alias in your .bashrc (or the equivalent for your OS/shell). A commenter noted that he did so, so I thought I might as well add it to the main post:

 

alias http="python -m SimpleHTTPServer"

which lets you start a server on port nnnn for the current directory with:

http nnnn

 

This post has been updated a few times to include improved one-liners pointed out by readers as well as for clarity and completeness. I've deleted the original text since it's quite obsolete now. That might make some of the comments indecipherable because of the lack of context, but it's all for the greater good. If you have any comments/suggests/observations, please speak up!  

Python Singleton mixin class

Here's a quite complete Singleton mixin class for Python, including such niceties as thread safety for singleton creation. Public domain. Based in part on ideas I found on this wiki page, followed by a substantial number of enhancements based on comments posted below and based on my own additional needs. If you find it useful and think of ways to improve it, please post a comment here.

Lately I've noticed some discussion of the evilness of singletons. One poster says:

It's almost impossible to subclass a Singleton, and if you manage it, then you shouldn't have been using a Singleton in the first place. You don't *even* want to go there. I've walked roads that I dare not recount. Just pretend you can't do it, and you'll save yourself amazing amounts of pain.

My singleton mixin source includes unit tests showing that at least rudimentary subclassing does work with the approach used here. I haven't run into any problems in real life either, although I use singletons sparingly. 

If anyone has examples or thoughts about problems with subclassing with this mixin, please post a comment.

Update, July 28, 2009: Now optionally allows multiple calls to getInstance() which all include arguments. See the docstring for more. Update, July 27, 2009: I try to keep the Singleton version here in sync with our svn version, so that any enhancements or fixes will be included. The code, still at the original link (above), has been modified today to add thread safety for singleton creation, and also incorporates ideas from some of the recent comments people have posted here. In particular, singleton construction can now handle keyword arguments.