« Profiles In Courage | Main | (Not) calculating the product of a Python list of for comparison purposes »

June 01, 2011

Selenium's WebDriver & Python: Dealing with script timeouts

Selenium's WebDriver is a great way to interact with web sites in an automated way. It's primarily for testing, though I've also had occasion to use it for other purposes (with the permission of the relevant site owners).

In order to interact with sites that use a lot of Javascript, it's helpful to use Selenium's Firefox driver. It works great, but I did have one problem, where scripts were timing out. You can modify Firefox's timeouts via changing preferences, but Selenium's Firefox driver uses its own profile, so it ignores Firefox's usual means for changing preferences.

I searched the web for solutions and found various suggestions that didn't work. Perhaps many of them worked with pre-WebDriver versions of Selenium, I don't know.

In any case, the following worked for me to change the timeouts to 10 minutes:

from selenium import webdriver

profile = webdriver.firefox \

.firefox_profile.FirefoxProfile()

set_pref = profile.set_preference

set_pref('dom.max_script_run_time', 600)

set_pref('dom.max_chrome_script_run_time', 600)

driver = webdriver.Firefox(fx_profile=profile)

Note that the dom.max_chrome_script_run_time preference has nothing to do with Google Chrome, it's related to Firefox's internal Chrome URL's. In fact, for my purposes, I only needed to change dom.max_chrome_script_run_time.

Note: if you're not familiar with WebDriver, do help(webdriver.Firefox) for a description of the methods you have available for interacting with sites. Hint: the oddest thing is that to fill in a text field, you get an object representing the element with the text field, and then call myElement.send_keys("the text").

Update: Docs on the Python bindings may be found here.

 

June 1, 2011 in Python | Permalink

Comments

Thanks! I didn't know about the help command, and the webdriver syntax is so verbose compared to the IDE syntax (1 line to get the element, 1 line to set it), that I figured I must be doing something terribly wrong.

Posted by: Mike Combs at Jul 11, 2011 4:30:55 PM

Thanks Gary ,

I needed to prevent Selenium firefox profile from downloading images and css on my sites page .
Your tip was really helpful

Posted by: anupam saini at Sep 23, 2011 12:42:21 AM

Thank you so much for this post, I've been googling for a solution to this with Python & WebDrvier all afternoon. I was setting the first preference, but not the chrome one, and this fixed the problem. THANK YOU!

Posted by: Carrie at Sep 19, 2012 9:08:19 PM

Post a comment