Modern Web Automation with Python and Selenium

Headless Selenium Testing with Python and PhantomJS

PhantomJS is a headless Webkit, which has a number of uses. In this example, we’ll be using it, in conjunction with Selenium WebDriver, for conducting basic system tests directly from the command line. Since PhantomJS eliminates the need for a graphical browser, tests run much faster.

Click here to watch the accompanying video.

Setup

Install Selenium with Pip and PhantomJS with Homebrew:

Shell
$ pip install selenium
$ brew install phantomjs

Having trouble installing PhantomJS with Brew? Grab the latest build here.

Examples

Now let’s look at two quick examples.

DuckDuckGo

In the first example, we’re just going to search DuckDuckGo for the keyword “realpython” to find the URL of the search results.

Python
from selenium import webdriver
driver = webdriver.PhantomJS()
driver.set_window_size(1120, 550)
driver.get("https://duckduckgo.com/")
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
print(driver.current_url)
driver.quit()

You can see the outputted URL in the terminal.

Here’s a look at the same thing using Firefox to display the results.

Python
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://duckduckgo.com/")
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
driver.quit()

Did you notice how we had to create a dummy browser size on the Phantom script? This is a workaround to a bug that’s currently an issue in Github. Try the script without it: You’ll get an ElementNotVisibleException exception.

Now we can write a quick test to assert that the URL brought up by the search results is correct.

Python
import unittest
from selenium import webdriver


class TestOne(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.PhantomJS()
        self.driver.set_window_size(1120, 550)

    def test_url(self):
        self.driver.get("http://duckduckgo.com/")
        self.driver.find_element_by_id(
            'search_form_input_homepage').send_keys("realpython")
        self.driver.find_element_by_id("search_button_homepage").click()
        self.assertIn(
            "https://duckduckgo.com/?q=realpython", self.driver.current_url
        )

    def tearDown(self):
        self.driver.quit()

if __name__ == '__main__':
    unittest.main()

The test passed.

RealPython.com

Finally, let’s look at a real world example that I run daily. Navigate to RealPython.com and I’ll show you what we’ll be testing. Essentially, I want to ensure that the bottom “Download Now” button has the correct product associated with it.

Here’s a look at the basic unittest:

Python
import unittest
from selenium import webdriver


class TestTwo(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.PhantomJS()

    def test_url(self):
        self.driver.get("https://app.simplegoods.co/i/IQCZADOY") # url associated with button click
        button = self.driver.find_element_by_id("payment-submit").get_attribute("value")
        self.assertEquals(u'Pay - $60.00', button)

    def tearDown(self):
        self.driver.quit()

if __name__ == '__main__':
    unittest.main()

Benchmarking

One main advantage of using PhantomJS over a browser is that tests are usually much faster. In this next example, we’ll benchmark the previous test using both PhantomJS and Firefox.

Python
import unittest
from selenium import webdriver
import time


class TestThree(unittest.TestCase):

    def setUp(self):
        self.startTime = time.time()

    def test_url_fire(self):
        time.sleep(2)
        self.driver = webdriver.Firefox()
        self.driver.get("https://app.simplegoods.co/i/IQCZADOY") # url associated with button click
        button = self.driver.find_element_by_id("payment-submit").get_attribute("value")
        self.assertEquals(u'Pay - $60.00', button)

    def test_url_phantom(self):
        time.sleep(1)
        self.driver = webdriver.PhantomJS()
        self.driver.get("https://app.simplegoods.co/i/IQCZADOY") # url associated with button click
        button = self.driver.find_element_by_id("payment-submit").get_attribute("value")
        self.assertEquals(u'Pay - $60.00', button)

    def tearDown(self):
        t = time.time() - self.startTime
        print("%s: %.3f" % (self.id(), t))
        self.driver.quit()

if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestThree)
    unittest.TextTestRunner(verbosity=0).run(suite)

You can see just how much faster PhantomJS is:

Shell
$ python test.py -v
__main__.TestThree.test_url_fire: 19.801
__main__.TestThree.test_url_phantom: 10.676
----------------------------------------------------------------------
Ran 2 tests in 30.683s

OK

Video

Note: Although the video is outdated (due to changes in the scripts), it’s still worth watching as the basic way to implement headless system tests for your websites has remained largely the same.

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

Python Tricks Dictionary Merge

Master Real-World Python Skills With Unlimited Access to Real Python

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

Master Real-World Python Skills
With Unlimited Access to Real Python

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

What Do You Think?

Rate this article:

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal.


Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session. Happy Pythoning!

Keep Learning

Related Tutorial Categories: front-end intermediate testing web-dev web-scraping