A new web-testing paradigm: Robot Framework & Page Objects

Though I am not a QA engineer, a year ago frishberghellmanj and I took it upon ourselves to help NCBI’s QA department replace their dying automated web testing framework, IBM’s Rational Functional Tester(RFT). The process introduced some innovative new tools that we released open-source, but perhaps more importantly, it introduced new practices to our QA group. In this post I’ll discuss our experiences with developing our own testing framework and our experience integrating some programming-best-practices to our QA group.

The existing RFT testing framework:

  1. was decaying. IBM was slow in responding to users’ needs and was falling behind modern browsers.
  2. was expensive to license.
  3. required writing tests in Java—a language too verbose and static for writing tests in an agile environment.

In addition, QA’s test-code base was hard-to-read, repeated itself, was not reusable, not discoverable by others in the organization, and hard for non-RFT specialists to write. To top it all off, their code base had decayed over the years due to a lack of formal test-writing processes. For example, QA’s test writing processes lacked:

  1. formal code-reviews
  2. automated builds
  3. standardized source repository workflows

A motivating factor in helping our QA department transition to a new framework was that NCBI is moving toward a more agile development approach, with programmers (and even stakeholders) wanting to write their own tests, instead of throwing testing duties over the fence to QA.

Laying the groundwork

With some options already out there, we certainly didn’t want to invent our own web automation framework, so we chose the venerable, open-source Selenium2 (WebDriver) framework as a starting point. It’s simply the most widely-used framework for automating web UI tasks.

Although QA’s language of choice had been Java, we insisted they would be more productive writing in a language like Python instead. However, all of QA’s existing tests were written in Java, and they were understandably resistant to switching to a new language like Python. We had them try the pure Selenium2 Python bindings, and they quickly saw the advantage of writing tests in Python rather than Java, with gains in readability and productivity.

English language tests

To satisfy our goal of allowing easier test writing (and reading), we looked at several behavioral driven development tools like Lettuce and Robot Framework. Our goal here was to have tests read like requirements, not like code.

We wanted a non-technical person to read the test and immediately understand the requirement being tested. The fact that automated tests are supposed to test some clear requirement is sometimes lost because test-code tends to get so interwined with application’s implementation. We settled on Robot Framework due to its flexibility, popularity, maturity, design, and community. So far we are glad we did. It excels in all these areas.

To enable web testing we paired Robot Framework with an external Robot library called Selenium2Library (Se2Lib), a very well designed wrapper for Selenium2 for use with Robot Framework.

A typical, well-written, Robot test written using Selenium2Library might look like this. Notice that although Robot allows variables and conditional logic within tests, this test factors those things out into keywords defined in a resource file:


 1 *** Settings ***
 2 Documentation A test suite with a single test for valid login. 
 3 ... This test has a workflow that is created using keywords from 
 4 ... the resource file, resource.txt.
 6 Resource resource.txt
 8 *** Test Cases ***
10 Valid Login
11     Open Browser To Login Page
12     Input Username  demo
13     Input Password  mode
14     Submit Credentials
15     Welcome Page Should Be Open
16     [Teardown] Close Browser

The English commands, such as Submit Credentials are called keywords and in this case are defined in resource.txt:

 1 *** Settings ***
 2 Documentation A resource file containing the application specific keywords
 3 ... that create our own domain specific language. This resource
 4 ... implements keywords for testing HTML version of the test
 5 ... application.
 7 Library Selenium2Library
 9 *** Variables ***
10 ${SERVER} localhost:7272
11 ${BROWSER} firefox
12 ${DELAY} 0
13 ${VALID USER} demo
14 ${VALID PASSWD} mode
15 ${LOGIN URL} http://${SERVER}/html/
16 ${WELCOME URL} http://${SERVER}/html/welcome.html
17 ${ERROR URL} http://${SERVER}/html/error.html
19 *** Keywords ***
20 Open Browser To Login Page
21     Open Browser ${LOGIN URL} ${BROWSER}
22     Maximize Browser Window
23     Set Selenium Speed ${DELAY}
24     Title Should Be Login Page
26 Go To Login Page
27     Go To ${LOGIN URL}
28     Title Should Be Login Page
30 Input Username [Arguments] ${username}
31     Input Text username_field ${username}
32     Input Password [Arguments] ${password}
33     Input Text password_field ${password}
35 Submit Credentials
36     Click Button login_button
38 Welcome Page Should Be Open
39     Location Should Be ${WELCOME URL}
40     Title Should Be Welcome Page
42 Login Should Have Failed
43     Location Should Be ${ERROR URL}
44     Title Should Be Error Page

This is pretty good. It’s very readable, and factoring out the common keywords into resource.txt makes it somewhat reusable, but we had some concerns that motivated us to explore further:

  1. Robot syntax has powerful, programming-like features, such as variables, conditionals etc. But when Robot code gets complicated it seems clunky. Complex things are best done in a “real” programming language like Python.
  2. Not everyone in our organization wants to write Robot tests. Robot is a pretty domain-specific “language” that’s not used elsewhere, and it’s an extra layer of coding.
  3. We wanted to be able to use the page object pattern, which allows test authors to separate how a web page is implemented from what services the page offers. Using procedural style Robot resource files doesn’t allow for inheritance. Page object classes do.

Exploring Robot some more

Playing around with Robot, we found it to be pretty slick. It’s easy to use yet has affordances for advanced usage and has a great API.

For example, Robot’s “plugin” (or library) system relies on the fact that Robot libraries are just Python classes inheriting from object. Any public method of a library class is exposed as a Robot keyword in Robot tests. So for example given a Robot library:

class MyLibrary(object):
        def do_something(self):

…the method do_something automatically maps to the Do Something keyword in a Robot test when importing MyLibrary.

Can Selenium2Library be used outside Robot?

Assuming we could somehow make the page object pattern work in Robot/Se2Lib, we set out to see if it would be possible to write page objects that could be used in Robot tests and outside Robot tests. For this to really work, we’d need to be able to use Se2Lib independently of Robot, because within our page objects, we didn’t want to fork the code depending on the test context. We’d much prefer to base our test code on a common Selenium2 wrapper library–in this case Se2Lib.

With a few quick experiments, we found that Se2Lib is a well designed and well factored library and can, in fact, be used outside from Robot. Here’s a quick test:

$ pip install robotframework-selenium2library
$ python
>>> from Selenium2Library import Selenium2Library
>>> se = Selenium2Library()
>>> se.open_browser("http://www.ncbi.nlm.nih.gov")
>>> se.input_text("id=term", "cat")
>>> se.click_button("search") 
>>> se.title_should_be("cat - GQuery: Global Cross-database NCBI search - NCBI -")
>>> se.close_browser()

This works because Robot libraries are plain old Python classes that have their methods mapped to keywords.

There were a few issues with using Se2Lib outside Robot, such as problems with screenshots. But we felt the issues were relatively minor given the possible benefit of sharing a common wrapper library for pure Selenium2. This proved correct. We’ve been able to smooth these problems out and/or contribute back to Se2Lib in order to make the package less coupled to Robot.

On to page objects

Knowing we could use Se2lib in Robot tests or outside Robot tests gave us the confidence to move forward with our experiment. We did, of course, look around for existing projects that implement the page object pattern in Robot/Se2Lib. There are a couple posts out there, such as:

  • Using the Page Object Pattern with Robot Framework. This is the post where I got the first Robot example above. The post claims that “Robot Framework is not an Object Oriented Framework,” so instead of a class-based approach, the author uses Robot resource files to define keywords. This is an improvement over having keywords defined all over the place, but really adds nothing novel to what we already have with Robot. There’s no inheritance (something that makes the page object pattern powerful), and as I’ve already written, we wanted the flexibility to use these page objects apart from Robot. Plus the fact that Robot tests themselves aren’t object-oriented ignores the fact that Robot itself and its supporting libraries are object oriented.
  • Adam Goucher’s robotframework-pageobjects: this is a more mature implementation, wrapping Se2Lib much like we did, but still exposing the Se2Lib instance. This gave us our inspiration, but we took it several steps further by hiding the Se2Lib instance and adding many other features useful for common testing tasks. You can check out many of these features in the project’s README. In addition, we’ve managed to allow Robot page objects to be used outside Robot.

Inspired by Adam’s attempt at using page objects in Robot, and knowing we could create page objects based on Se2Lib independent of the testing framework, we started working through the details of realizing our goals. We worked out an architecture to allow us to organize our code in a reasonable way and added features to improve the usability and maintainability of Robot tests based on page objects. A few of many improvements were hiding the Se2Lib instance, making keyword mapping more flexible, declarative URL templating, inheritable selectors (mappings of readable, arbitrary names to Se2Lib locators), and page object components.

We also organized an internal system for our organization, hooking Robot page object tests into our continuous integration system, packaging up different projects’ page object packages and making them discoverable by auto-documenting them using libdoc for less technical folks and Sphinx for page object developers.

A real world example

Now let’s take a look at a real Robot test using our robotframework-pageobjects package. This is from our demo on github. We’ll test that when searching Pubmed, for a term, the first result document has that term in its body text.

Here’s that test in just Robot/Se2Lib:

 1 *** Settings *** 
 3 Documentation  Pubmed tests
 5 Library    Selenium2Library
 7 *** Test Cases *** 
 9 When a user searches Pubmed for a term, the first result page's body should contain the search term.
10     Open Browser  http://www.ncbi.nlm.nih.gov/pubmed  firefox
11     Input Text  id=term  cat 
12     Click Button  id=search
13     Click Link  xpath=(//div[@class='rslt'])[1]/p/a
14     ${BODY} =  Get Text  css=body
16     # Lowercase the body so we can compare against "cat"
17     # using the user-defined keyword in the "Keywords" section below
18     ${BODY_LOWER} =  Lowercase  ${BODY}
19     Should Contain  ${BODY_LOWER}  cat 
20     [Teardown]  Close Browser
22 ***Keywords***
24 Lowercase   [Arguments]   ${string}
25     [Return]   ${string.lower()}

Clearly, using just Robot with just Se2Lib has problems:

  1. It’s no easier to read than a test written in Python. In fact, it’s less readable. The test doesn’t read like a requirement to be tested, rather it reads like page implementation details.
  2. There are element locators scattered about.
  3. The code’s not reusable.

Now we’ll write the same test using page objects:

 1 *** Settings *** 
 2 Documentation  My first IFT tests
 3 ...
 4 Library  pubmed.PubmedHomePage
 5 Library  pubmed.PubmedDocsumPage 
 6 Library  pubmed.PubmedArticlePage
 7 *** Test Cases *** 
 8 When a user searches Google for a term, the first result page's body should contain the search term
 9     Open Pubmed Home Page
10     Search For  cat 
11     Click Result On Pubmed Docsum Page  1
12     Pubmed Article Page Body Should Contain  cat 
13     [Teardown]  Close Pubmed Article Page

Ahh, much better. Now our test makes it clear what services the pages offer and we can more clearly see the point of the test. But how do we make this happen? And how do we make the underlying code maintainable and reusable?

Here are the page object libraries that make the above test work. pubmed.py:

 1 from robotpageobjects import Page, robot_alias
 2 from robot.utils import asserts
 5 class PubmedHomePage(Page):
 6     """ Models the Pubmed home page at:
 7         HOST://ncbi.nlm.nih.gov/pubmed"""
10     # Allows us to call this page
11     # something other than the default "Pubmed Home Page"
12     # at the end of keywords.
13     name = "Pubmed"
15     # This page is found at baseurl + "/pubmed"
16     uri = "/pubmed"
18     # inheritable dictionary mapping human-readable names
19     # to Selenium2Library locators. You can then pass in the
20     # keys to Selenium2Library actions instead of the locator
21     # strings.
22     selectors = {
23         "search input": "id=term",
24         "search button": "id=search",
25     }
28     # Use robot_alias and the "__name__" token to customize
29     # where to insert the optional page object name
30     # when calling this keyword. Without the "__name__"
31     # token this method would map to either "Type In Search Box",
32     # or "Type In Search Box Pubmed". Using "__name__" we can call
33     # "Type in Pubmed Search Box  foo".
34     @robot_alias("type_in__name__search_box")
35     def type_in_search_box(self, txt):
36         self.input_text("search input", txt)
38         # We always return something from a page object, 
39         # even if it's the same page object instance we are
40         # currently on.
41         return self
43     @robot_alias("click__name__search_button")
44     def click_search_button(self):
45         self.click_button("search button")
47         # When navigating to another type of page, return
48         # the appropriate page object.
49         return PubmedDocsumPage()
52 class PubmedDocsumPage(Page):
53     """Models a Pubmed search result page. For example:
54     http://www.ncbi.nlm.nih.gov/pubmed?term=cat """
56     uri = "/pubmed/?term={term}"
58     # This is a "selector template". We are parameterizing the 
59     # nth result in this xpath. We call this from click_result, below.
60     selectors = {
61         "nth result link": "xpath=(//div[@class='rslt'])[{n}]/p/a",
62     }
64     @robot_alias("click_result_on__name__")
65     def click_result(self, i):
67         # For selector templates, we need to resolve the selector to the
68         # locator first, before finding or acting on the element.
69         locator = self.resolve_selector("nth result link", n=int(i))
70         self.click_link(locator)
71         return PubmedArticlePage()
73 class PubmedArticlePage(Page):
75     uri = "/pubmed/{article_id}"
77     @robot_alias("__name__body_should_contain")
78     def body_should_contain(self, str, ignore_case=True):
79         ref_str = str.lower() if ignore_case else str
80         ref_str = ref_str.encode("utf-8")
81         body_txt = self.get_text("css=body").encode("utf-8").lower()
82         asserts.assert_true(ref_str in body_txt, "body text does not contain %s" %ref_str)
83         return self

We’d run the test like this:

$ pybot -vbaseurl:http://www.ncbi.nlm.nih.gov -vbrowser:firefox test_pubmed.txt

…and here’s a regular, old python unittest TestCase using the same page objects, demonstrating that the same page objects that drove the Robot test can manage the driver in any context:

 1 from pubmed import PubmedHomePage
 2 import unittest
 5 class PubmedTestCase(unittest.TestCase):
 7     def setUp(self):
 8         self.pubmed_homepage = PubmedHomePage()
 9         self.pubmed_homepage.open()
11     def test_first_result_page_body_should_contain_search_term(self):
12         pubmed_docsum_page = self.pubmed_homepage.search_for("cat")
13         self.article_page = pubmed_docsum_page.click_result(1)
14         self.article_page.body_should_contain("cat")
16     def tearDown(self):
17         self.article_page.close()
19 if __name__ == "__main__":
20     unittest.main()

And to run it:

$ python test_pubmed.py

Other features

There are a bunch of other features we worked in to the robotframework-pageobjects package that make enterprise testing more convenient. Here’s a sampling:

Option handling

While the page object pattern doesn’t necessarily relate to option handling, or test parameterization, we find ourselves configuring tests so often that it was convenient to address these needs together.

robotframework-pageobjects allows different methods for configuring test-runs. As you see, you can pass in options using Robot variables, use variable files or you can set environment variables. Using environment variables is especially convenient because you can set them once per session or persist them in your .bash_profile file.

One thing to note is that often you cannot easily pass test configuration data to pure unit test frameworks like unittest. Because of this, when you run your page-object based tests outside of Robot, you must set options via environment variables.

In our organization we simplify this by employing a utility script called ift (It’s Fun to Test). One of the things it does is help set options. For example:

$ ift set # List the set options, no options set, other than IFT_HOME
$ ift set baseurl http://www.ncbi.nlm.nih.gov
$ ift set browser firefox
$ ift set


Apart from creating page objects, you can also create page components. These encapsulate reusable functionality on your pages and are automatically attached to instances of your page object once declared. You could, of course, code these into base page classes, but components help to keep the page objects clean and allow you to use composition over inheritance.

For how to do this, see the README.

Cloud testing

Cloud testing via Sauce Labs is built into robotframework-pageobjects. We were constantly fighting with our Systems department to maintain a diverse and older set of platform/browser combinations. Now with Sauce we can do thorough cross-browser testing when needed on demand. A great feature of their service is video captures of tests. To test on Sauce, use the options prefixed with sauce_. See the option section of the README for more details.

So how’s it working out for us?

First the bad. Getting QA and developers to understand the page object pattern and getting them used to devoting the time it takes to properly model the application under-test is a bit of a downer. It’s much quicker to get people writing tests when you give them, say, Selenium2 and let them go at it. That being said, a lot of folks don’t understand that test-code is, well, code, and without sensible frameworks and the writing/editing discipline to go along with them, that quick and easy test-code writing can quickly become a great maintenance burden.

Another complicating factor is explaining to people when to use Robot and when to use something like Python’s unittest framework. I usually advise our folks to use Robot, but I encourage them to ultimately use their best judgment. Sometimes skipping the layer of Robot’s indirection just seems right. The important things are:

  • balancing test-readability with ease of test-writing
  • factoring out page implementation details from the tests themselves and…
  • getting used to incorporating twelve-factor concepts into test-development, not just app development.

Now the good. From what I see QA’s growing repository of new and migrated tests are much more readable. While I used to have to sift through reams of incomprehensible Java code littered with page-implementation details to understand what was being tested, now within a few seconds of looking at the tests, it’s clear what’s being tested.

I’m also seeing more collaborative test-development and code-reuse. Some of this is due to the framework itself, which facilitates this, while some of this is due to the new practices we’ve introduced to the test-writing process, such as git workflows, online code-reviews, and automated builds.

Crucial to addressing the challenges of working in a more deliberate, disciplined way is proper training and communication. One can’t introduce new tools and paradigms, walk away and expect success. If you are leading an effort like this, you must support the infrastructure (eg. continuous integration configurations etc.), and mentor folks until they feel comfortable enough with the new tools and processes. Only then will they see the benefits of working in unfamiliar ways.