The next level of Test Abstraction – Test Factory

Darius Bozga
Senior QA Engineer – Payments @ Betfair

TESTING

All new features determine us to add a couple of new cases to one or multiple data providers. What happens when you have two, three or more features that need the adding of various details to data providers? Eventually the data providers from the tests will get bigger and harder to follow.

Let's say we have an online store application that can be used for purchasing items with one type of account (regularAccount) and this store is available for 3 domains. This online store offers only English localization.

A basic test flow for a sanity check would be the following:

@BeforeClass()
public void beforeClass() {
    regularAccount = new Account();
    regularAccount.setUsername("Regular User");
    regularAccount.setLanguage("English");

}

@DataProvider(name = "getLoginDetails")
public Object[][] getLoginDetails(){
return new Object[][]{
            {"url .com",regularAccount},
            {"url .ro",regularAccount},
            {"url .uk",regularAccount},};
}

@DataProvider(name = "purchaseItemDetails")
public Object[][] getPurchaseItemDetails(){

    return new Object[][]{
            {"Phone",regularAccount},
            {"Sink",regularAccount},};
}

@Test(dataProvider = "getLoginDetails")
public void verifyLogin(String url, Account account) {
    System.out.println("Accessing the url "+url);
    System.out.println("Login with account " + 
     account.getUsername())
}

@Test(dataProvider = "purchaseItemDetails")
public void verifyUserCanPurchaseItems(String item, 
  Account account) {
    System.out.println("Login with username "
      + account.getUsername());
    System.out.println("Purchassing item: "+item);
}

@Test()
public void verifyStoreLocalization() {
    System.out.println("Login with user "
       +regularAccount.getUsername());

    System.out.println("Verify section is in "
    + "correct language " 
    +regularAccount.getLanguage().equals("English"));
}

@Test()
public void verifyItemIsDisplayedOnStore(){
    System.out.println("Verify desktop item is"
    + "displayed in store");
}

For longer product lifecycles new features are either implemented from scratch or implemented as modification of current ones. For our online store, the next step in product development would be to add an Admin section accessible only via an adminAccount which would add or change the content.

This brings a new type of account, since a normal user should not be able to access this section and since an admin is not able to purchase items from this store. Our sanity tests will need to accommodate these changes by adding hard-coded values to the data providers.

If we take a look at the first data provider (getLoginDetails) the data had to double to cover the same scenario for both types of accounts. The original 3 lines were transformed into the following 6 lines:

@DataProvider(name = "getLoginDetails")
public Object[][] getLoginDetails(){
return new Object[][]{
            {"url .com",regularAccount},
            {"url .ro",regularAccount},
            {"url .uk",regularAccount},
            {"url .com",adminAccount},
            {"url .ro",adminAccount},
            {"url .uk",adminAccount},};
}

What would happen if the online store became available to users from other domains, or to other types of accounts for managers, or if it accepted other languages? We would end up expanding these data providers to double the lines for each new type of change.

We can avoid any future changes in the tests employed, by altering the flows or by duplicating unnecessary code. The following rules need to be observed:

Identify all the common parts of the tests that we would like to unite and the values that we supply these tests with.
- In this case we can define an Account object that has a username, a language and an access level. We also have a well-defined list of domains where the customers can access the application from and a list of Pages that can be accessed by account types based on their permission level.
Remove the hard-coded values from the data providers and from the tests. If we make an abstract data provider that turns a list or a map into an Array of Object Arrays, then we can eliminate the hard-coded values.
- We can now supply our test class with a general set of rules that applies to each Account type.

In our case, the basic values required will be:

A list of domains available for the store
A list of available sections
An account object with its particularities

By moving these particularities into an Account object, we can supply our test class with a large number of account variations, and all the tests in our class will run for each one, eliminating our need to go and check all the data providers that need updating, or our need to duplicate test classes for each type of account.

When we provide the Account and two lists as constructor, turning out test class into the following one, we will have the ability to run the same set of tests for each value.

public OnlineStoreSanityTests(Account account, List listOfDomains, List listOfPages) {
    this.account = account;
    this.listOfDomains = listOfDomains;
    this.listOfPages = listOfPages;
}

@DataProvider(name = "getLoginDetails")
public Object[][] getLoginDetails()
{
    Object[][] objectArray = new Object[listOfDomains.size()][];
    for (int i = 0; i < listOfDomains.size(); i++) {
        objectArray[i] = new Object[2];
        objectArray[i][0] = listOfDomains.get(i);
        objectArray[i][1] = account;
    }
    return objectArray;
}

@DataProvider(name = "getPermissionDetails")
public Object[][] getPermissions()
{
    Object[][] objectArray = 
        new Object[listOfPages.size()][];
    for (int i = 0; i < listOfPages.size(); i++) {
        objectArray[i] = new Object[2];
        objectArray[i][0] = listOfPages.get(i);
        objectArray[i][1] = account;
    }
    return objectArray;
}

@DataProvider(name = "purchaseItemDetails")
public Object[][] getPurchaseItemDetails()
{
    return new Object[][]{
            {"Phone",account},
            {"Sink",account},
    };
}

@Test(dataProvider = "getLoginDetails")
public void verifyLogin(String url, Account account) {
    System.out.println("Accessing the url "+url);
    System.out.println("Login with account " 
      + account.getUsername());
    System.out.println("Login done with success");
}

@Test(dataProvider = "purchaseItemDetails")
public void verifyUserCanPurchaseItems(String item, Account account) {
    System.out.println("Login with username "
     + account.getUsername());
    System.out.println("Purhcasing item: "+item);
}

@Test()
public void verifyStoreLocalization() {
    System.out.println("Login with user "
     +account.getUsername());
    System.out.println("Verify section is in"
    + " correct language " +account.getLanguage());

}
@Test(dataProvider = "getPermissionDetails")
public void verifyUserPermissionsToSection(
String section, Account account) {
    System.out.println("Login with user "
    + account.getUsername());
    System.out.println("Access section "  
    + isUserAllowedToAccessSection(section,account));

}

@Test()
public void verifyItemIsDisplayedOnStore(){
    System.out.println("Verify desktop is" +
    +" displayed in store");

private boolean isUserAllowedToAccessSection(String section, Account account){
    return section.contains(
      account.getAccessLevel());
}

The obvious questions at this point would be: How do these tests run? Where do we set the test data?

A Test factory allows you to create tests dynamically using different values. In a test factory we will provide the required information for the tests to be created just like a data provider does for a Test method. A Test Factory is similar to a Data provider, but for Behavior defining test classes. By using this test factory for any future changes that bring new accounts to our application, we just need to add a new Account type to the \@Factory section.

Our test factory looks like this:

@Test(groups = "run")
public class OnlineStoreTestFactory {
  private static List listOfDomains= 
      new ArrayList(); 

private static List listOfPages= 
      new ArrayList();

 static {
        listOfDomains.add("url .com");
        listOfDomains.add("url .ro");
        listOfDomains.add("url .uk");
        listOfPages.add("Store Section");
        listOfPages.add("Management Section");
        listOfPages.add("Admin Section");
    }

@Factory()
public Object[] factoryMethod() {
return new Object[]{
   new OnlineStoreSanityTests(
      generateAccountForTestOne(),
      listOfDomains,listOfPages),
      new OnlineStoreSanityTests (
         generateAccountForTestTwo(),
         listOfDomains,listOfPages),
      };
    }
    public Account generateAccountForTestOne() {
        Account account = new Account();
        account.setUsername("Regular User");
        account.setLanguage("English");
        account.setAccessLevel("Store");
        return account;
    }
    public Account generateAccountForTestTwo() {
        Account account = new Account();
        account.setUsername("Admin User");
        account.setLanguage("Spanish");
        account.setAccessLevel("Admin");
        return account;
    }

}

We have isolated the behavior from the actual data, which gives us a clear view of what we want to change or view. When we want to check the flows, we can go to the Test Class, whereas, when we want to see or change values for an account, we can go to the Test Factory. In the test Test Factory, we now have a view of each account type without having to scroll through all the data providers and without checking what values we have for each type of account.

Account account = new Account();
account.setUsername("Regular User");
account.setLanguage("English");
account.setAccessLevel("Store");
return account;

The Test Factories can be run from suite files just like any other test class. We can run the tests in parallel to reduce the overall time required for the tests to run.

<suite name="Suite-A" verbose="1">
    <test name="test" group-by-instances="true" parallel="classes" >
        <classes>
            <class name="com.testFactory.example.OnlineStoreTestFactory"></class>
        </classes>
    </test>
</suite>

The full example project is found on git hub