Nowadays, one of the first questions you are asked as a Quality Assurance Engineer, is do you use Automation? Test Automation has definitely been a hot topic in recent years. Everybody is talking about it, trying to use it, complaining about it, and still it remains very much an in house business, with each company developing its private solution tailored to its own products and needs. This is only natural as products vary a lot and, especially in the game of Enterprise Software, it is hard to imagine a single Test Automation solution that can be used regardless of the actual product being developed. This being said , working in Test Automation for a couple of years in which you can see firsthand the obstacles and the rewards of automation, you eventually come to ask yourself the obvious question: How do you win the game of Test Automation?
This article looks into the key aspects of developing and implementing a Test Automation Solution. It provides a step by step guide based on an existing solution used by HP"s Server Automation Product called AXIS. While the focus of the article will be on backend automation and mainly functional testing, the ideas presented can be applied to any Test Automation Solution regardless of the technology behind it.
So let"s get started. As with any game that you want to be good at, it is important to first understand how the game is played. Think of this as the instruction manual for our game. So what is a Test Automation Solution?
A Test Automation Solution, in our vision, is made up of three key elements:
A Test Automation Framework is usually defined as a set of assumptions, concepts, and practices that provide support for automated software testing[4]. The Framework is the base of your Test Automation solution. It handles the resource allocation, test execution, option validation and environment management. It is very important to use a framework that is suitable for the type of automation and the type of product you want to test.
Our product facilitates the management of large data centers comprised of servers that have different OS flavors, providing all the tools necessary for a system administrator such as OS provisioning, package management, configuration management or compliance management.
So, in order to run automated tests on such a product we need a framework that can handle a multi-platform environment. The AXIS solution is based on an already existing automation framework called STAF (Software Testing Automation Framework). STAF is an open source framework designed around the idea of reusable components, called services (such as process invocation, resource management, logging, and monitoring)[5]. It allows us to set up the needed prerequisites and check the desired outputs for our tests across all major platforms used today. This is a fortunate example that you don"t always have to start from scratch when implementing a Test Automation Solution. Automation tools come in many sizes, shapes and flavors and even when implementing your own solution they can offer a much needed helping hand.
A key aspect you have to take into account when developing your framework is the Discovery process. What this does is simply translate the environment information from the product being tested to the automation framework. Discovery allows you to define a test environment by specifying the test targets and all other necessary information in a format suitable to your application before running the actual Tests.
The content is represented by the automated tests. A clear separation between the framework and the content can make your TA Solution applicable to more than one product. The tests themselves can be developed in the programming language of your choosing, as long as it is supported by the framework and the product of course.
Developing content requires a good understanding of the product being tested so, ideally, QA Engineers should play a significant part in writing automated tests.
Having a framework and automated tests is of no use unless people can use them. One of the most encountered problems with in house solutions is that only Automation Engineers know how to run the tests or interpret the results. As with any product that is only tested by the ones who developed it and know its strengths and weaknesses from the start, this approach will eventually lead to quality issues.
Usability is one of the key features of any Test Automation Solution. The more people use it, the better it becomes. That is why it is important to have a user interface that allows people to run the tests easily and understand the results.
The AXIS solution employs both a Graphical User Interface, based on a web server that allows users to select the tests they want to run and the test bed on which to run them and to view the test results in a simple tabular format where they can explore the test logs down to the test case level, as well as a Command Line Interface to back it up.
OK, so now that we know what the Automation game is about, we can start playing. This walkthrough will give you the necessary information to build your own Test Automation Solution. It is important that you don"t skip levels as you will most likely just end up having to go back and redo them at some point. Game on!
First things first, you need to choose what type of player you will be. Do you want to go towards GUI automation or will you choose backend automation? Or maybe you want to do both. Either way, it is important to know what approach you will be taking from the start.
The most important thing at this level is to know the product. Some products may not be suited for backend automation, while others might not have a GUI to automate. GUI test automation often relies on record and playback tools, while backend test automation relies on the product having an exposed API that allows you to call its methods from your test cases. Make sure the choice you make is suitable for the product that is being tested. You can work with the development team to make the product more automation friendly, but this must be done early in the product lifecycle.
Another thing you need to consider at this level is what part of the testing process you aim to automate. Do you want to automate only the test execution or also the result validation? Some Automation solutions focus on the execution part leaving validation in the hands of QA while others do both.
Going back to our example we find that AXIS is a good choice for Level 1. It focuses only on backend automation. Tests are written in Python and implement scenarios by calling methods from the products exposed API, replicating user behavior via scripts. Result validation is also automated with the tests verifying the method return values against a set of expected values and presenting the results to the user in a simple: Passed/Failed/Skipped format. Users can then drill down and check the logs to investigate failures if necessary.
Once you have chosen your automation approach and verified that it works for the product being tested, you can move on to the next level.
Now that we know how we are going to approach the game it"s time to immerse ourselves in the game world. All worthwhile games have a backstory and if you want to understand the context in which your quest is taking place, it pays off to read a bit of the game lore before starting to slash everything left and right.
So how do we do that? We need to review the existing testing process. Look at the manual test cases to make sure that they are written in a manner that makes them suitable for automation. Work with the QA team to put together a list of prime candidates for automation. The usual suspects will be the regression suites, smoke suites or product deployment tests. Identify key tests that upon failure will result in high priority defects and define from the start the notion of Passed and Failed tests.
One more thing to do at this level is to establish a clear link between the manual and the automated test cases. This will help to determine the automation coverage later on. This level might not sound like a lot of fun but if you make this effort before you start automating, you will find that your results will be visible much sooner. The AXIS solution uses data files attached to each test that contain the information which links the automated test to the manual one. These are used to upload automated runs result to the content management system used by management. Think of this as a way of keeping score. You will definitely want people to know what level your character is and what quests you have completed.
Once you have your list of automation candidates, pass/fail conditions and tests written in a format suitable for automation, you can move on to the next level.
So you know how the game is played and the backstory. It"s time to choose your character class. Choosing a class that is more suited to your approach will make your hero more effective. Do you want to be a melee fighter, a mage or a ranger? Each alternative has its strengths and weaknesses and will influence how you play the game.
In this level you must define what you want your Test Automation Solution to do and how you plan to accomplish it. You should define clear requirements as to what your solution will do. This way you will have a clear standard by which you can measure your progress and you will know when your solution is ready to be deployed.
Let"s say you want to automate the regression tests for your product. Regression tests are always prime candidates for automation. Having successfully completed levels 1 and 2 you should have an already defined list of tests that cover all areas needed in a regression run. You also know what the pass/fail conditions are and having reviewed the tests you know they are written in a format suitable for automation. All that is left is to define your requirements. Create User Stories for your automation effort. Provide estimates and done conditions. It is important to make your effort and your progress visible so that people know when automation becomes available and what areas it covers.
A common mistake at this level is trying to do it all. There is usually a push to automate as many tests as possible. Automating tests blindly helps no one. By defining requirements you will know exactly what you commit to, when you will be done and how your tests will be used.
Apply this process for all your automation goals: regression tests, build validation tests, product deployment tests. Once you have requirements defined for your automation tasks, you can proceed to level 4.
Enough with all the backstory and character creation, It"s time to start playing. As any gamer will know, in order to have a truly powerful hero that can take on any quest you need to do some grinding first. This means going through the easier quests in order to build up your skill tree, equipment and experience. You need to work your way to the big bosses in any game.
There is no quick way to go through level 4. Having successfully completed level 3, you will have a set of clearly defined requirements, or quests, that you need to complete. It"s always a good idea to start with Smoke Tests. These should be the most basic tests so they should be easier to automate and can be used for tasks like build validation. Another good choice is to work on Deployability tests. If you have the ability to deploy the product and run a minimal suite of smoke tests on it, you can already set up a continuous integration process. The main objective on this level is to create value as soon as possible.
One tip for this level is to try multiplayer. Experience grows faster when you are in a party. Work with feature teams to ensure the created tests are run and cover all the aspects defined in the requirements. They are the only ones who should decide when your quests are done. The more your tests are run the better they will become. Remember that, as an automation engineer, your job is to implement and provide support for the Test Automation Solution, not to run the tests for other teams and investigate results.
Once your quests are done and your party members consider that you are worthy enough, you can advance to the next level.
Having completed level 4, you now have some experience in the game of automation. You have completed some quests and you start to recognize the importance of having good gear. In order for your hero to be truly successful, you need to have the right items and to keep them in good shape. In this level you will also learn to keep different item sets for different quests
So what are your items in this game? Simply put the only items you have at your disposal; they are your test suites. Here are a couple of attributes that you should definitely look to improve on in your tests:
It is a good idea to use a versioning system for your test just the same as you do for your product. When working with multiple product versions, it"s good to have a version of your tests for each version of the product. Certain features may vary across product versions, so a single test might not be applicable.
Now that your gear is up to speed, it"s time to start making a name for yourself in level 6.
Sure, using your top level gear to slash away completing quests is fun but in order to advance in the game you will have to share your gear with other players. Crafting items for others will help build up a reputation and having strong party members will make your progress in the game much easier.
Level 6 is about sharing. Make sure your tests are available for other users. Package them so that they can be run from any platform. As we said in the previous level, versioning is good. Make use of your products versioning implementation. It"s safe to assume that you have one in place, so why not use for the tests as well. Having a new test package with every new build can help you with adapt your tests faster, especially in early development stages. Always remember that automation is only good if it is actually used. Providing easy access to the correct tests for all versions is what you need to accomplish at this level.
So now you have completed your quests, built up your gear, crafted items for your fellow party members, what else is there to do?
This is the final level of the game. You are now a true guild master. Lower level players look up to you for assistance, items and tips. But does the game end here? Just because you are number one does not mean you will remain at this level for a long time.
The nature of the automation game is always shifting. You need to be on your feet the whole time if you don"t want to be knocked back to lower levels. Always look to improve your Test Automation Solution. Keep your tests suites up to date. Add new tests, review and update existing ones and remove tests that are obsolete. Work on the automation framework to improve performance and reliability.
Share the knowledge you have gained. Hold trainings to keep QA teams up to date with the state of the automation solution in place. A well designed and implemented solution will allow QA engineers to develop automated tests themselves, freeing you up of working on enhancements. Provide support for teams that use automated tests by defining clear processes based on logged requests.
Changes in the product will always result in new challenges for you, so try to communicate with product management and development teams to ensure that automation is taken into account when designing new features.
Following this leveling guide should help to put into place a Test Automation Solution that can be used effectively and that provides useful and visible results. In the end, like in every game, if you study the game, plan ahead, put in the effort at all levels and don"t cheat, you will emerge victorious. So enough talk, let"s win the game of Test Automation!
1. Test Automation Architecture: Planning for Test Automation - Douglas Hoffman, 1999
2. Test Automation Frameworks - Carl J. Neagle, 2000
3. Common Mistakes in Test Automation - Mark Fewster, 2001
4. Wikipedia, The Free Encyclopedia
5. Software Testing Automation Framework (STAF) - http://staf.sourceforge.net/
by Vlad Ciurca
by Dhyan Or
by Tudor Bîrlea
