To consolidate QA resources, best practices and automation tools dispersed around the enterprise, and to optimize testing performance, many companies turn to Testing Centers of Excellence (TCoEs). At the same time, TCoE isn't a heal-all concept for every company, as its setup requires a lot of organizational efforts. TCoE performance results can't always outweigh them.
How can we know for sure whether a TCoE is a successful QA hub or the fifth wheel for the enterprise? Here you can find five main TCoE performance criteria to monitor and be realistic about your TCoE's effectiveness.
You can evaluate TCoE performance with the help of KPIs; each KPI is based on a particular TCoE goal resulting from the QA challenges TCoE is supposed to face. For example, TCoE managers are to advise a test lead on the most appropriate line-up of a test team for a project (challenge) to meet the budget limits (goal). So, from here you get the KPI for testing costs.
You need to prioritize the whole set of goals and transform only the most acute ones into KPIs. For the evaluation to be multidimensional, you need to choose, from your shortlisted TCoE goals, the ones corresponding to different TCoE focus areas, for example, testing quality, testing time and budget reduction, QA maturity nurture and the promotion of test automation.
A tricky point with TCoE performance measurement is that very good results for all KPIs simultaneously are not an adequate aim to be set. To get the right picture of TCoE performance, you need to acknowledge that impressive KPIs for all TCoE goals are utopian. One striking KPI is prone to negatively influence another, and only moderately good and balanced results are a key to performance success. For example, if TCoE managers have encouraged the test team to complete all the project's tests in 3 days, which sounds quite stunning, and the KPI of testing time proves it, this impressive result is likely to be bound to low testing quality (a lot of missed bugs) and the bloated number of test engineers involved (excessive testing costs).
Your KPIs are meant to highlight the TCoE performance results that are of utmost importance for your enterprise with regard to its specific business objectives and priorities. That's why there is no set of universal KPIs. Here's the top list of the most common KPIs, which can be used as a springboard for your tailor-made one.
This KPI is based on such a TCoE goal as testing quality enhancement. To find out whether the testing results of a specific project live up to the established quality standards and truly contribute to the quality of an application under test (AUT), it's handy to use the percentage of defect leakage.
The excessive number of missed bugs (more than 5% of the total bugs' number) may be caused by the following problems: the test team's misunderstanding of software requirements, an inefficient test plan not corresponding to the changing project requirements, the lack of coherent testing quality standards and methodologies.
However, if you see that a missed-bug rate equals zero, you need to check whether the test team managed to meet the iteration and project deadlines, and whether the number of found bugs results out of the iteration requirements' scope.
Test engineers are often pressed to complete testing faster to ensure that bug-less software reaches its users sooner. The average testing duration in comparison with the results observed without TCoE indicates its effectiveness.
If average testing time remains unchanged, it may indicate that your TCoE isn't successful in the development of a balanced and efficient test strategy (involving automated testing and the coherent allocation of the test team's roles and duties). It's likely that a testing methodology promoted by TCoE needs further improvement.
On the other hand, if you find out that the testing duration got a serious and rapid reduction after TCoE's setup (more than 30-40% in comparison with the "before-TCoE" rate), you need to check the missed-bug rate, as too little time may have been devoted to testing and the testing quality could have suffered.
The goal promoted by this KPI is to optimize the QA and testing budget. To understand whether your TCoE is successful in meeting this objective, you need to compute the average cost of testing before and after establishing TCoE.
TCoE's typical cost savings rate is about 35% in a 3-year time frame. A low cost reduction rate may be the result of unbalanced resource allocation or insufficient control over a test team's efficiency. Still, you shouldn't be too optimistic about an unexpectedly high cost reduction rate. TCoE managers may try to economize regardless of missed bugs and poor testing quality, for example, by giving their testing teams insufficient quantity and qualifications.
The achievement of higher QA maturity levels and promotion of QA culture around the whole enterprise are fundamental TCoE goals. QA maturity can be evaluated using one of the widely recognized models: Test Maturity Model Integration (TMMi) or Test Process Improvement (TPI). Both models describe what QA and testing activities are characteristic to each of 5 (in TMMi) and 4 (in TPI) maturity levels.
The gist of the TCoE presence within an enterprise is to ensure that QA activities (performance measurement and regular reporting, knowledge transfer, smooth communication between the departments, etc.) are promoted through the enterprise. Therefore, low QA maturity levels are not normally observed if there is TCoE, except for the case when it's still at the beginning of its formation.
In case this KPI shows the highest maturity level, you should pay special attention to TCoE managers not becoming oversensitive or perfectionistic about QA. The high quality of processes and output is just one of the business goals. The highest QA maturity level should not interrupt other business needs and priorities, as TCoE managers being overprotective about quality may impede meeting project deadlines and budgets.
Test automation is one the most booming trends for the near QA future, and its promotion is one of the main TCoE goals. The major reasons why test automation is so popular include: the reduction of testing time and costs, enhanced testing quality, and vast test coverage (unachievable with manual efforts). The test-automation rate is counted by dividing the number of automatically covered test cases by their total number.
However, you should mind that this KPI may be tricky or even misleading, if you are not fully aware of some test automation hidden pitfalls. Deciding on every testing project's automation rate demands deep analysis of the software requirements and testing scope. For some testing projects, the percentage of test automation can catch up with 70-80%, while for others it may remain 0. It's neither a good, nor a bad result, as, by far, not every testing project should presuppose test automation.
There is no point to perform automated testing for small-scale projects, as it won't pay off. Besides, it's senseless to insist on test automation in projects with constantly changing requirements. The automated test scripts will get outdated after each iteration or source code change. Some test types like: usability, ad-hoc and exploratory testing - can't be automated. Thus, TCoE should strike a balance between automated and manual testing efforts. So, if you find out that this KPI results in a generally high percentage through all of the testing projects, you should check whether the TCoE managers promote a reasonable approach to test automation.
To ensure that your TCoE performance results meet expectations, you should come up with an effective and coherent set of KPIs TCoE is to live up to. Your KPIs need to evaluate TCoE performance from different perspectives to provide a balanced view of how TCoE meets its various goals. Though no set of KPIs is universal, our top five KPIs may come to be handy as they reflect such crucial TCoE objectives as: shorter testing time, reduced testing costs, testing quality improvement, and higher levels of test automation and QA maturity. Still, don't forget to carefully analyze the assessment results, as one too impressive KPI may be the sign of some missing TCoE goal.
by László Papp
by Rama Anem
by Ovidiu Mățan
by Ovidiu Mățan
