At some point, one may be interested in finding out how well-developed the automated test cases which validate the Web Services functionality actually are. In other words, it would be good to know how much the test cases hit the code. One way to test the Web Services functionality is by using SOAP UI.
This article describes how we can use the Squish Coco application to determine how much the SOAP UI test cases impact the C# code.
Squish Coco is a coverage tool for C, C#, C++ and Tcl code. Using the instrumented source code, Squish Coco analyses the source code of the application under test.
Executing a test suite against an instrumented application produces data that can be analyzed later. This analysis can be used to understand how much of the source code has been hit by tests, which additional tests need to be written, or how the test coverage changed over time.
Squish Coco may give a measure of how much of the code is tested.
Squish Coco consists of two main tools:
A. CoverageScanner - This is a compiler wrapper which feeds instrumented source code to the native compiler. When using SquishCoco, it will tell the build tool to use CoverageScanner as the compiler. CoverageScanner is a C++ program that—in effect—replaces the usual compiler.
B. CoverageBrowser - This is a GUI tool for visualizing and analyzing code coverage based on the reports generated by running the test suite on an instrumented version of the application.
Squish Coco preprocesses the source code using the native preprocessor program, during which it inserts instrumentation code and, at the end, it compiles and links the project file, in a similar way to a normal build procedure. After the code is instrumented, CSMES files are generated - these files contain the instrumented code.
CoverageScanner's principle of code coverage analysis is not to highlight executed source lines, but rather to place marks on execution paths.
CoverageScanner parses all C++ / C language constructs and detects: executed functions, executed source code lines, execution paths, Boolean expressions which produce different execution paths (if, for, while, switch . . .).
After the detection phase, CoverageScanner inserts the instrumentation statements into the code. This modified version of the application is called instrumentation.
When the application is tested, coverage results are generated for each execution (.CSEXE file). These results may be analyzed using CoverageBrowser.
Instrumentation types:
Line coverage - Instrumenting the execution of every executable source code line
Branch coverage - Instrumenting the execution of each branch block (e.g., the body of any if statement).
Decision coverage - Instrumenting each Boolean decision for loop and selection statements (e.g., record both the Boolean expression itself—true or false—and the body of the while, for or if statement).
Squish Coco is a multi-platform tool:
Linux (32- and 64-bit)
Windows (32- and 64-bit)
Mac OS X (32- and 64-bit)
Solaris, AIX and other UNIX systems
Embedded Linux
Embedded Windows
Mobile operating systems
Coverage reports may be generated in the following formats: HTML, XML, EMMA -XML, text, Junit, Cobertura.
Squish Coco stores information per individual test, with an optional name, with FAIL vs. PASS status assigned and with additional comments. This way various types of analysis can be performed in the post-processing phase:
Calculation of the optimal order of test execution that will most quickly maximize the overall coverage.
Comparison of coverage between test executions
Comparison of coverage between different software builds
Safety standards and regulatory bodies mandate the use of code coverage analysis to ensure a proper degree of testing:
ISO 26262 - Road vehicles - Functional safety
EN 50128 - Railway applications
DO 178 - Software Considerations in Airborne Systems and Equipment Certification
IEC 61508 - Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems
FDA - Food and Drug Administration
In most cases coverage is achieved through the use of a unit testing framework. Squish Coco will be able to gather coverage information from whatever framework is being used. It has integrated versions available for frameworks:
CppUnit
QTestLib
GoogleTest
CxxTest
In addition, tests can also be driven through automation tools.
Coco can easily be integrated with various build and test frameworks to serve as a component of Continuous Integration (CI) system, maybe integrated with Jenkins, SonarQube, Bamboo or an in- house build system.
SOAP UI is one of the SmartBear Software applications that can be used to test Web Services.
Since we are instrumenting services that do not terminate, we need to write our own handler which triggers the saving of the .csexe file.
1.Install Squish Coco 2.Set the Environment variable PATH to have the .Net framework location: C:WindowsMicrosoft.NETFramework64v4.0.30319 3.Copy the WebApplication folder. From C:Program Files (x86)MSBuildMicrosoftVisualStudiov11.0 To C:Program Files (x86)MSBuildMicrosoftVisualStudiov12.0
4.In the project to be tested, let's call it the NAS Project, in the Common Service for example, add the following two methods, for starting the Coverage Scanner and saving the Coverage report.
/// Service added just to begin the coverage process
public void StartCoverage()
{
//var a = "asdasdsdasda";
#if __COVERAGESCANNER__
CoverageScanner.__coveragescanner_init();
#endif
}
/// Service added just to stop the coverage process
public void StopCoverage()
{
//var b = "asdasddsadassadsad";
#if __COVERAGESCANNER__
// set the filename of the .csexe file.
// NOTE: the location must be writable.
CoverageScanner.__coveragescanner_filename(@
"C:\SCoco_res\NAS_SCoco_CodeCoverage_results.csexe");
CoverageScanner.__coveragescanner_save();
#endif
}5.When Squish Coco is installed on a particular machine, the Add-on for the Visual Studio is also installed. In Visual Studio we may find the option for Code Coverage in the Tool menu.
We select Tools ->Cove Coverage, select the coverage options, select a build configuration, select all projects, and set Enable Code Coverage.
Rebuild the NAS solution, so that project gets configured to use the Squish Coco Coverage Scanner. In the Build options for NAS Services, we may notice the following options:
6.Modify the SOAP UI test suites so that:
The first SOAP UI request from the Test suite calls StartCoverage
7.Follow the steps from the following schema in order to obtain the code coverage report for the executed SOAP UI Tests.
8.First and second steps, from the above schema, may be put in a .bat file
CodeMain NAS_SCoco_CodeCoverage_instrumented.batSome parts may be excluded from the instrumentation, as they don't have to be analyzed for code coverage, like:
/Code/Main/NAS.Vendors.Data.Repository/Mappers
/Code/Main/NAS.Vendors.Domain.Manager/MappersThe content of the .bat file - NAS\SCoco\CodeCoverage\instrumented.bat :
REM build the NAS project with SQUISH COCO Enabled; REM exclude specific folders and files
REM build the NAS project with SQUISH COCO Enabled; REM exclude specific folders and files
cd C:\Workspacenew\Code\Main
set PATH=%SQUISHCOCO%\visualstudio;%PATH%
set COVERAGESCANNER_ARGS=--cs-on
--cs-exclude-path=C:/Workspacenew/Code/Main/NAS.Vendors.Domain.Manager/Mappers
--cs-exclude-path=C:/Workspacenew/Code/Main/NAS.Vendors.Data.Repository/Mappers
msbuild /p:UseEnv=true NAS.sln /t:ReBuild
REM copy all .csmes generated files into NAS.Services.REM Host\bin
for /f %%G in
('dir C:\Workspacenew\Code\Main\*.csmes /s /b') do copy %%G "C:\Workspacenew\Code\Main\NAS.Services.Host\bin"9.From the above schema, step 7 - Generate HTML code coverage report - may be executed from a bat file: NAS\Generate\SCoco\CodeCoverage\Report.bat
REM create Squish Coco html report
REM create Squish Coco html report
cd C:\SCoco_res
copy "C:\Workspacenew\Code\Main\NAS.Services.Host\bin\NAS.Services.Host.dll.csmes" "C:\SCoco_res"
cmcsexeimport --csmes=NAS.Services.Host.dll.
csmes --csexe=NAS_SCoco_CodeCoverage_results.csexe --title=NAS_SCoco_CodeCoverage_Result --policy=import_duplicates_and_empty
cmreport
--title="NAS_SCoco_CodeCoverage_Report" -m NAS.Services.Host.dll.csmes
--html=NAS_SCoco_CodeCoverage_Report.html10.Here are some examples of code coverage results when some SOAP UI test cases pass, and some fail. Because the test cases fail, some parts of the source code are not executed anymore.
11.Code Coverage HTML Report
by Ovidiu Mățan
by Ovidiu Mățan
by Mihai Varga
