TSM - Dev/Test challenges in complex Enterprise infrastructures and possible solutions

Mihai Tătăran - Microsoft MVP, Co-organizator ITCamp, GM @ Avaelgo

This is a continuation of the article “What are the challenges of Dev/Test/DevOps and how Microsoft Azure can help”. This time we will go into details explaining which are the organizational and technological challenges, and which are some approaches, processes, and solutions to create efficient Dev/Test environments.

Dev/Test Objectives

Efficient software development process

There are some obstacles in the path to efficient software development, some of technical type and some at the organization level.

As I mentioned in the article referenced in the Introduction, separation between development teams and infrastructure teams can bring a lot of friction. On one side, the development team is slowed down because creating test / staging environments takes a lot of time, and on the other hand the infrastructure team’s work is affected because of the environments’ costs which remain unused or underused for a long time (24 hours per day vs 8-10 hours which could be enough).

A solution could be providing the development team the means to create test environment in a self-service manner. They will not depend on the infrastructure team, anymore, and the whole deployment process is accelerated. On the other hand, they become responsible for stopping and deleting the environments when they are not used anymore.

Minimizing costs

The infrastructure team changes its role: from the team which creates dev / test / staging environments, they transform to the team which define how those environments should be created, using features like Azure Resource Manager and Desired State Configuration (more in the dedicated paragraph). So, the infrastructure team defines templates for environments creation, and those templates are then used by the development team who is actually creating those environments.

Also, the infrastructure team will continue to maintain and monitor the environments in Azure, and possibly still hold the responsibility over their costs.

Dev/Test with Azure for complex solutions

If in the previous article we have shown how to realize a simple Continuous Integration solution with the help of Visual Studio Team Services, now we will focus on more complex scenarios.

When we discuss about enterprise solutions, they are more complex than a simple web application linked to a database. We are talking about complex architectures, which physically map on complex infrastructures: layers upon layers, Virtual Machines (VMs) or PaaS instances behind Load Balancers (LB), internal Load Balancers, reliable messaging mechanisms like Service Bus, caching, local networks (VNETs), security configurations (Network Security Groups – NSGs), etc.

From our experience, exactly this kind of complexity of the environments – including the dev / test / staging, etc – determines a very inefficient software development process. Mainly because it is not easy to create those environments fast (automated), and identically every time.

Azure Resource Manager: environments creation templates

Microsoft Azure comes with a very clever way to create resources: Azure Resource Manager. We can specify with the help of a JSON file which are the components of our environment (it is called Resource Group in Azure), and ARM is responsible for creating those components, grouped in the RG, by „executing” the JSON template.

For example, a complex SharePoint production environment is not a simple one. Creating it component by component is a process very prone to error, but creating a JSON template first guarantees creating an identical environment – Azure RG – every time it is needed. These could be the components of a SharePoint environment in Azure:

You can find the JSON template here. Because JSON templates can become very complex and it could be hard to edit them by hand, there is a tool inside Visual Studio dedicated for editing JSON files:

Installing complex solutions in Dev/Test environments

The BUILD and RELEASE steps in VSTS are the ones already explained in the previous article. The next ones are more interesting for complex solutions.

Environment creation in Azure

This is the step where from VSTS we trigger the environmnent creation in Azure, and where we use the JSON template for dev / test / staging environments. The output of this step is a Resource Group in Azure, which is ready to receive the actual solution deployment.

Solution Deployment

After the RG has been created, we can install the solution with all its prerequisites.

A very useful mechanism to install prerequisites on VMs is PowerShell Desired State Configuration (DSC). With DSC, we can specify how the VMs should look like instead of running PowerShell scripts or installing and configuring features one by one. We simply describe the state we wish the VMs to be in, and they will recreate their state using one of the two approaches:

A very relevant feature of DSCs is that when we have the same VMs in different environments (dev, test, staging and even production), the same preresuites are grouped in the same DSC files, thus we are able to reuse the knowledge no matter where (in which environment) we need to create a VM.

Another relevant feature: we have observed major problems in different organizations when launching a new version of a solution into production:

Now a Staging environment needs to be created, one which is „identical” to the Production environment, to perform the last tests.

Problem: the Production environment is at the Client so how are we able to create an identical Staging environment? Obviously, this never happens and what we’ve become accustomed to is that we have a Staging environment „as close as possible” to the Production environment, sometimes even trying to guess some of the configurations. Which turns into a major issue for the final testing.

With DSC we can do it differently: we deliver the new version together with a pack of DSC files which will be used by the client to configure his VMs as needed, making sure the Production environment has the exact prerequisites needed for the solution, which are also the exact prerequisites we have tested the solution against in the Staging environment.

There are challenges at the organization level. It is not always simple to influence the client’s infrastructure. Probably the key is to try to explain in terms of costs and benefits why it is better to use the DSCs delivered by the software development team, thus obtaining a recommended and supported environment, with lower maintenance costs and with higher speed of moving the new version into production.

Installation confirmation

When we install the solution in a dev / test environment we want to notify the people interested: release manager, tester, etc. The confirmation might be an email which contains all the neccessary information for testing, like this:

Testing

The testers work from their office and the solution has been deployed into Azure, and the testers need a secure way to access the different components to test them. A simple option is to create a Site to Site connection between the local network and the VNET in Azure, thus the testers will see the solution at local IP addresses (as in the confirmation email above), of course with the precondition that all the dev/test environments (RGs) are part of the same VNET. This is a simple diagram which shows this connectivity:

Conclusion

With the help of Azure Resource Manager, JSON templates, and PowerShell Desired State Configuration, we can create a Dev/Test strategy for complex solutions, no matter how complex the needed infrastructures are. And DSC is also helpful for replicating the exact same prerequisites from Dev/Test to Production environments, thus reducing the time needed to migrate to a new version.