Lately we all have been bombarded with everyone saying that we need to have continuous delivery and that we need to put the application quicker into production. And yes ...why not? Having the application put live into production ten times a day is cool. But then, why did it take us so much time? Continuous delivery was there from the very start of agile. It is in the first principle:
"Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."
But doing something just because everybody else is doing it or just because it"s fun or cool is not the right approach, we need to make sure that we"re doing the right thing for the business, that we understand why we need to do a certain thing and even more important, that it brings value to the business.
Through this article I will show why it"s important for business to be onboard with continous delivery, what the differences are between it and other continuous buzzwords and how to get started. Besides introducing all of the above in the context of a developer, or architect working in a company, I will also pass some remarks concerning the context of outsourcing developers which is considerably different. Down the rabbit hole we go...
Looking at the internet...more and more tools have started to appear that allow us to transcend to this new state of higher existence, called "continuous delivery": puppet, chef, vagrant, docker, go CI tool asoWhen will I learn all of them and who is going to give me time to integrate them into my already behind schedule project?
But starting with the tools and the technical coolness factor is only half the battle as they are just means to an end. I believe that we need to ask ourselves is what wewill gain if we follow the first agile principle, and the answer is feedback. In order to make sure that we recognise the value, we need to learn what the customer or business understands through it, just like we do with qualities. Heck, better thinking about it, I might even say that here value is quality and quality is value. Why do we need to recognize it? Because we need to see whether we achieve it or not, or at least how close we are to achieving it.
But why am I saying all this? Because I have seen a lot of presentations, blogs, videos lately that say that in order to succeed you need to do this, and that, to use that tool, and to connect it to that other tool; but they completely forget to mention that it is not just you the technical software architect, who learns continuous delivery but also the business. And believe it or not you can do something about it.
Please don"t misunderstand me, you definitely need continuous integration and deployment in order to be successful with continuous delivery, but a lot of times people say continuous delivery but refer to continuous deployment. To be sure we have the same meaning for the principles, this is my understanding of these words:
Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day.
Continuous Delivery is about keeping your application in a state where it is always able to deploy into production. Continuous Deployment is actually deploying every change into production, every day or more frequently.
When I first read this last statement I though: "Cool, continuous deployment is more and better than delivery!" and two years ago I might have been right. Jez Humble says: the first includes the second but the reverse is not necessarily true. Now I believe that it is more important to have the business next to you. And while you might be doing continuous deployment to your acceptance environment alone it will not be good enough. Leaving to the business the control when to put it in production is what continuous delivery is about. Even more, some companies have diminished the role of the acceptance environment going as far as deploying changes directly in production - meaning that we have a merge between the two, because of which I"m more comfortable calling it continuous delivery.
But what is an outsourcing developer to do? Most of the time, he/she has no access, or at best limited access to the business people. Until quite recently I have thought that in order to start making a change you need to clean up your own back yard first, and once the people around you will see the benefits they will want to change things themselves, hence improving overall the company. And this is not bad, but it has a catch: is it really a benefit if I see it as one? Most probably not, we are techies, we"ll spend hours on some piece of code till we get it perfect.
I think that we should ask the business to tell us which their values are. We already do it to identify the quality attributes. If you have been to any of Tom Gilbs" talks you are already convinced that the first thing you should do is to ask: "Which are your top ten values?". So your focus should first be on things that will bring the highest ROI on the top values, you build it in the product and then the second one, and so on and so forth.
Doing agile, mainly scrum and kanban, we are all used to the phrase definition of done, which we all use to have an alignment between team members and project stakeholders with what needs to be completed before a backlog item is considered done. I remember when I started doing scrum and using user stories, I believed that done meant what I need to do in order for the story to be accepted by the PO. After a few sprints I learned that the meaning of a done story was when I had the story ready to be put in production.After all, scrum says "potentially shippable" quality, which, if I look back, is the meaning of done in the context of doing continuous integration.
Later on, besides the continuous integration efforts, we looked at unifying the way we were deploying the applications in different environments. It was a complete mess: we - the near-shoring development team - were responsible for deployment and stability of testing and of acceptance environments, while the operations team on the clients" side was responsible for the infrastructure of all environments, and deployment to the production environment.
But as time passed by, we started to unify the deployment procedures and split the responsibility between us and operations; we, the developers, were releasing the artefacts, meaning that out of the Continuous Integration tools we were creating environment independent deployables, and operations were going to take care of the actual deployments in an unified way to all environments. I have to admit that it was not an easy feat, there were quite a few bumps along the way, like adding a new setting; but all in all it got us working closer to operations: one might say we were working together. And this pushed the meaning of done even further: done as in: deployed in production, actually in the hands of the end users.
I"ve recently heard a talk by Adrian Cockcroft, the ex-chief architect at Netflix, where he was saying the at Neflix they are done with a piece of code, a.k.a. software, when it is retired from production. Meaning that we are not done with the code when it was written or when it was deployed to acceptance or production, but we are all responsible for it throughout its entire lifetime: during operation, taking care of its health, maintaining it, improving it and in the end retiring it from production in a controlled way, actually having a plan and following it.
If one was to look at the lean principle he would see that they are very much in line with all we"ve been discussing above, using a cycle of small increments from the business to the customer, looking at feedback from the usage of the product and adapting, growing it organically.
Through each cycle you deliver a small increment, but you need to choose wisely so that you pick the one which brings the highest value to the customer. Once it is used, you look for feedback through different metrics, and learn, only after which you decide what to build next. I"m not saying that you need to wait for a few weeks to get value out of the measurements, you need to have work planed in advance, and use what you learn in your planning. Still, if you see something problematic in production, something affecting your users dramatically, you can quickly identify it and react, dropping everything else and fixing it. Important here is that the expression "small increments" doesn"t mainly refer to time, but to the actual amount of changes, this means that you must also be careful about your work in progress (WIP). Kanban boards are a good start for visualizing the work in progress.
Another lean principle is building quality in, in your work, your processes, everything you do and produce. You will not be able to let the "Deploy Now!" huge red-button in the hands of the business if you do not have automatic mechanisms of assuring the quality of your product. However, in order to be able to automate it, you need to know what quality actually means. Don"t fall into the usual traps: What quality means for you is not what the business understands through quality. Quality is never the same for two different products. Quality costs, hence, make sure you know the thresholds of good enough. And last, what quality means will change over time - during initial launch you might need lesser functionality but more eye candy; later on, you will need to look at scalability as your user base grows. Thus, go to the business, ask what value is for them in the current context, what we could change in the future, build automatic ways to assure that the system is performing at the needed level for each quality attribute.
Lean also relies on eliminating waste, identifying bottle necks and parts that do not produce value. Once you have the process in place and the measurements to obtain feedback and learning, you will be able to see what part of the system or process is limiting you. As soon as this is identified, it is clear that this must be the highest priority for you to fix as part of the continuous improvement efforts. It is not worth improving anything else, since this will generate only more work to accumulate at the bottleneck.
Decide late! Taking good decisions requires detailed knowledge about the topic, which actually requires time, meaning money. So you will want to push the decision making until it is mandatory to do it. This way you reduce the risk of having to throw away work because priority changed, or, as time passes, you get a new understanding of the topic. Just make sure you don"t get into analysis-paralysis, postponing taking the decision because you don"t have enough information.
As mentioned several times up until here, lean relies a lot on learning, when making decisions, when deciding priorities. In other words, this means that you need to look at the past and continuously learn both from success and failure. Therefore, beside setting up metrics for the application and way of working, also put in place mechanisms through which you learn. Sprint reviews and retrospectives are a good start providing feedback for your product and process respectively, but there are other things you could do. Look at failures in production and perform postmortems, not to assign blame, but to learn what you can do in order not to repeat failures. Also don"t limit yourself by looking at your project only, look throughout your entire organization, and in the industry, but remember not to start doing something just because others are doing it , see why they are doing it and what problems it solves for you. Second to this, you also have the responsibility of spreading the things you learned.
Lastly, but not least important, lean relies on an empowered team. A team needs to be led, not micro-managed at every aspect of the development process, let them organize themselves, trust them that they will do all that is humanly possible for the success. As a manager, take a step back and look at the big picture, see where the bottlenecks are and make them visible to the team in a way that they learn. Trusting in people is the best way to let them exceed your expectations.
Therefore, continuous delivery is about delivering value to the business through quality software in fast cycles which in turn allows you to improve using feedback loops. I am actually a huge fan of TDD, and I practice it whenever I get to code no matter what programing language I use: Java, Nodejs, Xquery for Marklogic development, Javascript for client side development, Bash for scripting deployments - I first look at how I"m going to use it, then I put the tests in place and then I start implementing. But TDD is just the first level of feedback, continuous delivery assures that you have an action -> feedback cycle at the business level, and in order to achieve it, you need collaboration between business and technical people.
Up to this place, I have talked about why and how adopting continuous delivery helps an organization, especially in a context of outsourcing your clients" organization through a mutual feeling of win-win partnership. In this section we will look at how we will implement it: which are the principles and the practices.
First of all, create a repeatable, reliable process of releasing the software. In order to achieve this, you need to have a look at how a business idea, a feature, gets to be released to the user. And don"t you think that this is limited to picking a story, implementation and deployment. No! You need to start with the birth of a business idea, how it is defined, priorities versus the other business ideas, how you choose as a development organization to assign resource, to plan future work, etcetera, etcetera. At first you will be overwhelmed by the amount of information, or the mess it might seem, but learn to visualize it, setup boards for each stream, each stage will represent a column, and most importantly follow the lean and agile principles.
When you have this picture the feature travels as described above, you will be able to visualize your bottlenecks, what slows you down. These are the places that you need to focus on. Once you identify them you need to learn what makes it slow, why it takes so much time. In order to fix it: initially look at reducing waste - make sure that you do only what must be done, and for what is left automate as much as possible - so that it takes less time and is reliable.
Because you will want to move at fast pace for getting smaller increments, when something breaks, you can easily see what has changed since it was working correctly. In order to obtain this, you need to version everything, not only code, but also configuration, requirements, deployment procedures/scripts, APIs, anything else that you might think of, and have ways to relate requirements, changes, deployables, configuration, etc..
If something hurts, meaning that it is a pain to do it, it takes long, or you are afraid to do it, then the more often you will need to do it. You can start with the deployment process, but it is not limited to that. Think about what failure you can have and do fire drills, rehearsing how you would react in case of a real problem. I guess you all heard of the Chaos Monkey from Netflix that kills instances randomly in production. Not only did Netflix learn how to deal with service downtime, but they automated dealing with it. Even more, they created an entire army of monkeys, the Simian Army, which kills things randomly at multiple levels of the infrastructure.
Since you will be automating as much as possible of the validation of the software, first you need to make sure that you are looking at the right things, meaning that you are looking both at functionality and at the needed quality attributes. You will have to assure that they are measurable, putting in place metrics and gates with the needed level of each quality attribute, and, if one fails, then you stop everything and fix it.
Just because the code works as you need it to, it doesn"t mean that your job is done yet, it needs to be placed in the hands of the users. And once it is there, you need to make sure that it works correctly, and quickly react if changes are needed. Based on feedback, you will make improvements, and extend the functionality. Developers need to work together with operations; some organizations have given up on this separation and now have teams around functional verticals, which are responsible for new developments and the good going of the systems in production.
The things mentioned above are not things that you can just say "there, I am done!".No, it is continuous work: you need to see how you can make each part better.
We are technical people, what is up to us? Well, we have made sure we are aligned with the business, we are working on the right things and focus based on the right priorities, and now we need to build the infrastructure to support all of the above. We are going to work on the deployment pipeline.
A deployment pipeline is breaking up your build into stages. Each stage provides increasing confidence, usually at the cost of extra time. Early stages can find most problems yielding faster feedback, while later stages provide slower and more through probing.
We cannot prove that the software is defects free, but we can try our best to catch all of them. My recommendation is to start with the identification of your current deployment pipeline, I"m sure you have one. Visualize it, see where work accumulates, identify how things are priorities, what type of work you do. Make sure that everything is in version control: code, configuration, assemblies, and even artefacts that come out of your build system.
Once you have all of the above, the next step is to standardize your deployments: meaning to build things once and then to deploy them in each different environment, using the same mechanisms (tools and procedures). This will mean that your deployables are environment independent, your configurations are versioned, and your build and deployments are automated. For this, you will need a build system - which will take care of compiling, running unit tests, assemble environment independent artefacts and which will publish these artefacts to a repository. Secondly, it will also need to publish reports through which you can monitor the entire thing, store the reports in the repository, too, so that you can look at trends during your retrospective, add alerts for dropping bellow thresholds through email, twitter, irc whatever works for you.
Your deployment procedure needs to be a repeatable process, meaning automated. Most of the times, this is the most painful part of any development or operations team, the more often you do it, the less painful it will become. The best approach is to use the same automated procedure for deploying in all of your environments: dev, test, acceptance, pre-prod and in the end production. This helps because it creates a repeatable process and it increases confidence in a package. Others call it build, version, or better we should start calling it a pipeline instance.
Now, you make sure that more and more automated steps increase the confidence in the build; and that the manual checks are as far down the piepline as possible. You can start by looking at automated acceptance testing, integration testing of your APIs. You don"t do the entire application, you start with the sanity tests, and once these are done, then you pick up other parts that have a high risk of either breaking or represent a risk if they are broken. You do this both vertically and horizontally focusing on limiting the risk that the build is not good enough.
In parallel you can start making sure that what gets to testing is good enough, build-in quality and minimize waste for testing, do more unit testing, add code coverage, static code analysis and that you have thresholds for them, so that the build fails if not met.
Next, you should make sure that your pipeline is automated: once you have a change-set committed in version control, the entire pipeline instance is started, and when a step has passed, it automatically starts the next one. As I said before you need to see what is slowing you down, and you focus on each new cycle on the biggest bottle neck, heck, you might even give up sprints and move directly to a Kanban having on top the thing which will bring highest value, meaning here a faster pipeline.
Your work is never done, meaning that once you have these in place, look at what can be improved, where are the breaking points. When something fails, add new stages in your pipeline, add more test or improve thresholds for passing the build.
Also do not imagine that you can stop all business projects for 6 months, and work solely on this. Your business needs to survive, there is no pause button. You need to make sure that you build this together with your normal business projects and with each newly added step you can show the value that it brings to the company, no business man will say no, make it part of the cost of building stuff.
In ISDC we have IDAM, ISDC Defined Agile Model, a group that tries to help ISDC and teams in ISDC to be better by following the agile mindset. At some point we looked at why continuous delivery is so important for agile teams, and how to do it, beside recommendations and we came also with an image of what can be done under the umbrella of continuous delivery. It is not the ideal, not the best, not even the recommendation, but is a sample of a pipeline that has everything we considered do-able at that time, so use it as a menu for picking ideas.
One might say that all this is a bunch of blah-blah, that it doesn"t bring any value if not put in practice, meaning that we also need to understand the tools and it requires time to be implemented. I believe that to make sure that we bring real value to the business and not just technical beauty, we need to be aware of all I have stated above.
However, no theoretical advice compares with the experience gained in practice. Due to this difference in perception, I promise to come back in one of the next issues with a follow up, describing a case of moving a project to continuous delivery: what tools were used and which were the benefits, and, of course, the newly discovered best practices/ [or] the new lessons we have learned.
by Ovidiu Mățan
by Dan Suciu
by Ovidiu Mățan
by Sorina Mone
by Ovidiu Mățan
