"Know your enemy and know yourself and you can fight a hundred battles without disaster" said Sun Tzu some 2500 years ago. I will not review his "Art of War". Today I will talk about Agile and its most used flavour these days: Scrum. Is the Scrum/Agile movement the enemy of Software Development Craftsmanship? Is it a creativity killer? What about the team's productivity?
Throughout mankind, there have been many prophets claiming to have discovered the absolute truth. Their ideas gained traction, some being still valid to date. The best example that comes to my mind is Moses and the Ten Commandments. But, when we check how those ideas were put into practice, things did not quite follow as expected. Think about the crusades or the Spanish Inquisition.
If we look at the world around us, we can see it changing faster and faster. For the last 250 years, we have seen the horses replaced by steam engines. After that, they were replaced by internal combustion engines that were soon to be replaced by electrical ones. Workforce, workforce management, production processes, evolved as well: skilled workers were put together in assembly lines and, nowadays, robots are replacing us, whether we like it or not - see Peter Leeson at ITCamp 2015.
How is the digital revolution changing the already fast-spinning world around us? The digital revolution began some 50 years ago, but our industry is very much in its infancy (Ovidiu Suta in TSM). A team in the IT Industry is no more than one engineer, 2-3 foremen and 20 workers that work in shifts, 24 hours a day. Teams are now made of 20 engineers! Yet, the workforce management did not keep the pace. Therefore, work is still organized pretty much the same as Mr. Ford did at the beginning of the 20th century. Moreover, the organizational hierarchy still resembles an industrial-era factory. This is one of the reasons which make many software development projects be delivered late (if ever). Gradually, hardware and software became a utility -same as water or electricity supplies. Customers get to choose their suppliers, sometimes located on the other side of the world, making it even harder to survive.
Fast forward to year 2001. Another great, revolutionary idea emerged in the IT world. It was a huge step away from the traditional way of working. Remember the iconic image of God in the form of a burning shrub speaking to Moses? Look now to the background image from the http://agilemanifesto.org/ site!
Was this step taken in the right direction? Let's have a look, taking into account the 15 years that passed since that day.
In the early days of the Agile movement, the simple principles stated by the Agile Manifesto created expectations beyond anything humanly possible, increasing the pressure on an already stressed industry. Some entrepreneurial consultants smelled the opportunity for some (easy) money and pushed the snowball down the hill. As a result, most managers jumped aboard the rolling train, mostly because everyone else was doing it, and forced it down to their organization.
The first Agile principle states that individuals and interactions are valued over processes and tools. If we look at the Scrum framework diagram, isn't this a process description diagram? This could easily be the ideal description of a product development process in a factory, any time before WWII. It assumes that the PO can stand between the team and the swarm of "stakeholders" and translate everything into a language that the engineers can understand. It assumes the deliverables do NOT change during the sprint. It assumes the estimates are always accurate and the team will deliver what it committed to. It assumes that the little blocks (stories), that are completed in one iteration, are "finished work" and that they can be delivered as such. Of course, this may work in a perfect world. I should also mention that it is virtually impossible to develop software without modern day tools (IDE, versioning control, etc)
The second principle values working software over comprehensive documentation. Think to the popular image that depicts a skateboard that turns into a scooter, then into a bicycle, ending in a sport car. Does anyone here think that if you strap an engine to a bicycle you can sell is as a motorcycle?! Here is another principle: do it right or do it twice! And about documentation…. Chances are you already work on a project started by someone else, perhaps in another country! Don't you wish you had some documentation when you started? I think the keyword here is "comprehensive".
The third principle is about customer collaboration over contract negotiation. Collaboration… what a beautiful word! People have been collaborating for aeons, isn't it? This is how language was invented and the pyramids got built. Unfortunately, "we" and "them" speak different languages.
And the fourth principle (and this is my favourite): responding to change over following a plan. What image comes to your mind first when I say "cross-cutting, prepared, self-organizing, and responsive team"? A small commando unit that is able to stick together and complete the mission in hostile territory as quickly as possible or a bunch of nerds behind computers? "Before the Agile fad, 'Scrum' was a term sometimes used for what corporations might also call a 'Code Red' or a 'War Room emergency'. This is when a cross-cutting team must be quickly built to deal with an unexpected and, often, rapidly-evolving problem."
Then there is the instruction manual - the twelve Agile Software principles behind the Agile Manifesto:
* Our highest priority is to satisfy the customer through early and continuous delivery of valuable software - I could not argue with that!
* Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage - Unfortunately this principle backfires and it goes against the "rule", by saying the sprint scope is fixed. The Product Owner or the customer is not always willing to wait for the next sprint and such changes are not properly analysed. Moreover, the changes within the team / working environments are overlooked.
* Business people and developers must work together daily throughout the project and The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Is it?. Add "open space policy" to this and see if there is an increase of productivity (or if the stress of the employees is decreasing).
* Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done and Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Indefinitely? We're not machines and if code could be written by machines, it would have already been available. Does anyone here think the marathon can be finished at the same speed or pace as the 100m dash race?
* Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale and Working software is the primary measure of progress. A piece of software nowadays is a mechanism where everything is interconnected. Take one piece out and pray it will still work. Think about building an automatic watch following an iterative approach. Start building it without having the end result clearly described upfront. Could this be done? Working in iterations leads to a new challenge: technical debt, a real issue in today's world!
* The best architectures, requirements, and designs emerge from self-organizing teams. Do you think this could be done on the fly? Architecture / design should precede other activities, not the other way around!
* Simplicity--the art of maximizing the amount of work not done--is essential. - I could not argue with that either!
* Continuous attention to technical excellence and good design enhances agility. Teams are still organized hierarchically. Even if not spoken out loud, a hierarchy exists and our decisions and behaviour are (unconsciously) taking this into consideration.
* At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly. Teams cannot always self-organize. Let us think about seniority level, people that come and go, the business area that could be new for them and so on. I am not saying they should be led, but someone should be in charge and fully accountable for.
In conclusion, Scrum methodology:
* fits perfectly in an R&D project (skilled workers collaboration will help to get to the best solution). It also fits with a one-time Proof-of-Concept to validate an idea or to get the customer's attention.
* fits perfectly into small, new organizations, that struggle to get a foot on the market, or organizations that try to get into a new niche on the market.
* does not fit with large-scale / enterprise projects, not without an initial phase, where the goal and the way to reach it should be clearly defined and planned!
* is the enemy of creativity and causes more overhead than it is necessary.
So.. Is Agile / SCRUM the enemy of Software Development Craftsmanship, Creativity and Productivity?!
The answer is - in my opinion: Yes, Yes and possibly!
Software Development is about craftsmanship and a high level of creativity is required. When one commits to enough stories / tasks to fit its availability for the next sprint, they leave no room for trying new solutions or to look out for improvement ideas. Productivity will suffer in the long run, not to mention the most-feared "technical debt" that piles up in time. I know, the word we're living in today is plain crazy - everything needs to be done yesterday, better than the competitors. But this could not be sustainable in the long term!
Can this be improved? Of course! Challenging the status-quo is what fuelled the evolution of the human race. But, in order to change something for the better, every detail should be known. To know your enemy, you must become your enemy! Learn Scrum in all its details, practice it every day, become a Scrum Master or a PO, so you can see the flaws and fix it without breaking it! :)