“Real knowledge is to know the extent of one's ignorance.”
Confucius
**
I have seen in my 15 years career as a developer, technical lead, project manager, freelancer, trainer, agile/lean/technical coach and again developer many failures of software development. I have also seen unexpected successes.
From all industries, software development seems rather unpredictable. As a client, you can ask for a feature that looks simple and should take a day or two, (eg. Add Euro support to an accounting system designed for French Francs) and it takes two months. Or you can ask for a feature that looks awfully complex, and it takes one day. What should you expect? A 10 times budget increase for the feature set that you need, or to have the software earlier than expected?
The initial human reaction to unpredictability is to add more check points, more process, more management. After all, factories can produce faster when they're better managed, optimized and automated. Why not software development?
I was working in an outsourcing company as a technical lead when the management decided to get a CMMI appraisal. CMMI, in case you're not familiar with it, is a maturity model for software development. Level 3, the one that this company was appraised for, is focused on having a standardized and measured process. Level 4 is about continuous improvement based on metrics.
Being involved in the process, I learned a lot about CMMI. CMMI is not bad at core, but its requirements can be interpreted poorly. A company could easily go bankrupt by interpreting the requirements ad litteram. The easy way out of a CMMI appraisal is to take each requirement and add a document for it. This would create so much work that either the production has to halt or the company has to hire more people just to write the documents.
Happily, the Quality Manager took a different route. She read the CMMI requirements and found the simplest ways to respond to them. To understand the context better, CMMI requires that you offer a proof for following certain practices, but it doesn't specify what kind of proof. The proof can be a document, but often it's enough to have a report from a tool, a photo or a checklist.
The CMMI experience taught me two things:
I was at this point a software developer interested in how to repeatedly execute successful projects. CMMI had some potential, but it could easily turn into a heavyweight maintenance nightmare that didn't ensure success.
At this time in my career, I was looking around for any type of knowledge that might help. How were other companies developing software? Are there fundamental principles that we could follow to create high quality software while keeping the customers happy? What are the ingredients for success?
I started by reading books and articles about the topic.
Fred Brooks' seminal works “The Mythical Man Month” and “No silver bullet” opened my eyes to something I haven't acknowledged before: software development is different than other industries. After all, adding people to a construction project that is late will make the project end faster. Not so when developing software; adding more people to a software project that's already late will typically delay it even more (known as “Brooks' law”).
More interesting, Brooks' law can sometimes be avoided. The reason the project is delivered later is that people joining the project need to learn what the project is about. But if they are picked so that they already have the knowledge, they might actually help instead of slowing everything down.
Steve McConnel's book “Software Estimation: Demystifying the Black Art” was yet another interesting read. The way I was doing estimates back then was basically guessing. CMMI had already exposed me to estimation methods, but the book offered more insight. The idea that we could take advantage of statistics and peer review to improve estimates was appealing and made sense from an engineering point of view.
On the other hand, software development seemed to have little to do with statistics. One of the things I've learned from my readings (I'm afraid I don't remember the source anymore) was that bugs were not distributed naturally throughout the code. My expectation was that bugs would evenly distribute in a codebase. Strangely, they aren't: they tend to cluster. This means that the areas where a bug is found should be more thoroughly checked for other bugs. But then the normal distribution doesn't apply to bugs in code; it's worth wondering: would it apply to estimates?
While the books and articles were very interesting, there wasn't much I could conclude about successful software development. What would help: more process, better estimation, more practices, better developers...?
Nobody seemed to know.
It was at this point that I decided to move to a small product development company, to get involved in the Agile community in 7 Romanian cities and gradually turn to agile, lean and technical training and coaching.
… is unfortunately a very ambiguous, emotion-filled, marketing-bound term. Despite these challenges, I understood it as a set of principles, a set of practices and a list of potential benefits.
The benefits are on different levels:
The principles are stated in the Agile Manifesto; instead of quoting them, I will enumerate briefly my (certainly incomplete) interpretation:
Manage the team, not the individuals
Give the team autonomy and clear purpose, and let them do their work
Treat the development team as bright, motivated adults instead of children that need parenting
Continuously improve the way the team and company works. Continuously look at impediments and zealously remove them.
Feedback is the single most important process control tool. Make sure feedback is timely, has high quality and applies at all levels of software development (from code, design, requirements to working features). Most agile/lean/technical practices are about feedback.
The practices are the things most people think define Agile: daily meetings, retrospectives, continuous integration, Test Driven Development, to give only a few examples.
In my 5 years of training and coaching, I've learned about agile, lean and technical practices one thing that might come as a surprise.
Whenever a company wants to do an agile transition, the first question me and my colleagues from Mozaic Works ask is: “What do you hope to achieve with it?”.
We get various answers to this question. “We just heard agile is cool”, “our competitors are doing it”, “our customers asked us to do agile” are all answers that require more discussions. The best answer we're hoping for is “we need to improve X”, where X is typically: reduce release cycle, quality, productivity (based on clear criteria) or innovation.
By defining a clear business purpose, we can then agree on the principles and introduce the practices that make sense for the company. Sometimes, they need full Scrum implementations, other times Kanban, while others just need to start with visual management and continuous improvement.
Being agile is not important; having Business Agility is a competitive advantage.
But why would agile work? How is it useful? This question led me to another interesting conclusion.
I've learned this from a chat with Mike Beedle, one of the signatories of the Agile manifesto and co-author of one of the first books on Scrum. I felt like I've already come across this idea now and then, but that's when I finally got it.
A more accurate statement would be:
“Software is very precise, encoded knowledge”
Encoded, since it's written in computer code. Precise, because computers can't deal with ambiguity.
This simple statement has impact on every single aspect of software development. Here are some of the things I derived from it:
Software development means translating ambiguous requirements into precise code
This is equivalent with building knowledge
When building knowledge, you need all the brainpower you can get. That's why team planning and pair programming can be very efficient
How lossy the knowledge transfer is influences the quality of the result and the efficiency of the process. Best knowledge transfer is done through face-to-face communication because of instant, high quality feedback (body posture, face and hand gestures, eyes movement, voice inflexions etc). Documents provide no feedback to the reader; you need to talk to the author to understand it better. (That doesn't mean documentation isn't helpful in the long run)
People tend to underestimate the precision required. Continuous feedback is the only way to asymptotically move towards it.
A very high number of decisions are required to obtain precise knowledge. These decisions are split between the customer and the development team. Team needs a lot of context so that their decisions fit the customer's needs.
Structuring the knowledge built is something we're still struggling with in the industry.
Programmers need to have a special mind, able to turn ambiguous requirements into precise code
To wrap this up: why agile? Well, because:
Business Agility, Development Agility and Improved Productivity can be obtained by using the principles, values and practices of agile software development
Software development means building knowledge, and requires a fundamentally different approach than building nuts and bolts
Developers feel and perform better when they have autonomy, mastery and purpose (the key to intrinsic motivation according to Dan Pink) and work in an environment that favours flow (see Mihaly Csikszentmihalyi's works on happiness and productivity)
When isn't agile useful? There are situations, even in software development, when building knowledge doesn't play such an important role; for example customizations for existing software. There are business models that work and don't require agility; for example accountancy software. There are developers who are happier when they get the task and do it without making a lot of decisions; an agile environment will make them unhappy.
So, is agile the answer? I don't think so. We still have a long way to go to obtain predictability in software development. I believe though that one of two things can happen. The industry will either:
find ways to minimize the amount of knowledge building through specialization or
There's no telling yet which way we will go, but, to me, the second direction sounds very exciting and worth exploring.