In this blog post I’m taking a deep dive into Agile Development. We’ll see why and how the Agile software development approach:
- Creates better software that exactly matches the needs of the business.
- Allows that software to produced faster and at lower cost than the traditional ‘Waterfall’ software development approach.
- Creates an ongoing partnership between the business and the software’s developer.
The bad old days
For many, many years IT projects have been a frightening proposition; notorious for running over time and over budget.
There are usually just two main causes:
- It’s extremely difficult for any organisation to fully explain every nuance of their business processes. So the true scope of the work is unknown.
- Large software systems are complex. So, it’s essentially impossible to look at the overview of a system required and accurately estimate a time.
The traditional solution to this was to break the work down into ever smaller pieces. Eventually this yielded pieces small enough to accurately specify and estimate a time for. This was then done for every feature and the results compiled it into huge specification and software-design documents. The quote was then prepared based on those documents.
This approach works to a degree, but it is hugely time-consuming to do, which leads to a series of problems:
- You’ve spent so much time writing down every tiny detail that you might as well have just written the software. Effectively you’ve nearly doubled the duration (and therefore cost) of large parts of the project
- The whole process has taken so long that the business’s requirements will have evolve in the meantime.
- Most people can’t visualise how a piece of software by reading words in a document. They need to see and touch it.
- Large, dry documents are hard for humans to read. There’s the very real chance nobody will really understand the specification. Or even read it.
Furthermore, software developers know from long experience that even with a great specification, additional requirements and misunderstandings will still arise. Therefore they include large amounts of contingency in their prices to allow for this. This is a terrible approach, it means that:
- If the work goes as expected the client pays more than they should.
- If the work goes badly the software developer loses out. They then become reluctant to do further work on the system in future.
The only certainty of traditional pricing is that the price paid won’t accurately reflect the work done!
The Agile approach
What is Agile software development?
Clearly the approach above isn’t ideal for either client or software developer. It’s problematic in a large organisation, and it’s even less desirable in small and medium sized businesses that don’t have internal teams of business analysts to explain the operation of the business and IT departments to manage the software procurement process.
Because of this, over the last ten years much of the software development industry has moved away from the old approach (often called ‘Waterfall development’) to the ‘Agile’ methodology.
The idea of Agile software development is to stop trying to know everything about the software in advance. This eliminates the need to try and write a big specification for everything in advance.
Instead the goal is to produce an actual working system as quickly as possible.
Building the Backlog
Instead of writing a detailed specification, the Agile process starts by adding the broad objectives of the project to a master list, known as the Project Backlog. The organisation then prioritises that backlog. We recommend setting an order that:
- Gives some early quick wins so people can see progress
- Gets the biggest pain points being dealt with early
That approach will help engage people in the project.
Working in Sprints
The development work is then done in small increments called Sprints. Sprints can vary in length, but usually they’re between one and four weeks in duration.
A typical sprint looks like this:
- The business selects which features are the current priority
- Those are added to the backlog for the sprint
- The business and the developers discuss the exact requirements for just those features.
- The developers then create an initial rough working version of the feature
- The users check that version fulfils their needs and tell the developers of any changes needed
- The developers then produce the final version.
The key here is that scope evolution is an expected part of the Agile software development process. When the users see a feature up and running they will often realise ways it can be improved. Those improvements are incorporated and the optimum solution (but no more) is then delivered.
At the end of each sprint the work is reviewed. Any unfinished items are moved to the next sprint. New items from the Project Backlog are also added to the next Sprint’s backlog. The process then repeats until all items from the Project Backlog have been completed.
Planning and monitoring ongoing progress of the Agile software development process
So far, we have only defined rough estimates for how long work items will take. It may therefore seem like it will be difficult to track the progress of the project. Agile has a way of dealing with this too. Each item of work is given an estimated size (say extra-small to extra-large) or a number of Story Points.
Assessing how much work is involved using Story Points
Story Points are an arbitrary measure of the complexity of a piece of work. Story Points don’t describe an exact amount of time in hours and minutes. Instead, the developers and the business agree a scale. They may for example decide that 10 story points is a quick job, and 100 story points is a bigger job.
At the start of any project it’s hard for the team to assess exactly how long work will take. Factors like
- the complexity of the system’s interaction with other systems
- the speed of the business in answering questions that arise
- the relative speeds of the developers on the teams
are all unknowns at the beginning. So initially story points are only assigned to the work in the upcoming sprint
Calculating Project Velocity
At the end of each sprint the developer adds up the total number of story points completed in that sprint. This gives a measure of the number of points that can be completed per sprint, known as the Project Velocity. This Project Velocity is then used to determine how many items should be scheduled into the next sprint.
After three of four sprints the team naturally form a good understanding of the complexity of the project. They then arrive at a common understanding of what constitutes a story point and so the Project Velocity becomes accurate.
At this stage, the team can also choose to assess more of the future work items in greater detail. They can now assign a number of story points to those items too.
Using Project Velocity to predict delivery dates and monitor performance
Once that’s done the amount of work is then known (the number of story points) and the speed of the team is known (the Project Velocity) which means that the likely delivery dates of new features can be calculated. Work can be reprioritised to ensure that features are delivered in advance of when they are required.
The Project Velocity can also be used by the team to monitor their performance in real-time and to check that there aren’t issues causing the amount of work delivered to drop off. This means that issues that may cause delays are identified sooner than in a traditional project.
Changes to priority and scope are an expected and welcome part of Agile software development
The Project Backlog can be reordered at any time to allow for the changing priorities of the business. New items can be added into the Project Backlog whenever the business identifies new requirements. This process means that the most urgent features are always delivered first.
The exact requirements for each feature are determined just before coding. This eliminates the possibility of requirements changing between the specification and coding stages.
But how do you know what it’s going to cost?
The sticker price is never the final price
The biggest hurdle to overcome when adopting an Agile approach is the shift away from the perception that you know exactly what you are getting and for exactly what price in advance of the work. I say perception here because even with a traditional pricing model (with a specification and a quote prepared in advance), the number on the quote is rarely the final price.
Additional requirements are always discovered during the programming of the software which leads to the price increasing and the work taking longer than expected.
This is frustrating for all involved:
- The staff charged with running the project look bad for failing to keep the costs and timescales under control
- The software developers come under pressure to deliver faster. This often reduces the amount of testing done and therefore the quality of the product delivered.
Embrace the certainty of change
Agile software development does away with all the uncertainty and risk of the above.
Instead of trying to calculate the exact scope and price, the initial requirements for the project (the initial Project Backlog) are considered and given a ballpark estimated duration. This allows an approximate price range to be calculated for the work as it is understood at that time. This is a top-level assessment, performed using the developer’s experiences from past, similar projects.
This ballpark is then discussed to see if it fits within the business’s budget. If so, work commences.
If not the requirements list is reassessed to see if there are any less urgent items. Where those items can be left until the next budgetary period (or just not done at all) then the project can proceed.
It’s important not to get too caught in the detail of the features during that ballparking stage. The idea is only to check whether the cost of the desired work desired is in the same ballpark as the level of budget available.
I’m not writing the developers a blank cheque…
Committing to a specific expenditure without firm details of exactly what you are getting in return can certainly feel a bit like writing a blank cheque. But, that isn’t the case. Admittedly, in a traditional model a software developer gives you a list of what will be done for a certain price, but that typically follows weeks of discussion and needs analysis. Therefore, by the time the specification and final quote are delivered a substantial commitment has already been made. That commitment is often financial, as needs analysis is usually chargeable work, but even if the analysis has been done for free you will have committed large amounts of your employees’ time to getting the developers up to speed.
Things are very different with Agile software development. Rather than having a large period of analysis before any code is written, in an Agile project the first sprint is delivered within weeks of the outset of the project and at that point the business gets to see actual working software. The business can then assess whether the features promised for that sprint were delivered and whether the work is of good quality. If you don’t like what’s being delivered you can simply pull the plug at the cost of a few weeks work.
Compare this to a traditional model where you might not see working code until months of analysis and development have taken place and already been paid for. This is expenditure that almost certainly couldn’t be fully recovered if the system isn’t as hoped when it is finally delivered as the developers will have already done the work.
So, whilst Agile can feel like a leap into the unknown, the time and financial exposures are actually much lower and the continuous nature of the delivery of the system means you always know for certain that you are getting what is promised.
Why would we do this? How does Agile software development benefit our business?
Hopefully this document has already highlighted a lot of great benefits from the Agile approach, but there are plenty more too:
Agile software development delivers working code within weeks, meaning the business’s most pressing requirements can be addressed quickly. Not only is the initial delivery faster, but each feature takes less time to write because the developer isn’t writing a detailed specification in pseudo-code, and then subsequently writing the actual code.
Software developers sell time, so if the development is faster then it’s also cheaper.
But, that’s not the only reason it’s cheaper: For any company whose product is time, the biggest fear is having a resource that goes unused. That means that software houses have to build allowances into their overheads for periods of under-utilisation of resources. Agile projects where a set budget for a year is known in advance eliminate much of that potential for under-utilisation. That reduces the overheads and allows the hourly rates to be lower. This is a true win-win as the developers can achieve the same margins whilst charging their clients less.
Client and developer become a team
In traditional software development there’s a long initial period where the software developer’s analysts attempt to extract the full requirements from the business. Those requirements then become the specification which the project is judged against. That inevitably creates an adversarial situation; people will understand what’s written differently, which soon leads to an ‘us and them’ feeling.
In Agile that friction disappears. In Agile software development the developers work with the business team gathering their requirements just before programming the feature, instead of the developer’s analysts only speaking to the business’s managers once at the outset of the project, .
The feature is then delivered as early as possible (usually within days) so the users can check the feature is as required. Often the users will then realise a feature would work better if it was tweaked. Because there’s no hard and fast specification these changes can be immediately incorporated, which means both the users and the developers can work towards iterating the optimum solution. That close, ongoing working relationship swiftly builds a strong feeling of partnership between the organisations.
You get onsite business analysts for free
The collaborative nature of Agile working means that rather than only seeing the software developers at the start and end of the software writing process, they will instead be visiting site at the beginning of many of the sprints (to work with the team on the requirements for that sprint).
As well as bringing the team members closer together, this has another benefit, which is that the developers see more of how the business carries out its day to day processes. This can be invaluable, it not only improves the developers’ understanding of the business but it also allows them to spot areas where other software systems and features they’ve written for other companies can provide positive return on investment to your business. This effectively gives your business an ongoing business analysis and needs analysis team without employing additional staff.
You pay exactly the right price
As we’ve already seen, with the traditional software development it’s a certainty that the price paid won’t actually accurately reflect the amount of work done. With something as complex as a large software system the time estimates will always be some amount wrong and so either the client is paying too much, or the developer is getting paid less than they hoped. Because nobody likes to be consistently underpaid the developers will of course start building in contingency to their prices too.
That seems like a terrible outcome for both sides as somebody always loses out. With Agile software development there’s no estimating, you pay for exactly the time done, so the business pays exactly the right price. And of course there’s no contingency cost either.
What’s more, in the traditional model an out of date specification can lead to features being written that will never be used, because the business process has changed by the time the software is eventually finished and released to the users. That simply can’t happen with the on-demand nature of Agile development – an unneeded feature will never rise to the top of the backlog list.
Engages your staff. Continuous improvement.
The number one success factor for a software project is user engagement. The business managers and software team can write the best system in the world on paper, but if the users don’t engage with it and commit to learning it, the system will never improve the business’s operations in the ways hoped.
Long experience in delivering software tells us that users often fear new systems, they see their jobs being automated away or they think they aren’t being listened to and so swiftly feel the system is being foisted upon them by people who’ve never actually done the work they do.
Agile software development is brilliant for getting rid of this type of ill feeling, the users become a part of the team that decides the requirements of the feature at the start of the sprint and they are the ones to test and refine the feature as it’s delivered. That means they are fully engaged with the feature from the outset and get exactly the functionality they need to improve their efficiency.
It’s not just about delivering the right software though, that process of engagement is rewarding for staff members; it’s a new skill they have learnt and it shows them the business recognises their importance and the value of their ideas. The time you freed up by the new process efficiencies can then be used for more creative thinking by staff to find further ways to for the business to innovate and improve.
You get exactly the software needed, and no more.
Many ideas are discussed at the start of a software project and in a traditional project it can be hard to choose which are really needed. In Agile that process is automatic, the Project Backlog is continually reprioritised and the unneeded features simply never get to top of the list. What’s great about this is that it means that all ideas from any staff member can be welcomed and added to the backlog; the best ideas will then be implemented and the others won’t.
It’s impossible to go over budget
The budget is fixed, it’s the features which are delivered that change. That statement may ring alarm bells, it sounds like you aren’t getting the features promised for the price, but, the thing to remember is that in Agile software development the most important work gets done first.
The understandable fear for the business managers here is that they could end up in a situation where budget was assigned on the expectation of the delivery of certain features, but the evolution of the project priorities then caused some of those features to be pushed back. In reality however that’s very unlikely to happen, the headline features the budget was assigned for will almost certainly have been top of the priority list and so those features will not only have been done, but they will have been done first and delivered immediately (rather than waiting until the entire scope of work was completed).
It’s the secondary goals, nice-to-have and no longer required features that may not have been done, replaced instead by other more urgent items identified by the business during the period. The project’s performance doesn’t tend to be assessed on the secondary features, and the fact that other more urgent items were achieved within the original budget tends to mean that the project is seen as over-performing rather than under-performing. The less important features that are still required do get done, just in the next budgetary period.
For non-IT staff trying to manage IT projects this is expenditure security is a real boon as it eliminates the constant concern about overrunning timescales or costs, whilst still ensuring the headline features are delivered faster than would be the case with traditional software development.