Agile software development delivers better software faster and at lower cost than traditional software development. It also fosters a true partnership between business and software developer. In this blog post I set out to explore why that is.
The bad old days
For many, many years IT projects have been a frightening proposition; notorious for running over time and over budget.
Whilst there can be many causes for this, the root issues are usually that it’s extremely difficult for any business to fully explain every nuance of every process in their organisation to software developers and software is so inherently complex that accurately calculating the amount of work required for large pieces is for all practical purposes impossible.
The solution to this was traditionally to break the work down into ever smaller pieces until you got to something small enough to accurately specify and estimate a time for. You then did that for every feature and compiled it all up into one huge specification and software design to then quote against. This approach works to a degree, but it is hugely time-consuming to do, which leads to the two major problems:
- You have designed so much of the software, in such detail, that you actually might as well have just written it. Effectively you’ve nearly doubled the duration (and therefore cost) of large parts of the project
- Secondly, because the whole process has taken so long, the business’s requirements have almost certainly changed since the outset of the project. It’s quite possible to get into a loop of updating the specification in response to business changes whereby all that’s ever produced is an ever evolving specification rather than a piece of software!
There’s also another problem that’s not obvious at the outset, which is that it’s very hard for most people to conceptualise how a piece of software will work from just reading words in a dry specification document. So, after spending inordinate amounts of time producing a specification, there’s the very real chance nobody will actually fully understand it, or even read all of it.
Furthermore, even with a great specification, software developers know from long experience that additional requirements and misunderstandings will still arise and so typically factor in large amounts of contingency into their prices to allow for this, meaning that if the work goes as expected the client pays more than they should, and if the work goes badly the software developers lose out, and then become reluctant to do further work on the system in future. The only certainty of a pricing like this is that the price paid won’t accurately reflect the work done.
The Agile Approach
What is Agile?
Clearly the approach above isn’t ideal for either client or software developer. It’s problematic in a large organisation, and it’s even less desirable in small and medium sized businesses that don’t have internal teams of business analysts to explain the operation of the business and IT departments to manage the software procurement process.
Accordingly, over the last ten years much of the software development industry has moved away from the old approach (often called ‘Waterfall development’) to a different model, called ‘Agile’.
The idea of Agile development is to stop trying to know everything about the software in advance and instead to get working on producing an actual working system as quickly as possible. As a by-product the idea of writing a detailed specification is abandoned altogether.
Building the Backlog
Instead of writing a detailed specification, the Agile process starts by adding the broad objectives of the project to a master list, known as the Project Backlog. That Project Backlog is then given an initial prioritisation by the organisation based on the business’s most pressing needs at that time. Typically, that initial ordering will try to secure some early quick wins to get the biggest pain points being felt by the organisation dealt with early, which in turn helps engage people in the project.
Working in Sprints
The development work is then done in small increments, known as Sprints. Sprints can vary in length, but usually they’re between one and four weeks in duration.
At the start of each sprint, the items at the top of the Project Backlog are moved to the Sprint’s Backlog and discussions about the exact requirements for just those features occur between the business team and the developers. The developers then code the features, getting an initial working version of each feature up and running as quickly as possible so that the users can see exactly how the feature will work and feed back to the developers.
The key here is that scope evolution is an expected part of the Agile process; as soon as the users see the feature up and running they will realise ways it can be improved and so those improvements are incorporated and the optimum solution (and no more) is then delivered.
At the end of each sprint the work is reviewed, any unfinished items are moved to the next sprint and new items from the Project Backlog are added to the next Sprint’s backlog. The process then repeats until all items from the Project Backlog have been completed.
Planning and monitoring ongoing progress – Story Points
So far we have only defined rough estimates for how long work items will take, so it may seem like it will be difficult to track the progress of the project. Agile has a way of dealing with this too, which is to assign each item of work a size (say extra-small to extra-large) or a number of Story Points. Story Points are simply an arbitrary measure of the complexity of a piece of work based on a common understanding between all the team members (both the business and the developers).
At the start of any project it’s hard for the team to assess exactly how long work will take, factors like the complexity of the system’s interaction with other systems, the speed of the business in answering questions that arise and the relative speeds of the developers on the teams are all unknowns at the beginning. So initially story points are only assigned to the work in the upcoming sprint.
At the end of each sprint the total number of story points for all the items completed in that sprint is calculated. This gives a measure of the number of points that can be completed per sprint, known as the Project Velocity. This Project Velocity is then used to determine how many items should be scheduled into the next sprint.
After three of four sprints the team naturally form a good understanding of the complexity of the project and arrive at a common understanding of what constitutes a story point and so the Project Velocity becomes accurate. At this stage, if desired, the team can then sit down and assess more of the future work items in greater detail (not just the ones for the next sprint) and assign a number of story points to those items too.
Once that’s done the amount of work is then known (the number of story points) and the speed of the team is known (the Project Velocity) which means that the likely delivery dates of new features can be calculated. Work can be reprioritised to ensure that features are delivered in advance of when they are required.
The Project Velocity is also be used by the team to monitor their performance and to check that there aren’t issues causing the amount of work delivered to drop off. Because this is assessed at the end of every sprint any issues can be identified more quickly than in a traditional project.
Changes to priority and scope are expected and welcome
The Project Backlog can be reordered at any time to allow for the changing priorities of the business and new items can be added into the Project Backlog whenever the business identifies new requirements. This process means that the most urgent features are always delivered first and also that the exact requirements for each feature are only decided just before coding, eliminating the possibility of requirements having changed between the specification and coding stages.
But how do you know what it’s going to cost?
The sticker price is never the final price
The biggest hurdle to overcome when adopting an Agile approach is the shift away from the perception that you know exactly what you are getting and for exactly what price in advance of the work. I say perception here because even with a traditional pricing model (with a specification and a quote prepared in advance), the number on the quote is rarely the final price. Additional requirements are always discovered during the programming of the software which leads to the price increasing and the work taking longer than expected. This is frustrating for all involved; the staff charged with running the project look bad for failing to keep the costs and timescales under control and the software developers come under pressure to deliver faster, which often reduces the amount of testing done and therefore the quality of the product delivered.
Embrace the certainty of change
Agile does away with all the above. Instead of trying to calculate the exact price, the initial requirements for the project (the initial Project Backlog) are considered and given a ballpark estimated duration which in turn allows an approximate price range to be calculated for the work as it is understood at that time. This is a top-level assessment, performed using the developer’s experiences from past, similar projects. This ballpark is then discussed to see if it fits within the business’s budget. If so, work commences, if not the requirements list is reassessed to see if there are any less urgent items that can be left until the next budgetary period (or just not done at all). If so, the project can go ahead.
It’s important not to get too caught in the detail of the features during that ballparking stage, the idea is only to check whether the cost of the work desired is the same order of magnitude as the level of budget available.
I’m not writing the developers a blank cheque…
Committing to a specific expenditure without firm details of exactly what you are getting in return can certainly feel a bit like writing a blank cheque, but that isn’t the case. Admittedly, in a traditional model a software developer gives you a list of what will be done for a certain price, but that typically follows weeks of discussion and needs analysis. Therefore, by the time the specification and final quote are delivered a substantial commitment has already been made. That commitment is often financial, as needs analysis is usually chargeable work, but even if the analysis has been done for free you will have committed large amounts of your employees’ time to getting the developers up to speed.
Things are very different with Agile. Rather than having a large period of analysis before any code is written, in an Agile project the first sprint is delivered within weeks of the outset of the project and at that point the business gets to see actual working software. The business can then assess whether the features promised for that sprint were delivered and whether the work is of good quality. If you don’t like what’s being delivered you can simply pull the plug at the cost of a few weeks work.
Compare this to a traditional model where you might not see working code until months of analysis and development have taken place and already been paid for. This is expenditure that almost certainly couldn’t be fully recovered if the system isn’t as hoped when it is finally delivered as the developers will have already done the work.
So, whilst Agile can feel like a leap into the unknown, the time and financial exposures are actually much lower and the continuous nature of the delivery of the system means you always know for certain that you are getting what is promised.
Why would we do this, what’s in it for us?
Hopefully this document has already highlighted a lot of great benefits from the Agile approach, but there are plenty more too:
Agile delivers working code within weeks, meaning the business’s most pressing requirements can be addressed quickly. Not only is the initial delivery faster, but each feature takes less time to write because the developer isn’t writing a detailed specification in pseudo-code, and then subsequently writing the actual code.
Software developers sell time, so if the development is faster then it’s also cheaper.
But, that’s not the only reason it’s cheaper. For any company whose product is time, the biggest fear is having a resource that goes unused. That means that software houses have to build allowances into their overheads for periods of under-utilisation of resources. Agile projects where a set budget for a year is known in advance eliminate much of that potential for under-utilisation, which means that the overheads and therefore hourly rates for those projects can be considerably lower. This is a true win-win as the developers can achieve the same margins whilst charging their clients less.
Client and developer become a team
In traditional software development there’s a long initial period where the software developer’s analysts attempt to extract the full requirements from the business. Those requirements then become the specification and everything is then judged against that specification. That cannot help but ultimately create an adversarial situation; there will inevitably be differences in how people understand what’s written in the specification and that soon leads to an ‘us and them’ feeling.
In Agile that friction disappears. Instead of the developer’s analysts speaking to the business’s managers once at the outset of the project, in Agile the developers work with the business team gathering their requirements just before programming the feature. The feature is then delivered as early as possible (usually within days) so the users can check the feature is as required. Often the users will then realise a feature would work better if it was tweaked. Because there’s no hard and fast specification these changes can be immediately incorporated, which means both the users and the developers can work towards iterating the optimum solution. That close, ongoing working relationship swiftly builds a strong feeling of partnership between the organisations.
You get onsite business analysts for free
The collaborative nature of Agile working means that rather than only seeing the software developers at the start and end of the software writing process, they will instead be visiting site at the beginning of many of the sprints (to work with the team on the requirements for that sprint).
As well as bringing the team members closer together, this has another benefit, which is that the developers see more of how the business carries out its day to day processes. This can be invaluable, it not only improves the developers’ understanding of the business but it also allows them to spot areas where other software systems and features they’ve written for other companies can provide positive return on investment to your business. This effectively gives your business an ongoing business analysis and needs analysis team without employing additional staff.
You pay exactly the right price
As we’ve already seen, with the traditional software development it’s a certainty that the price paid won’t actually accurately reflect the amount of work done. With something as complex as a large software system the time estimates will always be some amount wrong and so either the client is paying too much, or the developer is getting paid less than they hoped. Because nobody likes to be consistently underpaid the developers will of course start building in contingency to their prices too.
That seems like a terrible outcome for both sides as somebody always loses out. With Agile there’s no estimating, you pay for exactly the time done, so the business pays exactly the right price. And of course there’s no contingency cost either.
What’s more, in the traditional model an out of date specification can lead to features being written that will never be used, because the business process has changed by the time the software is eventually finished and released to the users. That simply can’t happen with the on-demand nature of Agile development – an unneeded feature will never rise to the top of the backlog list.
Engages your staff. Continuous improvement.
The number one success factor for a software project is user engagement. The business managers and software team can write the best system in the world on paper, but if the users don’t engage with it and commit to learning it, the system will never improve the business’s operations in the ways hoped.
Long experience in delivering software tells us that users often fear new systems, they see their jobs being automated away or they think they aren’t being listened to and so swiftly feel the system is being foisted upon them by people who’ve never actually done the work they do.
Agile is brilliant for getting rid of this type of ill feeling, the users become a part of the team that decides the requirements of the feature at the start of the sprint and they are the ones to test and refine the feature as it’s delivered. That means they are fully engaged with the feature from the outset and get exactly the functionality they need to improve their efficiency.
It’s not just about delivering the right software though, that process of engagement is rewarding for staff members; it’s a new skill they have learnt and it shows them the business recognises their importance and the value of their ideas. The time you freed up by the new process efficiencies can then be used for more creative thinking by staff to find further ways to for the business to innovate and improve.
You get exactly the software needed, and no more.
Many ideas are discussed at the start of a software project and in a traditional project it can be hard to choose which are really needed. In Agile that process is automatic, the Project Backlog is continually reprioritised and the unneeded features simply never get to top of the list. What’s great about this is that it means that all ideas from any staff member can be welcomed and added to the backlog; the best ideas will then be implemented and the others won’t.
It’s impossible to go over budget
The budget is fixed, it’s the features which are delivered that change. That statement may ring alarm bells, it sounds like you aren’t getting the features promised for the price, but, the thing to remember is that in Agile the most important work gets done first.
The understandable fear for the business managers here is that they could end up in a situation where budget was assigned on the expectation of the delivery of certain features, but the evolution of the project priorities then caused some of those features to be pushed back. In reality however that’s very unlikely to happen, the headline features the budget was assigned for will almost certainly have been top of the priority list and so those features will not only have been done, but they will have been done first and delivered immediately (rather than waiting until the entire scope of work was completed).
It’s the secondary goals, nice-to-have and no longer required features that may not have been done, replaced instead by other more urgent items identified by the business during the period. The project’s performance doesn’t tend to be assessed on the secondary features, and the fact that other more urgent items were achieved within the original budget tends to mean that the project is seen as over-performing rather than under-performing. The less important features that are still required do get done, just in the next budgetary period.
For non-IT staff trying to manage IT projects this is expenditure security is a real boon as it eliminates the constant concern about overrunning timescales or costs, whilst still ensuring the headline features are delivered faster than would be the case with traditional software development.