Software estimation: insights from decades of successes and failures

A lot has been said about estimation in software development; there are tons of articles and blogs offering advice on how to do it better. And while it might seem a bit contradictory that this article is also about that, why don’t we flip things around and look at it from another angle? I don’t want to write another boring post about estimation. Instead, I want to dive into some unconventional ideas and share my own experiences with you.

Estimates in software projects

Acknowledging uncertainty

Estimating projects is hard, and the first thing to remember is that no two projects are the same. Each project has its own challenges and things that can complicate the estimation process. So it’s important to keep an open mind and not assume things based on past experiences. Instead, you need to carefully analyze all the details and factors of the current project to produce a more accurate estimate.

Another thing to keep in mind is that there will always be uncertainty. No matter how much we plan or analyze. Things can change, risks can arise, or problems can occur during implementation. Therefore, it’s smart to account for this uncertainty by adding extra time or resources in case things don’t go as planned.

Communication is also super important. Everyone involved in the project needs to speak openly and clearly throughout the entire estimation process. This means discussing expectations, making sure everyone understands what needs to be done, addressing any concerns or doubts, and staying updated on progress. Good communication helps prevent misunderstandings, catch problems early, and adjust estimates when new information comes up.

Also, doing a retrospective analysis of past projects can help make more accurate estimates. By reviewing what happened before and learning from mistakes, we can make better decisions when estimating tasks for new projects. It’s also useful to think about how our brains work when making estimates: we might have biases that make us overly optimistic or confident. Understanding these biases can help us be more realistic when estimating.

In short, estimating projects isn’t easy, but by keeping these things in mind—being open-minded, accounting for uncertainty, communicating well, and learning from the past—we can make better estimates that set us up for success.

Wrong ways to estimate

Alright, let’s talk about estimation, my friend. You know, there are some pretty bad ways to go about it. Like, imagine just blindly guessing how much time and effort a task will take. That’s like throwing darts in the dark and hoping for the best. Trust me, it won’t end well.

Then there’s the whole “one-size-fits-all” idea some people cling to. They think they can apply the same estimation method to every project under the sun. But hey—newsflash! (or maybe not so new because we already said it)—every project is different, with its own challenges and requirements. Ignoring those external factors is a recipe for disaster.

Now, here’s another big mistake: having a fixed mindset. See, estimation requires flexibility and adaptability. If you’re stuck in your ways and refuse to consider alternative perspectives or new information, you’re setting yourself up for failure, my friend. Embracing collaboration and diverse viewpoints is key to making realistic estimates that actually have a shot at being accurate.

So remember, when it comes to estimation, blind guessing and one-size-fits-all thinking just won’t cut it. And don’t even get me started on ignoring external factors or having a rigid mindset! Instead, let’s embrace collaborative practices and leverage the collective brainpower of everyone involved. It might take some open discussions and brainstorming sessions, but believe me—it’ll lead to better outcomes in the end.

The right ways to estimate

Here’s a secret the software engineering gods have been keeping under lock and key: there’s no single “right” way to estimate!

But before you close this article and leave disappointed, hear me out. I know you came here looking for the right way to estimate, and I just said it doesn’t exist, but I can share what has worked for me.

Let’s say someone asked me how long it would take to eat a medium-sized elephant. How would I go about it?

First: Cut the elephant

Step one for me is always breaking the work into small tasks.

Breaking things down as much as possible helps get more accurate estimates, but realistically, it’s not always practical. So my suggestion is to break it down until you’re confident enough to bet your monthly salary on your estimate for that part of the work.

(Still, don’t waste your time asking me about the best way to estimate tasks... just use the method you’re comfortable with, as long as it doesn’t become the main focus of your project.)

Second: Uncertainty chart

People who know me know I’m a huge fan of the uncertainty chart. Why? Well, it’s something we can all relate to and sympathize with the poor guy holding the estimates.

Now you’ve got every piece of the elephant chopped up, and you’ve estimated (a pretty wild analogy) how long it’ll take you to eat each piece. But how confident are you in each estimate?

Why not place each “slice” within the cone of uncertainty based on how unsure you are about that estimate? Then show the chart to everyone so they understand your concerns and uncertainties. This helps ensure that everyone involved in the project shares responsibility for those estimates.

Third: Sum up the slices

So, is the total time to eat the elephant just the sum of the time for all the slices? Not exactly.

If you do that, you’re not factoring in the slice’s position within the cone of uncertainty—ignoring that crucial aspect. To include this information, you can multiply your estimate by a factor related to how uncertain that piece is. This approach provides a more complete and accurate representation by considering the range of possible outcomes within the cone of uncertainty. By accounting for this uncertainty-related value, you improve your estimate by considering all potential outcomes within the cone, leading to a more robust and reliable analysis.

But what if I don’t have enough (knowledge, experience, information) to estimate a task? How can I reduce uncertainty?

• Call in the pro!: This is the fastest and easiest method, especially if an expert is available. Trusting an expert’s opinion is usually enough to estimate the project’s quantitative details, such as timelines and resources. This is also the most commonly used technique.
• Research: By reviewing past projects or industry best practices, developers can gain valuable insights to make more accurate estimates. Research also helps identify potential risks or uncertainties that could impact the estimation process. This proactive approach ensures estimates are built on solid ground, reducing the chance of unexpected issues and making software development more efficient and reliable.
• Spikes: When you’re trying to figure out a solution to a problem, sometimes you’ve got to dive in and start coding. It’s like a little experiment to see if your idea works. And well, sometimes that code ends up being useful for your project, but most of the time, it’s something you toss once you’ve proved your point. Just make sure you set a time limit for the spike—otherwise, you might end up spending forever on it.
• Do it like the pharma industry: They allocate money for their research projects, run them, and then decide whether to invest more based on results. You can do something similar by starting a task with a set budget. When the money runs out, assess how it went and decide whether to secure more funding to continue or stop the task.

Why are estimates so important?

Why do people argue so much about this, and why is there no definitive way to estimate software projects? Is it really that hard or important?

Let’s apply what we’ve learned and eat this elephant piece by piece.

First, estimates aren’t just important... they’re crucial in software development! Software development is a business, and unless you’re part of some ultra-chill product team or swimming in cash (or a few other rare cases), money is going to be on your mind.

Most articles on the topic only talk about money, and in the end, that’s what it all comes down to. But I want to highlight other aspects of estimation to really understand its importance.

• Planning and resource allocation: Estimates help business owners plan and allocate resources effectively for software projects. By estimating the time and effort needed for each task, business owners can make informed decisions about staffing, deadlines, and overall project management.
• Setting realistic expectations: Estimates allow business owners to set realistic expectations for clients and team members. By understanding estimated timelines for completion, they can clearly communicate those expectations, avoid misunderstandings, and keep everyone aligned.
• Task prioritization: Estimates help business owners prioritize tasks based on their estimated complexity or effort. This helps determine which features or functionalities should be developed first, ensuring critical aspects are tackled early in the project.
• Identifying potential risks: Estimates help pinpoint risks by highlighting areas that may require extra attention or resources. By understanding the estimated challenges or complexity of certain tasks, business owners can proactively address risks and implement strategies to minimize negative impacts.
• Improved decision-making: Estimates provide valuable insights that support decision-making throughout the software development process. Business owners can use estimates to evaluate alternative approaches or solutions, make informed trade-offs between features, and ensure decisions align with the project’s goals.

AI to the rescue

Okay, I get it... But is it really that hard? Why can’t anyone figure out the perfect way to estimate software development projects? I mean, we’re living in the AI era! Can’t I just ask ChatGPT for an estimate?

Simply put: no. Accurately estimating software development projects is a genuinely challenging task. While AI like ChatGPT can help in many areas, estimating project timelines and effort still requires human expertise and involvement. Here are some reasons why perfect estimates are so elusive:

• Complexity: Software projects can be extremely complex, involving countless variables, dependencies, and potential risks. Predicting all of this accurately is tough.
• Uncertainty: There are often uncertainties in requirements, scope changes, and external dependencies that can affect project timelines. It’s hard to predict and account for all of them.
• Human factors: Estimating software development involves human skills and effort, which are influenced by individual capabilities and experience. Different developers may have different speeds, approaches, or unexpected challenges.
• Emerging technologies: The ever-evolving nature of technology makes it hard to estimate projects involving cutting-edge tools or frameworks, as historical data is lacking.
• Communication and collaboration: Accurate estimation requires effective communication and collaboration among stakeholders. Misunderstandings or incomplete information can lead to inaccurate estimates.

While AI can support estimation processes by analyzing historical data or offering pattern-based insights, it can’t replace the need for human judgment and an understanding of the project’s specific context. The best approach is to combine AI’s capabilities with expert knowledge to improve estimation accuracy rather than relying solely on automation.

Conclusion

As we said, estimation is a crucial aspect of software development that goes beyond money. It helps with planning, setting expectations, prioritizing tasks, identifying risks, and making better decisions.

AI can help in various areas of estimation, but it can’t replace the need for human expertise and judgment. Accurate project estimation requires acknowledging uncertainty, avoiding flawed approaches such as blind guessing or fixed mindsets, and embracing collaborative practices.

So let’s improve our estimation skills by learning from past projects, seeking expert advice when needed, conducting research to reduce uncertainty, and using techniques such as spikes or budget allocation to improve planning.