When Should You Assign Points to Defects?

A common question that we get from teams who are new to agile is whether you should assign points (sizes) to defects.  From a Disciplined Agile point of view we know that the answer will vary depending on the context of the situation, or in other words “it depends.”   The really disciplined answer though is to say on what it depends, which is what we explore in this blog.

Rod Bray, a senior agile coach at Scott Ambler + Associates, suggests this principle:

Don’t reward people for shoddy work

An implication of this is that if a team is injecting defects, they’re producing shoddy work, then it makes sense that their velocity suffers when they have to spend time fixing those defects.  The decrease in velocity is a visible trigger to investigate, and address, the cause of the productivity loss.  But what if the “shoddy work” wasn’t caused by your team?  What if the “shoddy work” was previously acceptable to your stakeholders?  Hmmm… sounds like the context of the situation counts in this case.  The following flowchart works through common thinking around when to size a defect.  Note that this is just our suggestion, your team could choose to do something different.  Let’s work through common scenarios to understand whether defects should be sized or not.  These scenarios are:

  1. Is it existing technical debt?
  2. The defect is injected and found by the team
  3. The defect is found by independent testing
  4. The defect is found after the solution is released

 

When do you size defects

 

Do you size the repair of existing technical debt?

The quick answer is sometimes.  For the sake of discussion we’ll assume that this technical debt has existed for a long time, potentially years.  From a purist point of view technical debt may be injected at any time (this is obviously true) and as a result may only be seconds old.  In the case of technical debt that is very new, refer to the logic below.

The key issue is whether fixing the defect is going to be a substantial effort. This of course is subjective.  Is the work substantial if it’s less than an hour of effort?  Several hours?  Several days?  Several weeks?  This is a bar that your team will need to set.  Something that will impact your schedule is likely to be substantial (and something your Product Owner is likely going to want to prioritize so that it can be planned for appropriately).  Clearly a defect that takes several days or more to fix is substantial and one that takes less than an hour is likely not.  Something that takes several hours or a day or so is likely something you need to think about.

 

Do you size defects injected and by found by the team?

No, the exception being technical debt (see above).  This falls under the principle of not rewarding the team for shoddy work.

 

Do you size defects found by independent testing?

Some teams, particularly those working in regulatory environments or working in complex environments, may be supported by an independent test team.  An overview of the independent testing strategy is presented below.  With the exception of the defect being found in existing technical debt (see above), the defect should not be sized.  Once again, the principle described earlier holds.  Of course if you don’t have an independent test team supporting your team then obviously you can ignore this question.

Independent agile testing

 

Do you size defects found in production?

The answer is sometimes.  As you can see in the high-level lifecycle diagram below, the DA framework explicitly recognizes that change requests are often identified by end users of an existing solution.  These change requests are effectively new requirements that should be prioritized and worked on appropriately.  Many organizations will choose to distinguish between two types of change requests, defects and enhancements, so that they may be treated differently.  The issue is that defects are often tracked and, if the team is smart, they are analyzed to determine how they got past the team in the first place.  Additionally you may find that defects and enhancement requests are charged for differently (a topic for a future blog).

Disciplined Agile Lifecycle - High Level System

If you don’t distinguish between defects and enhancement requests then there’s nothing to do, you simply treat the change request as a new requirement and size it like you normally would.  But if you do distinguish between types then you need to think about it a bit.  If the defect is found during a warranty period then it shouldn’t be charged for. Sometimes, particularly when work is being outsourced, there is a warranty period of several weeks or even months after something is released where the development team is expected to fix defects for free.  In this case you likely wouldn’t size the work in line with the principle described earlier.  Once you’re out of the warranty period then it’s likely you want to assign points to it: The functionality in question passed testing and was accepted by your stakeholders, so they in effect have taken responsibility for it.

To summarize, the answer to the question “Should we assign points to a defect?” is “it depends.”  In this blog we explored in detail why this is the case and described when and when you wouldn’t want to assign points.

I’d like to thank Kristen Morton, Rod Bray, and Glen Little for their insights which they contributed to this blog.

5 thoughts on “When Should You Assign Points to Defects?

  1. Valentin Tudor Mocanu

    Sometime we want to evaluate defects, but is almost impossible. One of the main problems is to keep the defects in the “sizeable” area and/or keep the development in a quick fixing capability area. If we have too often these kind problems, with a product or product area, we are already in a zone where defects repairing will affect any kind of planning and make the quality questionable by default. In my experience these are one of the worse possible process smells.

    In such cases, we should think how to take necessary measures to bring back the product in the track of manageable quality. But we should not loose this track from the first place.

    Reply
  2. Brad Appleton

    Hi Scott! I think an important question is how the Defect “points” are going to be used.
    – Will they be counted toward Velocity? And is velocity supposed to reflect work capacity (of any kind)? Or only the velocity of value-demand work?
    – Are the defects in question going to be part of Sprint planning? Or do you always fix any and all defects within the sprint, before starting work on the next highest valued story?

    There is definitely a desire to be able to account for (even plan for) time spent working on defects and other non-value-add work. (Some even wonder if defects should be estimated using negative points)

    I think the original intent is for velocity to apply to amount of valued-work (in story points) to apply to planning and progress-tracking for a sprint. In which case, amount of capacity spent working on defects should not contribute to planned nor actual velocity.

    Then there is the question of whether to assign points to defects is essentially a different way of deciding if it is really a “bug” or a “feature” for the purposes of planning (or “change-control” of a “project baseline” in traditional waterfall). Let’s certainly hope not!

    I think the bigger issue is how to track/account for “friction” or “backwash” (e.g. “undertow”) against the cumulative flow of value-delivery). The initial/simple way is to say defects and other “failure demand” don’t count toward velocity and hence don’t get to have points. But then at some point a team evolves beyond that simplistic view and perceives a need to have more insight about impact on value (e.g., “cost of delay”) of non-value-added work.

    So what if defect sizing (“points”) are used, but aren’t counted as part of velocity, and instead as part of something else? (like friction/backwash, or “cost of delay”)

    Reply
    1. Scott Ambler Post author

      Brad, you make very good points. I think that the key issues are that you don’t want defects to count towards your value-add velocity yet there is need to track the loss/friction/undertow of quality problems.

      In the end, I would go back to basics and ask the questions – How are you going to use this measure (the defect points) to improve your decision making? What will it cost you to gather this measure? Is the decision-making value of the measure (e.g. are you making better decisions as the result of knowing this) greater than the cost?

      Reply
  3. Saket Bansal

    If we do not assign points to Defects , how do we quantify the failure work? Some of the team i worked with got lots of value by assigning points since it made the failure work visible to the team and stakeholders, while doing release planning we also get a feel of what is it going to take to clean the product from defects .

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *