Comments on PRACTICAL QA: METRIC OF THE MONTH – ERROR RATES

Rather than diving into a discussion about "c...

2009-09-17T15:36:30.544-04:00

Rather than diving into a discussion about "correlation versus causation" with regard to Error Rate and Customer Complaints, I'll just say - if it works for your shop, then it's great!

I wasn't suggesting that you or your staff tamper with the numbers. I was only pointing out the math involved in determining the Error Rate, and that I've seen and worked at shops that "play" with factors in order to manipulate their metrics (either intentionally, or unintentionally).

I'm not sure I know how to divide metrics into Evil versus Non-Evil, as I belive metrics aren't inherently evil. Like technology or weapons, all metrics can be used for good or bad.

This is a good discussion, Linda - thanks for tackling it.

I'll look forward to more Metrics of the Month. If I find one that I hadn't heard of and tried before, I'll give it a shot.

-joe

Sandeep, I understand your point in regards to cli...

2009-09-16T09:45:50.853-04:00

Sandeep, I understand your point in regards to clients not running tests in production. The total tests run is used as a denominator because we are evaluating the efficacy of the the testing process overall. We do run several metrics that are more specific, but I'm starting out with basics for this series. I think the formula I gave will help people get started and give them some useful information to kick off their analysis efforts. I'm going to talk about defects for the next "Metric of the Month"; we do something very similar to your process.

Thanks!
- Linda

2009-09-16T09:36:55.162-04:00

This comment has been removed by the author.

Joe, I wasn't trying to ascribe customer com...

2009-09-16T09:36:32.463-04:00

Joe,

I wasn't trying to ascribe customer complaints to metrics; I was trying to explain that once you have a metric, you have to determine whether that number is "good" or "bad" for your own company. You might have a 2% error rate and your customers might think that's terrible. Or have an 86% error rate, and clients that love you. "Good" and "bad" need analysis. that said, I have found that when our clients are unhappy and have a lot of complaints, our metrics usually indicate we had a problem as well. Metrics are great as generic indicators and bad as specific answers to questions or problems. We use them here as a "jumping off" point to ask questions.

If a platter were being prepared for my head, it wouldn't be because of metrics. It would be because of some problem that might have been highlighted by metrics and later found through analysis to be something I or my team missed. I've never, in my entire career, told my team to find either more or less bugs. In fact, I don't know how you extrapolated that kind of scenario from my post. That's really "tampering" with the numbers, and I put it in the same league as Evil metrics. I believe metrics are good for generating questions, helping with analysis, setting goals, and highlighting or explaining successes or challenges to executive management, but they are non-specific averages and always need analysis.

- Linda

"So your overall error rate is 25%. Is that g...

2009-09-16T08:52:28.903-04:00

"So your overall error rate is 25%. Is that good or bad? Neither. It’s a number. If your clients call to complain in droves, die, or your boss demands your head on a platter, I’d say it’s “bad”. If you get a board commendation, an invitation to play golf with the CEO, and your coworkers carry you around the building cheering and chanting your name, I’d say that’s “good”."

Here's where I find fault with this thinking. Trying to ascribe customer complaints, or lack thereof, to a metric is iffy.
My contention is that the "bad" or "good" responses here aren't actually related to the overall error rate. What you really want are "happy clients", and not "a good metric value".

For example, using your formula, if you increase the number of tests you run, your Error Rate could easily decrease, while the number of errors found could remain exactly the same. Would you expect a corresponding increase in golf invitations?

And if a platter is being prepared for your head, what should your response be? Should you tell your team to "find fewer bugs"? That will certainly decrease your error rate.

Thank you for starting this discussion, Linda. And thank you for never getting involved in Evil Metrics. While we may all disagree on which metrics (if any) are un-evil, I am still glad to hear it.

-joe

Thanks for the post. You never fail to entertain e...

2009-09-16T06:38:06.528-04:00

Thanks for the post. You never fail to entertain even while you educate. Let me tell you that I am a numbers man myself and I use them with both pragmatism and care. So relax and keep that gun and shield away.

One of the metrics we capture is the Phase Yield (across Development, Test and Acceptance Tests) which is a percentage of defects (duly weighted by severity) found in a particular phase against total defects across all phases.

Our goal is to find more defects in our tests as opposed to those in Acceptance which is expressed as a percentage of System Test defects against total of System + Acceptance tests defects. I call this 'Quality of Testing' metric.

We also keep a track of defects reported from Production. But I do not understand your suggested use of tests as a denominator since the tests would never be run by the customers.

Linda, Really great blog posting. It's always...

2009-09-16T05:08:23.142-04:00

Linda,

Really great blog posting. It's always good to see some balance between theory and idealism versus reality.

I too have worked in places where metrics were needed and they helped me lots. Right now I'm not needing them, but as soon as I do, I'll start using them again.

I think the key point you make is that they are just a number. Nothing more, nothing less. How we inteprate that number is based on our own needs, perceptions, ideas and aims.

Looking forward to the next blog.

Rob..

I agree with you that there seems to be a lot of &...

2009-09-15T22:16:28.983-04:00

I agree with you that there seems to be a lot of "metrics phobia" floating around.

Your discussion of defects being discovered by clients vs. staff is interesting. This would be interesting data to try visualizing with Parallel Sets