ARITHMOPHOBIACS INFILTRATE THE QA/QC COMMUNITY

No – they don’t know how many are out there. But they think “it’s a lot”. Well, at least more than one or two. Everyone’s afraid to count…


OK, am I the only one who’s read all of the articles lately on the Inherent Evils of Metrics in QA/QC and thought they were complete drivel? Well, I suppose if you’re a disorganized git who doesn’t know what you’re doing, having to tell someone else arithmetically would be pretty scary. And plotting a trend? Might as well paint a “666” on your forehead!!!

How much better to wallow in touchy-feely stuff. Yes, it might SEEM like there were a lot of errors, but one of our customers, Verne, I think his name was, thought the product was just great. THUMBS UP, GANG!!!! Those bugs, however many there were, didn’t “bug” Verne! We’re hot!!!!

Yeah, well, I know we added 14 testers to the effort (hey, we only charged you $65/hour per), and well, no, I can’t tell you what our coverage was, but I think it was pretty good! After all, look at Verne!!! No, well, I don’t know if they found more errors than say, those Elbonian vagabonds you imported last year (hey – have you cleaned the carpets up on the 4th floor yet?), but I think they found What Was Important, if you get my drift… Well, no, I don’t know how many errors are out there and what their severity levels are; after all, the term “severity” is totally Subjective! Do YOU think any of the errors are severe? If so, let us know and we’ll pass those on!! And gosh, who knows how long they’ll take to fix? It might be an hour and it might be a year!! Take your pick!! I DO know that we’ll be here for you, though, when they’re ready to retest. Well, no, we don’t know how long it will take – you tell us how long we’ve got and we’ll git ‘er done!! As long as the money holds out….


So was that unfair? Uh-huh. Probably. Did it feel good? Oh yeah. The articles I’ve read on the Evils of Metrics are equally unfair and when really analyzed look biased, based on extreme cases, illogical, and indicate the authors lack experience – particularly any kind of management experience. In short, they lack common sense. The reason I’ve chosen this topic is that this particular onslaught is something I actually think could have a negative impact on the field as a whole. In other words, I think it’s dangerous and self-defeating to avoid metrics, particularly as a lead or manager (or someone who would like to move into such positions).

Let’s look at reality, shall we? I’ll throw a bone out there and say it’s my reality, but I don’t think my experience is any different than that of most professionals in the field, so here goes…

There’s a biblical saying that goes “Do not kick against the pricks”. That saying can have multiple meanings in contemporary times (!), but generally means kicking against reality will mostly just hurt you, and it doesn’t keep thornbushes from continuing to sprout up.

The hard facts of working in corporate America are that they use a lot of metrics. They use metrics to tell them that they produced 200 widgets instead of 100 last month. They use metrics to tell them they cut costs by 20% last month. And they use all kinds of metrics for sales.

Executive managers are accustomed to metrics. They like metrics and graphs. It stands to reason that if you want to “sell” anything to executive management, you will need to do the same. This is NOT AN OPTION. It’s either learn to express and support yourself through metrics or to be unsuccessful – your choice. No matter how much you want that to change, no matter how fearful you are that the results will be used the wrong way, and no matter how much you feel it doesn’t make sense to equate the development of software to the development of a product, you and your opinions are NOT going to change executive management in the USA.

There are some advantages to metrics. One is that they are unemotional. You can argue with a person. It’s more difficult to argue with numbers.

If I want to tell a given group of PMs, for example, that the error rates on their projects are way up, my staff are spending twice as much time as normal testing, and that the errors are severe, I’m going to trend all of that out on a graph and show them some evidence that there is a problem. This is the quickest and most effective way to make them Pay Attention and Do Something.

If I need more testing staff for a given effort, I’m going to use # of test cases that can be reasonably run by one person, what is anticipated, timeframes and request staffing for the excess. I could just go to executive management and tell them I don’t have enough people, but it’s unlikely that will get me any funding. I need a nice report with statistical, unemotional data and metrics. You’ll note that I’d need to know # of test cases that can be run by one person. Gasp of horror! We have test cases!!! We have heuristics for what one person can handle!!!!

Call me a rebel.

The number one question I’ve been asked as a manager is “How long will it take?”. Telling executive management “well, it depends” is not an option. If you do not pull metrics and have no historical data to do some analogous estimating, you’re in Major Trouble. Executive management also wants to know how many people are needed for a given effort, how much it will cost, etc. They want progress plotted and percentages complete during testing efforts. They like the root causes of errors categorized to allow focus to be brought to problem areas. They want to know if various quality initiatives have actually improved quality. They’d like to know if they’re getting better or getting worse. All of these things can be captured and expressed numerically – again.

I’ve heard plenty of arguments against metrics. For example, 20% complete – what does that mean? What if it’s the most difficult part of the project? What if the remaining 80% will be done twice as fast? My answer to that is so what if it is? Metrics for that day will show progress has speeded up. A decent manager will take a look at the metrics and if everything is at 40% done but one area is at 2%, they’ll ask some questions. The answer might be a blocking error; testing can’t proceed for that app. Metrics are just a tool and what they tell you must be analyzed to be meaningful. There are plenty of cases where judicious analysis of metrics has helped to either head off a potential problem or focus attention on a problem area.

Some of the people who argue against metrics do so because they don’t KNOW how many tests they’ll run. They don’t KNOW how long it should take. They are incapable of justifying additional staff in any way other than emotional. Yup, I’d say you’ve got a problem there. Everyone working on a project has some sort of schedule they work from and goals in terms of date. QA cannot be exempt.

Metrics are based on averages, and the larger the effort, the more accurate metrics become. That means some tests might take 5 minutes and some might take 4 hours to execute, but the average might be half an hour. Averages give you enough information to do some planning and determine how far you’ve come and how far you have to go during a testing effort.

I think a great deal of the hype in regards to Evil Metrics is based on extreme cases of managers that were incompetent. A good manager determines what statistics will be useful and meaningful and pulls them using the same parameters every time. They ANALYZE the results. In other words, if numbers for some category are up (or down), they look for the reasons why and make decisions as to what should or should not be done to solve issues.

I too have worked for people that used metrics inappropriately. I had a client that only wanted to know how many test cases were written per week. It didn’t matter how many were needed or whether those test cases were useful. He just wanted a nice, high number. That’s exactly what he got. Such people are not qualified to be managers. But there are bad managers everywhere; the QA/QC field is not unique.

I could go on about this topic indefinitely; using metrics wisely is a key to success for management staff. But I’ll leave it here and just suggest that you refuse to allow yourself to be intimidated by metrics or to be swayed into making a serious career misstep by avoiding them. QA/QC managers need to survive in the corporate world and that will not happen unless they are capable of producing and are comfortable with providing metrics. I'd suggest instead determining what type of metrics will be useful to you, your team, and your stakeholders and using them intelligently to move forward.