HOW THEY TEST SOFTWARE AT MICROSOFT

Well, after finally getting my book chapter done for Beautiful Testing, recovering from pneumonia, and watching our first (major) migration of 2009 finally wind down, I’m able to focus on some other things and went back and read “How We Test Software at Microsoft” by Alan Page, Ken Johnston, and Bj Rollison.

First of all, this book ($29.95 at Amazon) is a TOME, and I wasn’t sure about it; I sometimes have the attention span of a hyperactive squirrel and I have to be “captured” pretty fast. Otherwise, I just start “skimming” and looking for the juicy bits.

I didn’t skim.

When I worked as a consultant, I was used most often as an expert assessor. That means I went to a company that needed help/advice on something relating to QA or QA, and through interview and review of people, deliverables, corporate environment, etc., made recommendations as to how they could address whatever issues prompted them to hire me.

That work was great; I’ve always enjoyed learning about other companies, environments, and culture.

This book grabbed me right away; it was a glimpse into the culture of a vast, complex, and interesting company with some challenges that are unique in the field. And after reading this book, I’m STILL fascinated – only now I have a lot of questions as well. I’ll get to some of those later. Some questions were answered in the book itself.

When I told my staff I had purchased this book and that it would available for them to read when I was through with it, most of them asked to get on the loan list. Such is the power of the word “Microsoft” alone. At the same time, I got a lot of comments that are typical when you mention Microsoft and either Quality or Testing in the same sentence. This time it was “What about Vista??”.

Generally, Microsoft is not viewed as producing quality products. It’s viewed as producing products that make the company a lot of money.

Most people haven’t thought much about the enormity and challenges inherent in producing software used everywhere by virtually everyone. But I’ve thought about it several times, especially in light of some of the testing superstars they have on their staff. How can a company who employs people like Bj Rollison, Alan Page, and James Whittaker produce anything that’s BAD? It’s a puzzle, and I don’t think I’ve solved it based on what I’ve read, but I do have a better understanding and maybe a few hypotheses now.

First, it’s obvious MS employees love their company and consider their products to be very high quality efforts. That alone is pretty riveting stuff. Can it be Microsoft people don’t know how their product quality is viewed in the field? I read about an app that had over a MILLION TEST CASES. I broke that app in one minute. What gives???

As you can tell, I was totally caught up in the book and HAD to read on.

There was little in the actual technique portion of the book that I had any issue with, and the explanations of some of those techniques was absolutely superb. It was worth the price of the book for the chapter on Model-Based Testing alone, and my apologies for blatant plagiarism, but I’m confessing right here that I’m lifting some of that for the next training class I give in-house.

Although there was some mention of exploratory testing, no actual techniques were discussed and I got the idea MS doesn’t know that much about agile test technologies as yet. So if you’re an agilista, this book will still be an interesting read, but there’s nothing there that will help you understand or train agile testers.

Some of the more interesting tidbits:

The nature of the products produced at MS does not lend itself well to testing methodologies that do not produce re-usable test assets.

The ginormous number of tests and the number of times those tests have to run in order to test a given app almost “demands” a high degree of dependence on test automation. This, in conjunction with the technical nature of many of their products, has resulted in a test force that consists of programmers who test – or SDETs (Software Developers in Test).

Half their workforce is recruited directly through colleges. Even when hiring experienced staff, they rarely hire people who have testing titles. They want people who can code. Yes, I know – it’s pretty funky stuff.

Bj Rollison cited studies that proved Exploratory Testing does NOT find more or better errors than structured technique and that it’s actually less effective in some circumstanced.

Oh baby.

As someone who has maintained the analysis/thought processes of agile/structured testers is a common activity unaffected by methodology, this again was worth the price of the book. Furthermore, I would like to buy Bj an adult beverage to show my appreciation…

Overall, there’s so much stuff of interest in this book, I advise you to read it yourself. I could have taken notes and written blogs for months on just single snippets of information. On the Wilkinson Scale of Goodness, I give it a 4 out of 5.

And now for my questions/comments:

I personally find it difficult to assess whether a newbie straight out of college is going to be successful in ANY field. They are “untried” potential. Even with experienced staff, for every 10 testers you find working today in the field, you might only find ONE that has what could be referred to as having “tester DNA”. What does MS do with all of the new hires that turn out to NOT have “tester DNA”? Or do they just retain them in those positions? And what is the rationale behind having a testing staff primarily made up of inexperienced coders? Is the reason financial, the ability to “mold” new people into the type of resource they want/need, or a combination of reasons?

Although I totally understand that to test a programming tool, it would be criminally stupid to not be a programmer yourself, it seems to me that MS has still not given up the “programmer as hero” mentality and is hiring too many programmers as testers. I started my career in systems analysis, and worked on nothing but automation projects for over ten years, and I think it is a very rare individual who can be talented at both testing and programming. The focus and work of automation is totally different from the type of analysis done to extrapolate test ideas from specs or people. I have rarely found an automator that was especially good at functional testing.

In most shops, white box testing is the responsibility of the developers, who know their code best; it’s actually viewed as a more efficient use of time and personnel. Automation is also normally handled by separate staff (who DO have development backgrounds). This frees up the tester to focus on functional issues; what is referred to in the book as “black box” testing. It occurred to me that some of the quality perceptions of MS in the field might be traced back to the fact that many of them are NOT coders and do not think like coders; what is important to an end user might not be something that would be considered important by a coder. I’m thinking that what many of us see after a major release by MS is really an app(s) that has been thoroughly unit tested.

Related to that, I’m really interested in the triage process at MS. How much is typically deferred and why? How much is “never done”? How often does the “fragile code that no one wants to touch” come up and what is being done to ensure it doesn’t happen in the future? I have a feeling the decisions made in these meetings might have different criteria for what is or is not critical for a prod release and I'm genuinely curious...

The stuff on “bug bars” was equally interesting – I’ve never worked in a company where a developer could ignore even ONE bug in an app that was going to prod, let alone 20. They’re required to drop what they’re doing and fix it NOW. It encourages ownership and submission of clean code from the get-go. Isn’t the AUT priority #1? If not, why not? Inquiring minds want to know, especially when the process is so very different than much of the field…

Maybe we can all hear more in Book #2; I can tell you that I personally would have found the scope of the book so intimidating, it would have taken me two years to even gear myself up to start paragraph 1…