WORLD WATCHES IN HORROR AS EXPLORATORY TESTING IMPLODES UNDER ITS OWN WEIGHT…

Just when I thought nothing could be worse than RUP……

Overthinking and overdesigning have caused exploratory testing methodologies to become a bloated toad; mnemonic-laden, admin-heavy, and full of warts.

Don’t believe me? All I have to say to you is FCC CUTS VIDS. And that’s just “uber” mnemonics – there are lots and lots of heuristics (rules of thumb) under those. Those particular mnemonics just describe the various “tours” one can take when examining software. And it’s just one (very well-known) person’s view; you can make up your own if you like. Many “explorers” are wasting valuable time doing just that. Then they compare and critique mnemonic lists or make efforts to come up with something cute or memorable. Argh.

I once had a “discussion” with Michael Bolton in a forum. He said he had a “heuristic” called ”Overwhelm the field with characters”. I asked him what made that different than a simple “boundary test”. I did get a 4-page reply, but he never answered the question.

Let’s consider some standard heuristics for an input field:

Enter a standard number of characters
Overwhelm the input field with characters
Enter the maximum number of characters
Enter the minimum number of characters
Enter no characters
Enter invalid characters
Imbed spaces (before, within, after)

Now if I was going to memorize those and try to train someone else to use them, I might call this “SOMMNIS”. But consider how much easier life is to just train a tester on basic boundary testing, which includes all of the above and more, so that all they have to remember is “boundary testing”. Furthermore, all of the testers they meet ALSO understand “boundary testing”, rather than trying to comprehend SOMMNIS and how that fits into their OWN set of heuristics.

Boundary testing has been around, in print and practice, since at least 1973. So I can substitute two simple words for 7 heuristics, one master mnemonic, and 35 words.

Simplicity is the ultimate sophistication.

I read articles (primarily from the contextual crowd; they’re a pretty prolific bunch) about acknowledging different test schools, how differently they think and work from other others, etc., and it’s all Hooey. They feel it’s difficult to communicate with another tester unless they understand their context and “what they call what” – which they’ve tried to categorize. The sad thing is that a lot of communications issues are their own fault. They’re the ones with 3755 heuristics for something the rest of us put into one simple category. But here’s the interesting part of the entire equation. The tests are exactly the same. I had to laugh – the other day, I saw an article about “universal heuristics”. This from a group that regards “standards” as akin to “poison”. But there ARE standard tests; you can attack them, memorize them, mnemonisize them, or categorize them however you wish, but the basics remain the basics for any competent tester. So why (oh WHY???) bury basic truths under layers of complexity that make it difficult to communicate with each other, train others, or establish a true professional community?

Let’s move on to managing exploratory test sessions. When the idea of exploratory testing started to take off (it’s been around for at least 25 years) as something legitimate, I thought it was a good idea to wrap some sound practices around a chaotic process that we all got stuck with on occasion. But it has since changed from something we don’t do unless we have no choice to something testers do BY choice. And that’s where the problem comes in.

I’ve read everything, and I do mean everything, I could get my hands on in regards to managing the testing process itself for exploratory testers. And I’m not impressed. As anyone could have anticipated, they found out they actually needed to track who was doing what, what was accomplished, and present information to management in regards to where they were in the process, progress, and how much was left to do. That’s right – METRICS.

Duh.

I’ve looked at all of the examples, description, etc., I could find, and here’s what I’ve found out. Test session management is a serious issue. The tester will start a given session (either short – around 45 minutes) or long (around 2 hours) with a testing charter (what they will explore). The tester tests and makes notes on what they exercised, errors written, and questions during the session. They then consult with the testing manager/coordinator at the end of a session to further strategize or explain results. Good God, y’all.

This would mean every day you could probably expect between 3 and 7 “charters” to be tackled, and if you had a testing staff of 4, the test coordinator or manager would have about an hour left of their day to report findings to management or hold defect meetings. I have a staff of 34. That would mean putting multiple test coordinators in place, who would then have to coordinate their staff and with other coordinators, with one uber coordinator who could see the whole picture.

Let me tell a story here. When I took over at company X, there were 4 testers. They had roughly 300 “charters” to test in a 4 week period. Let me make something clear. All of the sample charters I’ve seen are identical – yes, identical – to what structured testers would refer to as a “test case”. They were stressed out, overworked, and could barely make through their tests, even with significant overtime. The error rate in production was well over 40%.

It took me about a month to figure out what the problems were and in the meantime – utter chaos.

It turned out that the QA team was informed of what was going into production the day before they started to test. You’d think that would be a perfect setup for exploratory testing. What was happening was that they were “exploring” the system and writing down what they tested DURING the test sessions (again, that would seem familiar to exploratory testers). They ALSO had to ask their questions and find out about “expected” or “desired” functionality DURING the test sessions. All of this analysis work was pushing the test sessions out as far as they could go and then some. No one could figure out where they were or how far they had to go, because no one could anticipate how many tests would be needed to test new functionality – they were finding out as they explored.

Now let’s talk about what we did about it.

First, project managers were required to engage the QA team in advance and we were notified 2-4 weeks in advance of what was coming our way. Whatever documentation was available (and sometimes there wasn’t much!) was given to us. We analyzed that IN ADVANCE, wrote up structured outlines IN ADVANCE, and tried to get issues resolved IN ADVANCE. This also benefitted the development team, as we weren’t finding major issues requiring significant development effort late in the process. The outlines were stored in a central repository (QualityCenter). When we had sketchy specs and no time, we held Just In Time test sessions and drove out the test outline with the entire project team in about an hour. Again, those were stored in a central repository. Issues/questions were brought up immediately and addressed prior to the testing period.

Tests were run from the outline, which was nothing more than an intelligent, hierarchical list of what the tester wanted to examine, with any pertinent notes of things to remember. The tester added to the outline, changed it, etc. during the testing itself, but the burden of having to write everything down as they were testing was lifted. There were fewer issues and questions to ask, because analysis had been done in advance. Anomalies were noted in an automated defect tracking system.

The capabilities of individual staff increased from 20 tests per week to over 100 tests per week. Error rates in production were reduced over 40%.

It was unnecessary to fill out “session sheets”, nor did the test coordinator have to meet with every tester at the conclusion of a “session”. All of the up-to-date info was in Quality Center; reports were available to anyone on any project team at any time. The coordinator produced a daily report to management on progress, and a daily defect meeting was held every morning to make assignments and check on progress.

I think there are no differences whatsoever to the thought process employed by skilled test personnel between QA groups. I think the primary difference is WHEN the activities take place. “Structured” testing analyzes in advance of testing; “Exploratory” testing analyzes during testing.

I’d like to suggest a meeting of the minds. Staff that analyze in advance have advantages and will be faster during the test sessions themselves, as a portion (often the most boring portion!) of their work is done. Actually, this frees them up for more creative analysis work during the actual sessions. An outline is just a list of what the tester wants to examine. It’s like a map with destination points. It doesn’t tell you what to do to get there. It doesn’t tell you what side trips to take. Or what pictures to snap. You can modify the outline during the session. Or add a completely new one. Still, operating this way has advantages. You can consider each line (or header) in your outline as a “test”, “test case”, or “charter”. This means you can count them and report progress against them. Since you designed your outline using whatever specs were available (written, verbal, formal, informal), it makes your work SOX-compliant, as well as compliant with many types of regulated test environments. Generally, as long as you can trace what was tested back to what was requested, such regulations are met. It allows for flexibility, intelligence, and creativity. Keeping the information centrally located adds to an organized effort; it’s easily trackable, with metrics readily available. It cuts admin requirements.

In addition, once the testing period is over, structured shops can expand those outlines into test cases with steps for their regression test bank, and/or turn them over for automation. Looser organizations can continue to just work from outline forever.

Overall, I think we need to recognize what everyone has in common, rather than our differences, and work harder to establish methodologies that use the best ideas across the board and self-regulate against time-wasters. Maybe we should think a bit more before we design something “new” without any understanding of what’s already out there and would work well for us with a few tweaks. Sometimes I think we end up redesigning the wheel. As a triangle.

Then we wonder why the ride is so bumpy…