TESTING VS. EXERCISING

Are you taking the dog for a walk or making him jump through hoops?



I see a significant difference between testing code and exercising code. If someone sits me down at a PC and tells me how to access a given app, I can immediately begin to exercise the code. I can move through the screens, try various options, etc.

But is that testing?

Not really.

I consider that exercising the code. I might, as I move through the options and try different actions, run into something that APPEARS TO ME to be an anomaly – an error of some kind. I can write that up. But the things I find that appear to be incorrect will be based on my own experience and opinion as to proper system operation. I have over 25 years of experience, so chances are pretty good I’ll “guess” right. But what if I have only one or two years of experience? Chances are I’ll miss a lot of errors because they look OK to me (or I never tried certain scenarios) and/or I’ll write up a bunch of errors that aren’t really errors. This is, to my mind, the worst type of “exploratory testing”. It isn’t “testing” and I feel it drags out testing sessions unnecessarily, as someone relatively clueless tries to find out what “should” be happening.

Testing implies a comparison of to . Car analogies are a bit passé, but consider if you were “testing” the thickness of the outer shell of a car door. There are federal standards for this. What if you didn’t know what the standards were? What would you be able to determine? Well, you could determine the thickness of the outer shell. But you wouldn’t know if it was good or bad. Maybe you’d have a personal opinion as to good/bad, but you wouldn’t know if that test “passed” and would meet federal standards.

Performance/load/stress testing is interesting in this way. You might get a requirement that says at peak load of 2000 users, response time must remain at under 4 seconds. That is something measurable; usually it can be tested. But you can get other types of requests to just start at and add 100 users until the team “finds out when it breaks”. “Breaks” may mean anything – timeouts, lockout, crashes, etc. Then the project team looks at the results and decides if the breaking point looks acceptable. The dangerous thing about this type of exercise is that the team can guess wrong. Many times they’ll make their decision as to what is acceptable based on wishful thinking or optimum conditions. Or even time left in the schedule to tune. I’ve watched project teams ignore results (logically trying to justify “why” results were bad in QA) because the QA system did not match production (more power in production), only to have exactly the same catastrophic events occur within one hour of going live. That same type of risk is inherent in any type of “exercising” vs “testing” scenario. You’re guessing. Some people are good guessers and some are bad guessers.

Say I’m given a new website to “test”. I have no specifications to speak of; maybe a few Emails. I bring it up and the color scheme is puce, orange, and violet. All of the submit buttons contain little happy faces. Now because I’ve been in the field for a while, I’m probably going to go back and ask if that’s REALLY what was desired. I’m also going to make my professional opinion clear. But if you give that to someone with little experience, they might just blink a few times, and start exercising features. In other words, they will ignore what they are unable to definitively view as an error. They will focus only on items that their own limited experience tells them is absolutely wrong. If you’re working with an off-shore team, they would also be reluctant to tell a client their color scheme appeared to be vile and to ask questions.

I believe that in order to “test” something, you must understand and have knowledge of what the system is supposed to do. You should have some understanding of what it should look like and how it should behave. You can then design tests that verify it does what it is supposed to do, looks the way it should look, etc. It then becomes significantly more apparent what type of tests you need to run in order to ensure you can’t make the system do what it is NOT supposed to do.

Structured testing is a logical progression. Unstructured exercising of code is all over the map. Structured testing finds a significant number of errors. Unstructured exercising might find 10%, and is based on luck more than skill.

Here’s the “rub”. Exercising code is fun and easy. Testing code requires thought, intelligence, and a skill set.

I believe that without specifications of some kind – format doesn’t matter – using the term “testing” is specious.

We have many common issues in this field and they haven’t changed much in over 20 years. One of those is poor specifications. The format and methods for arriving at specifications have changed, but one of the major problems most QA staff cite when discussing their jobs is poor specifications. It affects everyone – development, QA, the end user, and on and on. Experienced QA staff learn to ask questions and find the answers they need. But not everyone is experienced – either on the QA team, development team, or downstream groups. So what can be done?

Well, mentoring and training of your own QA staff is first on the list. A tester afraid to ask questions is inevitably a bad tester. An untrained tester won’t find as much as a trained tester. Every member of my staff is required to go through a 3-day bootcamp to ensure we’re all on the same page. That means we all use the same terminology, and everyone has been trained on the same basic test techniques in the same way. This doesn’t mean we’re locked into ONLY those techniques, but it does ensure everyone on our staff is trained and expected to be familiar with the basics. Everyone in our field is so very dependent on specification of some kind, I’d advise every QA professional in the field to start supporting and promoting the importance of the BA (or whomever writes specs in your firm) and the criticality of specification to the smooth, on-time delivery of a product your end users really want. Are you on an XP project? What are the user stories like? How do they progress in level of detail?

I realize users change their mind and yes, I realize requirements of any kind can change significantly during the course of a project. I realize that just the word “specification” is an anathema to some people. But format of specifications is immaterial, and even an informal change approval process can solve that problem. The point is that the project team needs to know what to deliver and the test team needs to know what they are testing against. And they need to know with sufficient time left in the schedule to do their jobs the right way. If you do not have anything to test against, you’re just walking the dog – exercising the code hoping to stumble over something important. Anyone can walk a dog. Does it make you crazy when you hear developers, BAs, etc. tell you they’ve done QA? Every fool who has ever walked through a website or clicked on a button thinks they know how to test. Are they right? Is that all you do? When I’m feeling Evil and I get that type of comment, I ask them where their test artifacts are located (blank stare). I ask them about their favorite test technique – equivalence partitioning (blank stare)? Paired testing (blank stare)? By then they get the idea and I can just laugh and say “Everyone tells me that.”.

My blog here is not going to magically change The Way Things Are. There will still be applications “thrown over the wall”, bad specs (or no specs) will still abound, and some testers will still meander around in new code, totally clueless sightseers, tripping over a bug now and then out of sheer luck.
But if this makes you think – even for a few minutes – about what you supposedly “test”, the QA world will become a better place for a few shining moments… I sincerely hope all of you champion improvement and positive change. It’s part of our mission. I feel a Blues Brothers song coming on….