BECOMING MOBILE

Walking that fine line between ambulatory and agile…

Many articles and posts I read involve decisions that can only be made by managers or PMs. It is unusual, to say to the least, for a testing practitioner or manager to be able to dictate project methodology. That’s usually handed to you. And the methodology you get handed provides some framework for what you need to do in order to provide services that will feed that project’s success.

I think it’s pointless to try to tell a highly regulated shop that they need to become agile. That involves a culture shift from the very top that impacts all areas of an IT shop. Unless you have both the power and authority to effectively drive that kind of massive change successfully, you’re going to be in a position where you need to provide deliverables based on what has been mandated to you.

This is equally true with agile shops that need more structure.

Overall, it’s been proven time and time again that people resist change. Think that’s not true in the IT field, which changes all the time? I can only say that those that have done a lot of “selling” of ideas that are different from what they are currently doing to IT folks would disagree with you. And in my experience, the younger personnel are less flexible than the more seasoned staff. Often younger, less experienced personnel have only done things one way and therefore hold on to the only processes they know. With maturity you can sometimes get understanding that there is more than one way to go from A to B.

So I’m not going to argue about whether regression testing is/should be required. In most cases, it is. And B.J. Rollison (IM Testy) is expressing the whys and the wherefores of that far more eloquently than I. I don’t really care if you don’t believe in producing plans, specs, test cases, or metrics. In many cases, it’s going to be required. You won’t get a choice. So saying they suck, suck, suck is just an exercise in beating your head against a brick wall. You’d need to change the entire IT organization, not just testing, to change some of those realities. And the opportunities to change an entire IT organization are few and far between.

What I WOULD like to talk about are general groupings of project methodologies and strategies for moving your testing activities from a structured shop to a more flexible shop, where possible. I’d call a structured shop “ambulatory”. They move from point A to point B in a very deliberate way, with many mandatory, formal touchpoints. Then there are those shops that are truly “agile”. Daily builds, XP or Scrum-like methodologies, frequent deployments, often with many informal touchpoints. And then I think there is the “silent majority”. Everything that falls between those two. For the sake of this blog, I’m going to call that group “mobile”.

What I’d like to discuss is how to implement some “mobile” test strategies into an “ambulatory” project methodology.

Again, most practitioners are not in a position to dictate or influence how they do their jobs. That is established by their company, their manager, or some combination. There are those test personnel, however, that are either the “only” tester in their organization, the “senior” or “lead” tester in their organization, or the manager/director for their testing organization. I’d like to address that group specifically, as they are the group most likely to be able to influence the way QA/QC provides support to a given project. Consultants hired to handle the testing for a given project are also often in a position to negotiate how the testing is performed for that project.

Some of the problems I’ve encountered with providing deliverables in a highly structured environment are as follows:

1. The process requires detailed specification, which is neither complete nor
accurate, and which is not available in time to produce detailed test cases.
2. Standards are used more as weapons (“You didn’t follow Paragraph 3, Section 6!”)
than as aids to producing good products.
3. The testing organization will not begin their work until the specs are complete
and have sign-off. The development organization, however, started coding
immediately.
4. The process requires an overview and signoff of the test cases, and there are so
many test cases, none of the reviewers take the time to review them. Another
hitch may be that reviewers take weeks to read them and focus on the incorrect
test step in test #253, or the misspelled word in the test description of test
#642, rather than the intent of the test.
5. Clients, PMs, or executive managers request plans before anyone has any idea of
the length and breadth of the project.
6. Documents get requested in order to check that document off a list and not
because the document provides value to the team. I’d like to mention that
several consulting firms are heavily into producing useless documentation as
well; it takes time to produce and they bill their clients for that time.
7. More time is spent waiting, attending meetings, or playing political turf games
than working.
8. There is dependence on an international, national, regional, or other standard
that no one has actually read or analyzed. This is where statements like “We
have to do this because of ISO Standard X (or IEEE, or SOX…)” come in.
9. The company or testing organization has been doing things in X way because a
highly-paid consulting firm or past manager (long gone) set it up that way for
them, and what they set up was the most time-consuming, expensive, and therefore
lucrative proposition for themselves. Some consulting firms are also so lacking
in ethics that they actually set up processes that make it virtually impossible
to do the work without them.

Now you might work in an environment that is highly structured and has none of these issues. Well, if it isn’t broken, there’s no need to fix it.

But for those of you that have (or are) experiencing some of this pain, I’d like to make some suggestions based on what has been successful for me and for my teams in similar situations.

I’d like to start with point 8, because there is so much miscommunication involved with adhering to standards. First of all, I’d strongly suggest that you take the time to actually READ the standard. Many standards provide GUIDELINES and they say so. Further, many may recommend given deliverables, but they do not specify what those deliverables “have to” look like and/or when they are produced. These are important points.

Many, if not all, practitioners that have to work under a given standard have no choice. SOX, for example, is mandated for anyone working for a financial concern. That means if your company is owned by a much larger holding company, you will have to be SOX-compliant, even if your primary business has absolutely nothing to do with finance. It is common with IEEE or ISO shops to be so mandated because your company regularly does business with overseas firms that will not do business with you unless you adhere to those standards. I once worked with a company that had to be certified mafia-free in order to do business in Italy. No one normally adheres to some standard for fun or because they don’t know what else to do. They do so because they must in order to do business.

You do, however, have more flexibility than you think within those standards.

Let’s start with SOX compliancy. In order to be SOX-compliant, you must be able to prove that what was moved into production was what was requested by the end user. An auditor is normally going to want to see specifications, tests that link to those specifications, that the tests passed and/or that non-passed tests have defects written up, and that the “passed” application was moved into production. In essence, you must prove that what was requested was tested and delivered.

SOX does not actually specify how or in what format you have to provide ANY of these things. Your specs can be in any format and so can your testing artifacts. They can be produced at any time and in almost any order. The key piece of information involved in remaining SOX-compliant is some sort of traceability. You must be able to prove to an auditor that you tested and passed what was requested by the business user.

Well now.

One of the overriding problems with trying to produce full test cases in advance of a migration is that the specs often aren’t complete or at the level of detail necessary to do the work with any degree of accuracy. Even if you were given enough time to write full test cases, the amount of re-work necessary after the system is delivered makes it extremely difficult to keep on top of the task and perform it well.

What if you could be SOX-compliant (or meet other standards), prep for a migration, maintain traceability, perform the testing with the same amount of rigor or more, and produce test cases for future migrations and automation, retain your structured focus, and do it 400% faster? Would you try it? Do you think your management would allow you to try it?

Mine did. In multiple organizations. And if it worked for me, it can work for you too.

The basic premise is simple and can also be applied/adapted to those testing organizations that want to eventually fully adopt exploratory testing techniques. The key to gaining acceptance is in terminology and the centralized structure you use to store your “artifacts”. The word “structure” is very comforting to companies highly dependent on standard. At the same time, your “centralized structured outline repository”, which links to your specifications (even if those specs are written on bar napkins), is really nothing more than an intelligent list of test ideas. But in a structured shop, you call them “test conditions”. This is an important point if you need to sell this concept to your management. “Test conditions” will get faster buy-in than “test ideas”. I'm not into business psychology, so I couldn't tell you why, but I can certainly pass on the information so that you can use it to your advantage. When working in a structured organization, always utilize structured terminology. "Structured outlines, centralized repositories, test conditions, test cases, test artifacts, etc." are Your Friends.

If you are fortunate enough to have an automated test manager, such as QualityCenter, this task becomes even more simple. There is nothing easier than using the requirements tab of QC to list your test conditions/ideas. It will even number them for you, or if you have to link your test idea to a specific line in a spec, you can easily add a user field to do just that. If you don’t have an automated test manager, it doesn’t matter if you put your lists into Excel spreadsheets or even Word, as long as you and your group pick one format and put it into a centralized location.

So you’ve got a list of test conditions. What’s next? Well, the list of test ideas is sufficient to satisfy SOX requirements. All you would have to do from there is ensure you could note whether each one passed or failed, and that any defect you wrote would link back to the test condition and spec. Basic traceability. This will also satisfy most auditors and if you examine those ISO and IEEE standards, test conditions also fit into their recommended workflows. Overall, the point here is that you can conduct your testing from nothing more than these test conditions/ideas. Are there disadvantages to this? Yes. You generally cannot “hand off” a list of test conditions/ideas to someone that is not familiar with the project/app under test. If, however, the author of the test conditions is the same person that will do the testing, there is no problem. It may seem strange, at first, that there would be no other “down” sides to working without a stepped-out test case, but think about the way you yourself test a given app. For example, if I’m testing interest or maturity on a deposit, my test idea might be “withdraw funds; verify change in rate”. Now in order to execute that scenario, I have to be a customer, have an account, and have deposited money into something that accrues interest and changes rates at certain deposit levels. But I, personally, know how to do those things. I don’t need instructions. The only person who would need instructions is someone who had never worked with these applications before. Therefore, if I am the tester, there is no loss of rigor or “completeness” in executing the test condition. In addition, I can write thousands of test conditions/ideas in a few weeks. I can only produce 40-80 detailed test cases in that same timeframe.

However, many testers that work in structured environments also have to produce test cases that become part of a regression test case base and/or get turned over to an automation team. This means that ultimately the person who wrote the test conditions will not necessarily be executing them going forward.

The process I’ve found most successful is to write the structured outlines (these are your test ideas/test conditions), and then to write “stub scripts”. This also lends itself well to a test manager, where you have to pull tests into a “lab” in order to run them and the tool allows you to mark them pass/fail and to provide completion percentages and some reports.

When you have a list of test ideas, generally you organize them into areas that make some logical sense to you. If you were an exploratory tester, perhaps you’d call it a “tour”. Or you have a number of test ideas that are going to be part of a given “test charter”. When writing structured test cases, it is equally common to put a number of test conditions into one test case. So a “stub script” can be a tour, charter, or test case. It is a logical grouping of a number of test ideas or conditions into one testable entity. That entity can be as large or as small as you wish. The stub script itself contains very basic information, like what test ideas/conditions are part of the stub, any prerequisites, and the purpose or a brief description of the stub. Again, these are very fast and easy to write. The advantage to a stub script is that it CAN BE COUNTED AND THEREFORE USED TO REPORT PROGRESS. For those that are in structured organization, these stubs can be expanded into test cases AFTER THE TESTING IS COMPLETED.

If you really look at most standards, they might recommend or mandate test cases, but they rarely say “when” that task MUST be done. There are a lot of advantages to putting your “steps” into a test case after the testing is complete. Have you ever had to write test cases for a product already in production? I’ve always found that extremely easy – the end product acts as a far better prototype than anything you’d receive before the fact. Logically, you can consider expanded test cases to be a “maintenance” task. Cleaning everything up and getting it ready for turnover to someone else (or to a regression test case base or automation team). Maintenance generally takes place after production. One of the reasons I prefer to expand test cases after production is that I am not spending enormous amounts of time (and money) modifying and changing individual test steps before or during the test process. There is far less change involved once a given product is in production.

Overall, this type of process can address problems 1, 3, 4, 7, 8, and 9 above. Reviewing a list of test conditions, for example, is far easier than reading stepped-out test cases and if you work in an environment that requires a sign-off on test artifacts, it shortens the time required (and gets better coverage from reviewers). You don’t have to “wait” for completed, signed-off specifications to start writing down test ideas. What’s more, you can, if you wish, move to even more agile methods for getting those test conditions/ideas, like just-in-time test technique. You haven’t “skipped” any artifacts, you’ve just rearranged them to make the process more efficient. This allows your test team to be ready to test more quickly and to be more responsive to change. In some organizations, where the testing group is viewed as a bottleneck, this type of process can turn that perception around. And last but not least, it’s likely that this is similar to your “real” process anyway, but that you experience a lot of thrash every migration attempting to do the impossible before testing begins. It’s very difficult to get enough time to write fully expanded test cases prior to the start of testing. So why not tweak what you’re doing, and end that particular pain?

As this blog ended up far longer than I anticipated, I’ll have to deal with some strategies for the other problems in another blog, but if any of you have experienced the same issues I have in my list above, I encourage you to try re-arranging and simplifying the process in a similar way. You can always start by trying it on a small project to see how well it will work in your environment, and most managers/executive managers will be willing to allow such an experiment on a smaller effort, especially when the results might end up saving them both time and money going forward...