I CAN NAME THAT TUNE IN....

Do any of you remember the Game Show “Name That Tune”? Two people went back and forth trying to trump each other as to how they could successfully name a tune in the least number of seconds. The person who bid the lowest number of seconds had to name the tune and if they did it successfully, they won Fabulous Prizes.

The arguments between manual and automated testing staff are a lot like Name That Tune. Automated testing inevitably runs faster than a manual tester can work. But sometimes faster isn’t better. Sometimes, you can’t name a tune in 2 seconds.

I was recently asked for a cost/benefit analysis for automation. This isn’t my first cost/benefit analysis for this company. It’s possibly my tenth.

I’ve been asked for many similar analyses at other companies. So I thought I would share some of what I’ve learned from these exercises.

First, there are various types of “automation”. For the purposes of this blog, I’m going to focus on automation of manual regression test cases. This is outside of JUnit, CUnit, internal-to-the-code, white-box testing. This is not performance, stress, or load testing. Those, by their very nature, need to be automated.

I’m talking about the typical company, who wants to automate the test bank run by their manual testers in order to save time and money. They’ve talked to some tool vendors and while they like the look and smell of the KoolAid, they aren’t drinking it quite yet. Hence, the cost/benefit analysis.

It’s infinitely easier to do such an analysis if you are NOT an employee of the company and do not already have an automation team, which I do. I already know the constraints that exist in this environment, and believe that given the direction of the company, they must invest in automation if they want to retain current coverage, keep headcount down, and maintain current error rates (which are low) in production. If they’re willing to give any one of those things up, we can move to a manual model and save some money.

That’s right. Save some money. Right now, today.

If you’re experienced, and you’re NOT an employee of a given company, the first thing you know is that automation is expensive in terms of time, resources, and money. Many companies come into it believing what those tool vendors tell them – that in 6 months, their 100,000+ regression tests will be fully automated and will run in one day. Unattended.

Not.

I spent ten years working on nothing but automation projects, and we have an automation effort underway right now. In my experience, it will take at least 2 years to get your automation program running smoothly and to the point where it starts paying for itself. That’s right. 2 years.

And for those two years, the company will do nothing but fling money down what may appear to be a giant black hole.

Automation is simply not for the faint of heart. It’s not for the tight of budget, either. And above all, it’s not for the inexperienced.

Automation of this type is not, ever, a short-term proposition. Automation, particularly in terms of benefit, requires mature management willing to INVEST in a project that, if done intelligently, can pay off Big Time in the long term.

Overall, many companies will not benefit, at all, ever, by automating existing manual regression tests. That said, some companies will benefit to the tune of millions (that’s right – millions) of dollars per year.

So what kind of environments benefit from automation, and what kind of environments don’t?

When you’re considering cost/benefit, the first thing you need to look at is how the company operates, what kind of projects/products you produce, and what you’ve got in the way of test assets right now.

For example, if your company produces products with a “shelf life” of less than 2 years, and your extremely short-staffed team works exclusively on agile projects, funded reluctantly by a company that continually wants all of you to do more with less and perform heroics to get a 10-man job done by 3 people and the janitor, and your test assets consist of notes you took on a toilet-paper roll when you finally got to take a body break after working 12 hours straight, you have a problem.

If your company produces products with a longer “shelf life” and has more than 10,000 tests that HAVE to be run for every migration, or you have one set of tests that have to run through 10,000 different configurations, it’s a different story. If your company spends at least a million dollars per year on testing, you’ve got some traction. And if your company is interested in long-range planning and savings, I’d say you should explore the possibility of automation further.

Here are a few realities as I’ve experienced them:

1. Your automation team, whether consultant or FTE (full-time employee) will cost you plenty, particularly if you want your team familiar with one particular tool set. You might want to focus on programming skills, rather than skill with one particular tool in order to keep things realistic. Someone who kicks automated butt is likely going to be equally skilled, motivated, and driven to kick butt on any tool set.
2. Since good automators are tough to find, whether you’re hiring FTEs or contractors, expect the search to take you some time, and expect to pay whatever top buck prevails in your part of the country. Most companies that have their hands on someone talented is going to keep them, so finding great people is a combination of determination, skill, and luck. In addition, make sure you have people making hiring decisions that can ask the kind of technical questions that separate the wheat from the chaff. You need programmers, not people who can set checkpoints. I’ve found, particularly in the past few years, that there’s a strong division of talent. More than half do work that involves MODIFYING what has already been set up. What you want is the kind of talent that can DO the set-up. It’s not uncommon for anyone who can spell “test tool” to put test tool experience on their resume.
3. It’s common for managers or executives to believe that the bigger the team, the more work will be produced. And often that’s true. However, the first steps in a good automation effort are to hire the right people to do the job, and to get your framework and processes “stood up” before you throw a bunch of people at it. It doesn’t take ten people to design and develop an automation framework. Plan on two for 3-6 months. THEN staff the rest of the team. And consider investing in a lead who really knows what they’re doing.
4. The tools themselves will cost you some money, and the add-ons, maintenance contracts, etc. will continue to cost you money. HP/Mercury tools are really “famous” for this. It would behoove you to check your region and find out what tools are most prevalent, because it will make it easier for you to find people with the right skill sets.And you’ll need more than just tools for your automation staff. If you want them kicking off multiple runs, you’ll need additional PCs. You’ll need space to store tests and results. Since they’ll be working as auto tests are running, they’ll likely need dual PC’s and monitors. Some things can be set up virtually, some can’t. All of it will cost money, time, and resources.
5. It takes TIME to automate a test. Around 3 hours. That’s a blended estimate, of course. Some will take 3 days. Some will take 5 minutes. But your team will probably never automate as many tests per year as you think they will when you start the automation effort.
6. Before anyone automates a single test, you’re going to have to decide what kind of automation framework you want to build, and then you’re going to have to build it. Forget what you’ve heard about capture/playback. You can go that route, but maintenance is an expensive nightmare. It’s more likely you’ll settle for a keyword/data driven model, just for ease of update. How long will that take? 3-6 months.
7. Before anyone automates a single test, you’re going to have to establish object repositories and function libraries, if you opt for a keyword/data driven model. How long will that take? It depends on the size of your apps. We have about 15,000 tests and it took us a year. We were somewhat held back due to a ginormous technology change project; they converted screens in stages and we had to follow them.
8. Before anyone automates a single test, you’re going to have to analyze your tests and determine what can, cannot, should, and should not be automated and come up with a plan that prioritizes the work. Contrary to popular belief, the work should be structured so that the easiest, low-hanging fruit is tackled first. Why is that? Because you’re going to run into technical snafus and you want those solved before you tackle the more complex and critical functionality.
9. You’re going to need to manage expectations and do a lot of education. Many companies believe you’ll automate everything and will be able to fire your entire manual test team. Generally speaking, a good automation effort will allow you to maintain your CURRENT headcount. We were adding 4-5 resources per year just to handle the expanded test case base. We’ve been able to maintain the current headcount for 2 years because of our automation effort. Without it, we’d be paying another 7 or 8 personnel right now.
10. Another common misconception is that you can hire people to do the automation and then they can all go away after the project. Nope. By then you’ll have new tests to automate. The existing auto tests will have to be maintained. And it’s unlikely your manual testing staff will have the knowledge base to make program changes for complex technical issues.
11. In regards to expectations, you’ll also have some managers that think all you do is a press a button and everything will run itself in an hour. Ahem. This is a common misconception encouraged by tools vendors. Yes, kicking off the job is merely the press of a few buttons. But results HAVE TO BE ANALYZED. Errors will either need bug reports or the auto script itself might need to be modified (due to data, programming errors, etc.).
12. Some managers and executives will think everything can be automated. It can’t. For example, we have a set of Grid Controls from Hell that do not lend themselves to automation. You will run into similar issues. Some tests SHOULDN’T be automated. Most efforts, if you’ve got a great automation team, can be 60% automated or more. That said, we have some apps that are at less than 20%. We have some over 70%. We don’t have a single app at 100%.
13. Some managers and executives will think that not only can everything be automated, it can be automated before the application is available in (some) sort of integrated environment. Not. It is a Big, Fat, Honking Waste of Time to automate what is going to become a regression test before it’s been run manually at least once and the results are as expected. There may be similar issues with people who think a test bank designed to run in one environment is going to magically work in some different environment/configuration. Unlikely, to say the least. Most automated tests are technology-driven and are somewhat dependent on internal code or controls that are transparent to a manual tester.
14. Your cost avoidance (not having to hire additional people) is a cumulative savings. So if you don’t have to hire another 4-5 people this year, and don’t hire another 4-5 people next year, your savings in cost avoidance will be 8-10 resources next year.
15. Your automation resources and assets are pure expense. You could, given sufficient bodies, do the work manually. Automation is normally considered an internal efficiency effort.
16. You’ll probably have to change the way you do business in order to accommodate the needs of an automation team and automated test bank. It’s more difficult (not impossible, just more difficult) if you have test data/prerequisites that constantly change. It’s better to create test data than to work from a production overlay. Changes to functionality and screens, etc. have to be communicated to the automation team ASAP. That can be difficult if communication of changes isn’t handled well in your company.

Whew! All of that said, why does anyone automate?

Money, my dears.

All of that expense and time is worth it if you’re saving hundreds of thousands of dollars per year.

So how does one start determining whether you can and will save hundreds of thousands of dollars per year?

Sometimes it’s quite obvious. You have more than 5K regression tests which must be run for every migration, your annual budget for QA is at least a million dollars, the growth of either your work or your team has been over the top for at least the past two years, and you’ve determined that model is not supportable long-term. This all points towards automation as a partial solution. Why partial? Remember that automation will not address every test in your regression test bank. If you can automate 60%, that leaves a 40% growth rate for your manual testers that you’ll have to address in some other way.

Or maybe you have 500 tests, 2 people on your staff, and an annual budget of less than 200K. There is no point in investing in automation for this scenario. Your costs to get an automation program in place and running will far exceed what you can hope to recoup. And this, to my mind, is what is most disgusting about tool vendors and what is told to people that lack experience with automation of this type. It has to be worth the money. If you’re small, it quite simply IS NOT GOING TO BE WORTH THE MONEY. You’ll spend a million or two on any effort that’s actually good – just to get everything going and reach the point the team starts to pay for itself. If your company doesn’t have a million or two to spend, you’re usually going to either end up with something incredibly bad, or you’ll end up abandoning the effort partway through. Running those 500 tests manually will be cheaper. And unless the two manual testers can build an automation framework and do the automation themselves in their spare time, the effort will never be cost-effective.

Where have I seen automation used effectively? Large banks. Insurance companies. Large software houses. Large companies that offer world-wide services of some kind. Telecom companies. State, federal, and heavily-regulated industries. Any type of company involved in travel or logistics. Several types of scientific research organizations. Companies with large client bases. Manufacturing concerns that are heavily regulated. Web-based companies with a large market share. See the theme here? These are the types of organizations that typically have large regression test banks.

Where have I seen it fail? Most automation efforts fail the first time. It’s incredibly difficult to hire someone who really knows what they’re doing or who can handle the inevitable snafus that will come up along the way. But the companies I’ve seen who have abandoned the effort altogether have all been small companies with limited budgets, or excessively large companies that don’t care if they have to hire ten quadzillion manual testers, especially if they’re consultants. In many large companies, consultants are not considered (financially) in the same way as FTEs (Full Time Employees); the bucket for “bubble” resources is easier to tap. That means they’re more likely to hire contractors and less likely to invest 2 million dollars in automation and a permanent automation team.

What kind of questions have my management asked me? Most recently, my management has asked to understand the costs and benefits of automation, by quarter, for the next 3 years. Fortunately, our investment (i.e. “throwing money down a pit”) years are behind us. We will be “in the black” this year, and savings will be racking up to the tune of over 500K in the next two, and close to a million thereafter. And THAT is what makes automation of a manual regression test bank worthwhile. I’m not even mentioning some intangibles, like running repetitive tests that would take 8 people 5 weeks is now done in 2 days. Or that the automated scripts find as many or more errors than human beings did. Of course, the type of testing I’m talking about would probably be referred to as “checks” by some people and I wouldn’t necessarily disagree. I, personally, am quite appreciative of any automated procedure that takes away what is a relatively thankless and mind-numbing process for an expensive and easily-bored human being and actually does it better.

Overall, however, the warning here is that you need to remember that Naming that Tune might be achievable in 2 seconds, as long as you’re willing to pay $500,000 per second to make it happen. Is it worth it? Well, if you have thousands of tunes to name, yes. When you make good choices as to when and how to automate, and you see a plan actually come together (and many thanks to my own A Team!), it truly is a Beautiful Thing. Cost-effective, too.