SO YOU NEED TO SET UP A QA/QC ORGANIZATION....

This blog is targeted towards those in a position to hire someone to set up a QA/QC organization. You may be a CEO, CIO, VP, Director, or Manager.
It is probable that this need has come up in response to either a problem or series of problems that need to be solved, or you are pro-actively taking steps to ensure problems do not occur in the future.

Normal concerns in such cases are time, cost, and finding a solution that gets the job done right. I’m going to make the assumption that you want to hire someone to take charge of this “project” or perhaps assume management/directorship of a QA/QC organization.

In order to hire the right individual, some assessment of what you need is in order and realistic expectations in regards to cost have to be set.

First, how big is your organization? Are you a small shop with six programmers? Or a large organization with 200 programmers? Your strategies should be different for these two situations.

Do you need a QA organization, a QC organization, or both? If you want a group that helps you become certified through some governing body, like ISO, CMMI, or similar, or you want a group that will write, update, and validate deliverables against a Systems Development Life Cycle manual, then you need a QA organization. If you’re looking for a group that will test new and existing code for errors, you need a QC organization. Most companies start with QC and move into QA, and the terms “QA” and “QC” are interchangeable in many industries.

QA staff and QC staff have different skill sets, as do experienced managers or consultants in those areas.

Let’s begin examining options related to size of the IT organization.

Small shop, no QA/QC

It’s likely in this case that you envision just one or two testers helping out your developers. If you don’t picture any future growth or need for additional testing staff, you can either just hire testing staff and have them report to the development or project manager, or pull in consultants as right-to-hires, which will allow you to “try before you buy”.

If, however, you feel the development organization might expand in the future, I’d suggest considering the possibility of hiring a talented, experienced lead and giving them the word “manager” in their title.

Why? Because you’ll get a better quality of candidate for a management position and because titles are free. The general rule of thumb for any startup is to hire the absolute best candidate you can find for the primary position. Like tends to hire like. That means an experienced, talented lead or manager is going to hire experienced, talented staff.

One pitfall to avoid is to move a resource from some other area (like development) into the quality role. This resource will have a huge learning curve, will likely not know or be able to recruit experienced, talented staff, and will not know how to handle common quality issues. If the resource is talented, it will come in time, but do you want to wait 2-5 years for the resource to mature into the job? In addition, the frustration involved when there is no senior, experienced mentor to assist often causes these resources to quit and move on to less problematic roles elsewhere.

You’ll also need to think about reporting structures. While is easier to have resources just report to the development or project manager, and it’s a workable solution for environments that develop projects using XP methodology, it is difficult on a long-term basis to have testing staff reporting to development management. Quality concerns, bugs, or issues are often shuffled under the table at the development/project manager’s discretion, and few development or project managers understand QA or QC, and therefore have difficulties developing and supporting staff involved in the effort. This means the testing function is no more than a very light risk management group, and you will not be able to retain talented career professionals. If you have an Operations or Support organization, that is a good option for placement of QA/QC. Placing QA/QC under Development or PMO organizations are generally poor choices.

Small shop, existing QA/QC

The immediate tendency if there are existing QA/QC staff is to put one of them in charge. There are several problems with that. First, is the candidate you’re considering experienced enough for the job? Maybe they do well right now, but do they have proven experience with the expanded job responsibilities? Are you (and they) prepared for the fallout from their peers when you elevate one of them? How will non-QA/QC personnel react to this person’s new job responsibilities? Often, they won’t recognize it at all. Will this resource be given the authority or report to someone with the authority to handle these situations?

What often happens with an existing QA/QC group when either one person is elevated or a new manager is brought in is that the existing staff either consciously undermine their new leader or they leave. All the more reason to ensure the person you put in charge is experienced and capable of handling the situation.

Existing QA/QC groups, whether large or small, are the most difficult of all scenarios to step in and improve or manage. Generally, it is better to hire from outside. If, however, you have a stand-out employee that you want to promote, that individual is going to need significant support in order to be successful.

Reporting issues are the same as “Small shop, no QA/QC”.


Medium shop or large shop, no QA/QC

This is where things get interesting and choices become more varied. A medium shop usually has at least 40 developers. A large shop has 200 or more.

The process involved with setting up an organization in either case is the same.

There are so many variables involved with establishing a QA/QC organization, including environment, existing staff, existing processes, etc., that the easiest and most prudent, cost-effective method is to pay someone to come in and assess what would be needed to set up QA/QC for your organization. A good assessment will take 4-12 weeks, cost between 50 and 200K, and will give you a roadmap of what needs to be done, both short-term and long-term, to get from Point A to Point B. It will make recommendations specifically based on your organization, setup, and needs, and if done properly, can be implemented in stages by personnel other than the assessor.

I’ll be spending some time talking about assessments (for practioners performing assessments) in a later blog.

Why pay all that money for an assessment? Primarily because if you have a medium to large size IT organization, your investment in QA/QC is going to exceed a million dollars per year. It makes sense to ensure the effort is performed only once and is done right. A good roadmap will save months of work and rework down the road. It’s like paying an architect for a blueprint. You can build a house without one, but it probably won’t stand up.

Can you just hire a QA Manager/Director/VP and expect them to produce the assessment and then implement the findings? Yes.

The key in both cases is finding the right person to do the job.

This is a good time to bring up snake oil salesman. There are many consulting firms out there that know you have money. They want it. They are going to tell you exactly what you want to hear in order to get it. Many firms will try to sell you a “package” that will set up your QA/QC organization for you. They’ll take care of the processes, procedures, environment, people, etc. They’ll guarantee that you’ll have it all in 6 months.

Caveat Emptor – buyer beware.

That is one tempting offer. Remember what your mother said about “if something sounds too good to be true, it usually is”? Mom is always right. There is no set of processes, procedures, and tools that is going to fit right into your environment. “Ah,”, the salesman will say, “Our suite is Totally Customizable!”.

What that means is that they have no idea what is going to work for your environment. They have a certain set of templates, processes, and procedures that they’re going to try to MAKE work in your environment. Their contracts and SOWs will be filled with caveats. Your staff will have to be available, they’ll have to follow the new regime, equipment/software will have to be purchased as recommended, etc., etc., etc.

The cost for these packages is going to be over 150K. It will take at least twice as long and cost twice as much as advertised (hey, once you’re hooked and have spent 150K, you’ll spend another 150K in order to not have to start over). And the entire time the firm is there, they’ll be trying to get funding for additional efforts or place staff from their own firm into yours.

It’s incredibly difficult, time-consuming, and expensive to get involved in these efforts. It’s never easy to set up a QA/QC organization, but these firms can make a painful process excruciating and it’s close to impossible to extricate yourself once you’re “hooked”.

One of the dangers with these firms is that it is probable the ”lead” on your project is not going to be flexible. What they understand, promote, and implement is THEIR package. Your agenda is not necessarily their agenda. Their agenda is to take as much of your money as they can.

This does not mean I don’t like or support consulting firms. I do. But not used in this capacity.

You will get more “bang for your buck” and get more of what you need implemented (and more quickly!) with an assessment by an expert not affiliated with a particular “package”. So how do you choose such an expert?

I’d suggest first making a decision as to whether you want to hire a QA Manager/Director/VP and have them take care of the assessment and implementation tasks, or whether you would prefer an independent assessment. Many managers prefer to be involved in the decision-making process, rather than being “handed” a roadmap, but a roadmap will give you a better idea of what kind of manager to hire and what qualifications will be required to fulfill your vision. On the other hand, a good manager has the same type of experience as an assessor, and you get both the manager and assessment for one cost.

Second, taking the time to find the right resource for the job is time well-spent. If you want an independent assessor, expect to pay between $150-250 US per hour for their services. The assessment time will depend on the size of your organization.

When interviewing a potential assessor, what should be of interest is how long they’ve been in the QA/QC field, how many types of methodologies they’re familiar with, what kind of tools they’ve used, etc. At a MINIMUM, an assessor should have worked on successful projects that have used the following types of project methodologies:

1. Waterfall
2. Iterative
3. RUP/TPI
4. Agile

There are many “flavors” of these basics. Ask if they have any preferences or biases towards one type of methodology. If they say no, they’re lying to you. All of us have preferences. A good assessor is aware of their own biases and can work around them. The reason it’s important that they’ve worked on successful projects using a variety of methodologies is that anyone who has worked just one time on an unsuccessful project using a given methodology will never recommend or want to work on another using that methodology. In other words, the prejudices of an assessor can prevent them from recommending what you need. For this reason, it also advisable to ask if the assessor is identified with any particular “school” of QA/QC thought. The answer should be no; anyone who strongly affiliates themselves with one type of methodology has difficulties remaining objective. The best assessors have seen it all, done it all, and can pull the best pieces from many types of practices/methodologies to help you achieve your goals.

Ask how the assessor handles problems – either personnel that don’t want to answer their questions or give them deliberately misleading information. Ask how they perform their assessments. It should be a combination of interview and examination of artifacts (documents, files, logs, etc.). Ask who they’d interview first. The answer should be you! When I’ve performed assessments, after the client sponsor, I always interview development first, as they’re a primary “client” of a QA/QC organization. If there is an existing QA/QC organization, I talk to them and examine their artifacts LAST. This is because I already know what kinds of issues QA/QC is going to have and I do not want their problems to affect my ability to listen to the rest of the company without prejudice. What is critical to success is understanding what the rest of the company needs from the QA/QC team, and only then going to the QA/QC team to determine how they are meeting those needs.

How comfortable is the assessor with talking to you? If they are awkward or uncomfortable, or have obvious problems answering your questions, they will be poor interviewers and will be unable to perform a meaningful assessment.

Find out how much experience the assessor has examining project/product artifacts. Ask what types they’ve examined and what types of issues they’ve found. Ask if they have any experience assessing staff.

Last, but not least, ask for references and copies of past assessments. Beware if the assessor says they cannot give you a copy of past assessments due to “confidentiality issues”. Reputable assessors have samples of their work available with any identifying corporate-specific information removed. You should know exactly what you’re buying. In addition to the document itself, the assessor should offer a presentation with questions/answer period to you and your senior management staff as part of the assessment cost.

Medium to large shop – existing QA/QC

This is the most difficult of all situations to handle well.

If you have an existing QA/QC organization, then what you’re looking for is either improvement or replacement of some or all of your existing QA organization, processes, or procedures.

I’d suggest always going the assessment route in this case. It gives the existing staff someone to dislike, and your assessor should be accustomed to handling hostility and fear with grace and humor. The primary fear of your existing staff will be that the assessor is there to get rid of them. It’s actually unusual for an assessor to recommend removing staff. If the existing staff lack experience, it’s more likely the recommendation will be to hire more senior members to mentor/train the less experienced, or to recommend specific training to fill gaps. It is rarely economical to replace people with specific application knowledge with staff that have a significant learning curve. They will also resent bringing in an outsider when they’ve been telling you for years “what’s wrong”. This is a maturity issue; it’s unlikely these resources have interviewed every area that interacts with them, categorized the results, and presented the information in a way that is actionable, with costs attached. Particularly if they don’t support a particular viewpoint. It is also normal for people to fear change. Again, an experienced assessor will be able to allay these fears and extract meaningful information that will support your goals.

A good assessment is a set of findings from which a roadmap of both short-term and long-term recommendations is presented, complete with probable schedules and costs. The recipient can then pick and choose what to implement, based on corporate timeframes, management vision, and cost. I’d also like to mention that it is possible to set up a working, viable, effective QA/QC organization without negatively impacting development schedules.
_________________

Well, I’ve finally run out of wind; my next blog will be directed to those who get charged with setting up a QA/QC organization…..