Software Defect Reduction Top 10 List

Software’s complexity and accelerated development schedules make avoiding defects difficult. These 10 techniques can help reduce the flaws in your code.

Barry Boehm, University of Southern California
Victor R. Basili, University of Maryland

Recently, a National Science Foundation grant enabled us to establish the Center for EmpiricallyBased Software Engineering. CeBase seeks to transform software engineering as much as possible from a fad-based practice to an engineering-based practice through derivation,organization, and dissemination of empirical data on software development and evolution phenomenology. The phrase “as much as possible” reflects the fact that software development must remain a people-intensive and continually changing field. We have found, however, thatresearchers have established objective and quantitative data, relationships, and predictivemodels that help software developers avoid predictable pitfalls and improve their ability to predict and control efficient software projects.

Here we describe developments in this area that have taken place since the publication of “Industrial Metrics Top 10 List” in 1987 (B. Boehm, IEEE Software, Sept. 1987, pp. 84-85). Given that CeBase places a high priority on software defect reduction, we think it is fitting to update that earlier article by providing the following Software Defect Reduction Top 10 List.

ONE
Finding and fixing a software problem after delivery is often 100 times more expensive than finding and fixing it during the requirements and design phase.

As Boehm observed in 1987, "This insight has been a major driver in focusing industrial software practice on thoroughrequirements analysis and design, on early verification and validation, and on up-front prototyping and simulation to avoid costly downstream fixes.” For this updated list, we have added the word “often” to reflect additional insights about this observation. One insight shows the cost-escalation factor for small, noncritical software systems to be more like 5:1 than 100:1. This ratio reveals that we can develop such systems more efficiently in a less formal, continuous prototype mode that still emphasizes getting things right early rather than late. Another insight reveals that good architectural practices can significantly reduce the cost-escalation factor even for large critical systems. Such practices reduce the cost of most fixes by confining them to small, well-encapsulated modules. A good example is the million-line CCPDS-R system described in Walker Royce’s book, Software Project Management (Addison-Wesley, 1998).

TWO
Current software projects spend about 40 to 50 percent of their effort on avoidable rework.

Such rework consists of effort spent fixing software difficulties that could have been discovered earlier and fixed less expensively or avoided altogether. By implication, then, some effort must consist of “unavoidable rework,” an observation that has gained increasing credibility with the growing realization that better user-interactive systems result from emergent processes. In such processes, the requirements emerge from prototyping and other multistakeholdershared learning activities, a departure from traditional reductionist processes that stipulate requirements in advance, then reduce them to practice via design and coding. Emergent processes indicate that changes to a system’s definition that make it more cost-effective should not be discouraged by classifying them as avoidable defects.

Reducing avoidable rework can provide significant improvements in software productivity. In our behavioral analysis of how software cost drivers affected effort for the Cocomo II model (B. Boehm et al., Software Cost Estimation with Cocomo II, Prentice Hall, 2000), we found that most of the effort savings generated by improving software process maturity, software architectures, and software risk management came from reductions in avoidable rework.

THREE
About 80 percent of avoidable rework comes from 20 percent of the defects.

That 80 percent value may be lower for smaller systems and higher for very large ones. Two major sources of avoidable rework involve hastily specified requirements and nominal-case design and development, in which late accommodation of off-nominal requirements causes major architecture, design, and code breakage. A tracking system for software-problem reports that records the effort to fix each defect lets you analyze the data fairly easily to determine and address additional major sources of rework.

FOUR
About 80 percent of the defects come from 20 percent of the modules and about half the modules are defect free.

Studies from different environments over many years have shown, with amazing consistency, that between 60 and 90 percent of the defects arise from 20 percent of the modules, with a median of about 80 percent. With equal consistency, nearly all defects cluster in about half the modules produced.

Obviously, then, identifying the characteristics of error-prone modules in a particular environment can prove worthwhile. A variety of context-dependent factors contribute to error-proneness. Some factors usually contribute to errorproneness regardless of context, however, including the level of data coupling and cohesion, size, complexity, and the amount of change to reused code.

FIVE
About 90 percent of the downtime comes from at most 10 percent of the defects.

Some defects disproportionately affect a system’s downtime and reliability. For example, an analysis of the software failure history of nine large IBM software products revealed that about 0.3 percent of the defects accounted for about 90 percent of the downtime. Thus, riskbased testing—including understanding a system’s operational profiles and emphasizing testing of high-risk scenarios—is clearly cost-effective.

SIX
Peer reviews catch 60 percent of the defects.

Given that finding and fixing most defects earlier in the project development cycle is more cost-effective than finding them later, we seek techniques that find defects as early as possible. Numerous studies confirm that peer review provides an effective technique that catches from 31 to 93 percent of the defects, with a median of around 60 percent. Thus, the 60 percent value cited in the 1987 column remains a reasonable estimate.

Peer reviews, analysis tools, and testing catch different classes of defects at different points in the development cycle.

Factors affecting the percentage of defects caught include the number and type of peer reviews performed, the size and complexity of the system, and the frequency of defects better caught by execution, such as concurrency and algorithm defects. Our studies have provided evidence that peer reviews, analysis tools, and testing catch different classes of defects at different points in the development cycle. We need further empirical research to help choose the best mixed strategy for defect-reduction investments.

SEVEN
Perspective-based reviews catch 35 percent more defects than nondirected reviews.

A scenario-based reading technique (V.R. Basili, “Evolving and Packaging Reading Technologies,” J. Systems and Software, vol. 38, no. 1, 1997, pp. 3-12) offers a set of formal procedures fordefect detection based on varying perspectives. The union of several perspectives into a single inspection offers broad yet focused coverage of the document being reviewed. This approach seeks to generate focused techniques aimed at specific defect-detection goals by takingadvantage of an organization’s existing defect history.

Scenario-based reading techniques have been applied in requirements, object-oriented design, and user interface inspections. Improvements in fault detection rates vary from 15 to 50 percent. Further, focused reading techniques facilitate training of inexperienced personnel, improve communication about the process, and foster continuous improvement.

EIGHT
Disciplined personal practices can reduce defect introduction rates by up to 75 percent.

Several disciplined personal processes have been introduced into practice. These include Harlan Mills’s Cleanroom software development process and Watts Humphrey’s Personal Software Process (PSP).
Data from the use of Cleanroom at NASA have shown 25 to 75 percent reductions in failure rates during testing. Use of Cleanroom also showed a reduction in rework effort so that only 5 percent of the fixes took more than an hour, whereas the standard process caused more than 60 percent of the fixes to take that long.

PSP’s strong focus on root-cause analysis of an individual’s software defects and overruns, and on developing personal checklists and practices to avoid future recurrence, has significantly reduced personal defect rates. Developers frequently enjoy defect reductions of 10:1 between exercises 1 and 10 in the PSP training course.

Effects at the project level are more scattered. They depend on factors such as the organization’s existing software maturity level and the staff’s and organization’s willingness to operate within a highly structured software culture. When you couple PSP with the strongly compatible Team Software Process (TSP), defect reduction rates can soar to factors of 10 or higher for an organization that operates at a modest maturity level. Results tend to be less spectacular if the organization already employs highly mature processes.

The June 2000 special issue of CrossTalk, “Keeping Time with PSP and TSP,” offers a good set of relevant discussions, including experience showing that adding PSP and TSP to a CMM Level 5 organization reduced acceptance test defects by about 50 percent overall, and reduced high-priority defects about 75 percent.

NINE
All other things being equal, it costs 50 percent more per source instruction to develop high-dependability software products than to develop low-dependability software products. However, the investment is more than worth it if the project involves significant operations and maintenance costs.

The analysis of 161 project data points for the Cocomo II model resulted in an added cost of 53 percent for its “required reliability” factor, while normalizing for the effects of 22 other factors. Does this mean that Philip Crosby’s landmark book, Quality Is Free (Mentor, 1980) had it all wrong? Maybe for some lowcriticality, short-lifetime software, but not for the most important cases.

First, in the Cocomo II maintenance model, low-dependability software costs about 50 percent per instruction more to maintain than to develop, whereas highdependability software costs about 15 percent less to maintain than to develop. For a typical life cycle cost distribution of 30 percent development and 70 percent maintenance, low-dependability software becomes about the same in cost per instruction as high-dependability software—again, assuming all other factors are equal.

Second, in the Cocomo II-related quality model, high-dependability software removes about four times as many defects as average-dependability software, which in turn removes about four times as many defects as low-dependability software. For example, consider an average-dependability system such as a commercial billing system, in which the operational cost of software defects—due to lost worker time, lost sales, added customer service costs, litigation costs, loss of repeat business, and so on—roughly equals life-cycle software development and maintenance costs. For such a system, the increased defect rate of using low-dependability software would make its ownership costs roughly three times higher than the ownership costs of high-dependability software.

TEN
About 40 to 50 percent of user programs contain nontrivial defects.

A 1987 study in this area (P.S. Brown and J.D. Gould, “An Experimental Study of People Creating Spreadsheets,” ACM Trans. Office Info. Sys., July 1987, pp. 258-272) found that 44 percent of 27 spreadsheet programs produced by experienced spreadsheet developers contained nontrivial defects—mostly errors in spreadsheet formulas. Yet the developers felt confident that they had produced accurate spreadsheets.

The creators of Web-programming facilities face the challenge of providing their tools with the equivalent of seat belts and air bags, along with safe-driving aids
and rules of the road.

Subsequent laboratory experiments have reported defective spreadsheet rates between 35 and 90 percent. The analysis of operational spreadsheets reveals defect rates between 21 and 26 percent; the lower rates probably stem from corrections already made during operation.

Now, and increasingly in the future, user programs will escalate from spreadsheets to Web scripting languages capable of sending agents into cyberspace to make deals for you. The ranks of “sorcerer’s apprentice” user-programmers will also swell rapidly, giving many who have little training or expertise in how to avoid or detect high-risk defects tremendous power to create high-risk defects.

One study for the Cocomo II book estimated that the US will have 55 million user-programmers by 2005. If we classify active Web-page developers as userprogrammers, this prediction appears to be on track. Thus, the creators of Web-programming facilities face the challenge of providing their tools with the equivalent of seat belts and air bags, along with safedriving aids and rules of the road. This software engineering research challenge is one of several identified by a National Science Foundation study, “Gaining Intellectual Control of Software Development,” which we recently summarized in Computer (May 2000, pp. 27-33).

Surely, our list can benefit from refinement and further empirical research on defect reduction. Much of the data we’ve reported, for example, fails to account for the interaction between many of the variables that, if known, could provide answers to questions like

• If I invest in peer reviewing, Cleanroom, and PSP, am I paying for the same defects to be removed three times?
• How much testing would this investment enable me to avoid?

We hope to involve the software community in expanding the Software Defect Reduction Top 10 List and other currently available data into a continually evolving, open source, Web-accessible
handbook of empirical results on software defect reduction strategies. We also plan to initiate counterpart handbooks for commercial-off-the-shelf-based systems and other emerging software areas.

We welcome your participation in this effort and urge you to visit the CeBase Web site at http://www.cebase.org for further information. You can also find an expanded version of this column at http://www.cebase.org/defectreduction/top10.

Barry Boehm is director of the University of Southern California Center for Software Engineering. Contact him at boehm@sunset.usc.edu.

Victor R. Basili is a professor in the Institute for Advanced Computer Studies and the Computer Science Department at the University of Maryland. Contact him at basili@cs.umd.edu.

Careers in Software Testing

Software testing has broken free from the shackles of being just a part of software development process to a full fledged industry domain. If you have the ability to think out-of-the-box, can look at a situation from zillion different angles, can withstand work related stress and crack an application using stress tests then you can look at this domain as a sound career option. It not only provides tremendous opportunities for career growth but also decent remuneration

Rahul Sah and Isha Gakhar

Friday, May 01, 2009

http://pcquest.ciol.com/content/content_ITcareers/2009/109050101.asp

Those who say software testing as a career is inferior to software development need to get a sanity check done. Software testing is as important as software development activity. Do you know how much a simple software mistake can cost you? Rs 1 lakh, 2 lakhs, 3 lakhs...?? No clues?
$125 mn!!. Yes, in the year 1999 when NASA lost the $125 million Mars orbiter, a spacecraft that was meant to be launched in space to study Martian climate, weather, CO2, etc. This happened due to a flaw in its software.

Actually, there were two different teams involved in the mission-Lockheed Martin Engineering which built and developed Mars Orbiter for NASA and the other was the NASA flight team-both of which used their own units of measurement during the operation. Lockheed Martin engineering team used English units of measurement while NASA's team used the conventional metric system for a key spacecraft operation, which caused the navigation information to go haywire and the spacecraft crashed. Had proper procedures for the navigational software been followed, this irregularity might have been detected and the loss could have been averted. This is a classic example that shows the importance of having good software testing practices at place and the consequences one might have to face for not following them.

Arun Rao, Vice President, Global Human Resources, AppLabs A Testing professional needs to understand the technicalities of databases, languages, GUI frameworks and OSes. He also needs to have analytical sense of mind and the ability to break open applications that seem to be perfect, and that how he can find loopholes in the applications. Today, organizations don't just want testers to have knowledge of testing but also require to have business domain knowledge, as well.

Mukesh Sharma, Founder & CEO, QA InfoTech The demand for niche skills like SOA testers are on the rise. Also, there are lot of avenues in test automation areas - scripting skills in the tools languages like VB, Java and other scripting languages like Perl, Shell, Python etc. Technical resources with capabilities to evaluate automation tools, create automation framework and reusable components are on demand. There is always a demand for good performance testers who can analyze performance test results, identify the bottlenecks and recommend tuning techniques.

Today organizations have realized the importance of software testing and they do not want to face the embarrassment of their product or solution failing just because of a small mistake that gets overlooked.

Ramanan R V, Head of Global Delivery and Chief Software Architect, Hexaware Technologies Software defects are so prevalent and detrimental that they cost the U.S. economy an estimated $59.5 billion annually -about 0.6% of the GDP, according to a study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST) in 2002. So Quality assurance is a very decisive stage as testers should guarantee that all functional needs are met even when there's higher load.

Ananda rao Ladi, Vice President, MindTree Specializations like security testing, SWOT testing, performance testing are the ones one can pursue in addition to create a lane for himself in the industry. There is a lot of need for testers to have industry domain specialization. For example, if you have been testing in the insurance world, you will be valued very high. Specializations with respect to type of testing or specialization with respect of the industry domain can be a good thing to pursue. Unlike developers, testers are expected to know everything about the product and application. Therefore, testers can even become a domain specialist, or a business analyst or even a product manager.

Nowadays companies work at avoiding such defects from the start, so it is not just about detection of defects anymore. That's why they adopt standards like CMM, ISO, Six Sigma etc and taking all these developments as a cue, some IT companies have begin to leverage the power of good quality assurance practices, which has opened several new avenues in the field of software testing. Some companies have started positioning themselves as Independent Software Testing (IST) service providers who provide specialized software testing services to other organizations, to test their products or to device software testing practices for their processes.

Shreyas Merchant, PMP Associate Practice Director - Applications & Integration Services, Fujitsu Consulting India Today clients are very cognizant of the cost implications of poorly tested products or solutions and strive to avoid this situation under all circumstances, since these directly result into financial as well as opportunity loss. Comprehensive and effective testing is the only �knight in shining armor� that could help businesses to plug these holes successfully. The only challenge is the current gap between demand-supply of Professional Testers which is widening across all geographies.

Manish Rathi, Head Delivery Management, Version 1.0/ New Venture Services, GlobalLogic In software testing area any knowledge which is gained as part of courses can just act as a starting point; however the major evolution happens on the job. Testing pros should look for opportunities to get hands-on experience which will help them sharpen their software testing skills. Software testing is more about being applied and pragmatic rather than just following academic experience. My organization while assuming that pros have a good academic back ground looks for a good attitude & ability to work in fast-paced environment.

Looking at the pace of the recent developments happening in this field, there is no denying the fact that software testing is fast emerging as a lucrative career option; it enables rapid career growth, and has substantially moved away from the myth that if somebody can't make a career in development, then only does he settle down into testing.

Opportunities
Although testing is an integral part of SDLC, emphasis is given to keep it independent from development. In the past, testing was carried out after development was over. Today this has changed and the testing activity starts right from the requirements definition stage. In many organizations today, 'test driven development' is the norm. This increased focus on quality has greatly increased the scope of testing activity. Testing has become a serious career option for software engineers; in fact a fresher or some one with 2-3 years of experience, might eventually find testing to be the fastest way to gain domain expertise. This niche area created demand for testers and quality assurance professionals for myriad of opportunities. Careerwise and also remuneration wise, the growth opportunities for a professional in testing domain has increased. Testing is now compared at par with other IT services in terms of revenue for companies and also for the kind of remuneration and career growth it has on offer for a professional to enter this domain.

However, software testing should not be taken as an alternative to software development. And as a career option it is not just open for software professionals but also for professionals from other industry domains, as today software testing has not just been confined to application testing but to the whole process of development and also the need to adhere to quality norms. Because of that, testers do not have to look at the propriety of the software code, but have to look at all the scenarios that the software will be subjected to when it is deployed across production systems or when it will be deployed at the client's end. This not only requires software knowledge but also requires you to have to have the knowledge of the domain itself. For example, if you have to test a BFSI application, you not only have to check the UI or security or load but the actual requirements of the domain you are testing the application for. For eg, for the BFSI domain, you can use your knowhow of user requirements in banking and the various working procedures to create test scenarios for the application.

Harini Raghavan, Senior Software Test Development Engineer, Microsoft India Development Center Understanding different types of testing (performance, stress, security, etc) as well as knowledge of a few testing techniques, experience in creating test plans and test cases, knowledge of different software development cycles are a plus for a fresher. If one possesses the right skill set with a good technical background the opportunities are plenty. At Microsoft, the developer to tester ratio is 1:1. About half of our annual hiring for Software Development Engineer in Test (SDET) comes through the campus recruiting pipeline.

Shashikant Mohite Chief Manging Director, SQTL The knowledge gained in academia software testing is not sufficient as chances of the person going in depth to study a specific topic is rare. For example, topics like GUI testing, database testing, systems test design may not be taken up in their curriculum. Certifications for software testing help them in gaining additional knowledge about the subject and also in terms of employment. Certifications like CSTE (QAI), CSTP (IIST), ISTQB Foundation Level, ISEB Foundation Level are some of the recognized certifications recognized worldwide.

Skillsets required
The primary skill that one expects from a software tester, apart from computer science fundamentals, is good problem solving and analytical skills, ability to understand patterns in problems and creative thinking to fulfill requirements of a million customers. They should also have skills on test automation tools for regression automation and performance testing, database skills, analytical and lateral thinking skills, programming skills, knowledge of test management and test processes (including test planning, test design, test execution, defect management and testing metrics), knowledge of the platform used for developing the application (Microsoft, Mainframes, Java, etc). Software test engineers need to have knowledge of programming languages like C, C++, VB and scripting languages like VBscript, Shell, Python, Perl. These skills are in demand with the increasing demand for automation engineers. The technical skills should be supplemented with domain expertise also to form a holistic team for testing. Gaining prior exposure to usage of some of the prominent automation testing tools from companies like IBM, HP, Mercury etc can boost chances of employability especially for fresh graduates who aspire to become professional testers.

Nowadays domain knowledge is being preferred from software testers by the companies. Professionals coming from different industries like banking, Finance, Insurance, Retail, etc have strong understanding of business knowledge and business processes. These people can work as business analysts in software organizations. They can also help to validate business processes and provide valuable comments during software requirements and design phase and also devise requirements for the end user for an application or product.

The visual shows a typical career path for a fresher who can choose either Testing or QA has the growth path.

Automation, non-functional testing (performance testing, security testing, usability testing, localization testing) are niche areas in testing. One can learn these skills after getting expertise in the basic testing area. Test professionals can also learn new test case designs and optimization techniques to boost their career in this domain.

New trends in testing
New and emerging technologies such as Service Oriented Architecture (SOA), Software-as-a-Service (SaaS), Cloud Computing, virtualization and advent of mobile technologies are radically changing the trends in application testing. Some of the recent trends in software testing domain indicate an
upsurge in:
a) SOA Testing
b) Security testing
c) Testing in Cloud Computing environments
d) Tool based regression automation and performance testing
e) On-demand testing services
f) Risk-Based testing
g) VoIP application testing
h) Health care

Risk-based testing is a new concept. As software organizations face increased pressure to complete testing in available time, risk-based testing approach is followed. Due to increase in business complexity, testing scope has been steadily increasing, but at the same time the testing window remains the same or is shrinking. In risk-based testing, areas having major business risks are tested before others. This way, it is ensured that at least business critical functionality is tested properly.

Anthony D'souza, Director-Testing Business Unit,Tech Mahindra �Although there is an economic slowdown, focus on software quality has not been reduced. In many cases there is a tremendous pressure on IT services providers to do more testing in lesser time. This is resulting in testing attracting some of the finest talent as we see many game-changing methodologies being introduced that are bringing greater efficiencies and provide opportunities to testing professionals. Product testing is a very big market, and many product companies spend upto 50% or more of product development costs on testing products. This trend is a good one for software testers.

Professionals from these domains can become business analysts and quality testers by lending their expertise for creating test bases for applications. Having the domain experience helps an individual to understand user requirements and also to devise test scenarios for applications based on experience, thus going beyond the manual testing of a code for a given application.

Career path
Software testing professionals have many interesting options today to design their career path based on individual preferences. One can take a plunge into the world of business applications by moving into functional testing or keep in constant touch with technology by focusing on automation testing.

They also have the option to get into ground-breaking areas such as professional games testing and mobile device testing.

A fresher who enters the industry as a test engineer, can choose the testing line or else move towards the quality assurance line where he can become QA lead and then QA manager. A typical career path for a professional tester would unfold as a junior test engineer, test engineer, test analyst, test lead, QA manager, followed by program manager/COE head.

Certifications
Academic knowledge in software testing is the starting point which brings in more awareness of the available career options in IT industry. This needs to be further augmented with specialized trainings in different areas of testing based on individual preferences and market needs. Certifications in testing also play a major role in meeting the industry trends. A stamp of approval from a credible institute by means of certifications improves the confidence of the employers as well as clients on the testing professionals. There are several institutes such as, International Software Testing Qualifications Board (ISTQB), American Society For Quality (ASQ) and Quality Assurance International (QAI) that offer vendor neutral certifications. Tool specific certifications are available from OEMs such as IBM, HP, Borland, etc. Obtaining certifications like CSTE (Certified Software Test Engineer) from recognized institutions or from online certification site like Brainbench can help a professional get a competitive edge over others during recruitment.

There are institutions like Edista Testing Institute dedicated for Quality and Testing curriculum. Professionals who want to seek their career in software testing or want to enhance their understanding of various quality standards can gain certifications from these institutes. For freshers, the organizations themselves train them on various testing tools and quality standards. But, a test engineer who has already gained some experience can look forward for a specialized certification to jump-start his career.

Talking about the requirements in Indian IT industry for software testers, Pradeep C, CEO of Edista Testing Institute (ETI) says, Indian market itself requires 35,000 testers approximately to bridge the gap. And, that gap is continuously increasing to almost 1,650,000 in the year 2013, as per the projection. Thus a professional can safely bet on this niche domain for his career and having some domain experience behind can be of much demand by software testing organizations.

Interview of Rex Black

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=8)

Rex Black is founder and principal consultant of Rex Black consulting services. He has written two books Managing the Testing Process and Critical Testing Processes. He has provided many templates for download on his site http://www.rexblackconsulting.com.

WhatIsTesting interviewed him recently. We hope that this interview will be useful for you.

Q1. Rex, please tell something about your background, your experience, and non-testing interests.

I have spent over 20 years in the software, hardware, and systems engineering business now. Most of that time has been as a tester and test manager, though I have also worked as a programmer and system administrator. I?ve worked with shrink-wrapped software for start-ups and big Information Technologies projects for banks. I?ve worked with the Department of Defense and with university research facilities. I?ve worked on PCs (starting back in the CP/M days), Linux, SGI, HP/UX and Solaris workstations, Unix, VMS, and AS/400 minicomputers, and even big mainframes.
In my varied experience, I?ve learned that, in testing, common themes run through projects that involve different technologies, different business problems, and different teams. The specific context matters, of course, but the commonality of problems?and solutions?is what strikes me.
When I?m not working, I try to spend time around the Texas Hill Country with my wife and daughters, out hunting with my dog, or fixing up our house.

Q2. What are your comments on the state of software testing in industry today?

Software testing?like programming?is in a state of intense change. We are seeing new technologies in both; testing tools and programming languages are evolving rapidly. We are seeing new project methodologies in both; quality risk analyses and solid unit testing are gaining ground. We are seeing new approaches to organizing projects in both; outsourcing is big for testing and development. As is the case in all industries throughout history, times of intense change are times of what the economist Schumpeter called ?creative destruction.? These changes will destroy many existing organizations, approaches, and accepted ideas, but those changes will also create amazing opportunities and growth for the educated, the clever, the disciplined, and the industrious.

Q3. Even after decades of practice software testing still remains a much misunderstood process in industry. Everybody agrees that it is important but nobody agrees on exactly how or what to be done. Your comments?

Indeed, there is a large gap between known best practices and typical practices in software testing. I believe this is true for software engineering in general. Think of how many projects are managed without even a cursory work-breakdown-structure. Read Brook?s The Mythical Man-Month: You?d think he was describing software engineering projects being done today, but he wrote that book based on experiences in the 1960s and early 1970s!
Education is part of the answer. People who want to prosper in this time of change will need to master the known best practices. There are plenty of books, articles, and courses available for those committed to doing so, in software testing in particular and in software engineering in general.
Just knowing the ideas is not enough. You must be clever enough to adapt these ideas to your circumstances. As I wrote in Critical Testing Processes, there is no such thing as the One Right Way to do anything. However, there are general ideas that clever people can fine-tune to fit their circumstances.
Discipline and hard work are also part of the answer. The lazy and the undisciplined cannot expect to survive in tough economic periods, particularly when major changes are occurring. Many of these best practices require patiently gathering data, incrementally improving, sticking with hard work, over long periods of time.

Q4. Do you think that software testing profession is maturing as an engineering discipline?

Yes. We are clearly seeing more people understanding the need for specialization in testing. We are seeing more books, educational offerings, and consulting services that specialize in testing. That said, we still have a long way to go, especially in terms of the gap between known best practices and typical practices that we spoke of earlier.

Q5. What is the future of software testing in your opinion?

It?s hard to say right now. I see software as in a position very similar to the automotive industry in the 1970s and 1980s. Up to that point, the US carmakers dominated the global market for automobiles, but, frankly, they built and sold low-quality cars. By applying ideas from people like Juran and Deming, the Japanese carmakers showed the world that it was possible to have high quality cars without high price tags. The Japanese helped people understand that quality does not mean luxury and ostentation, bells and whistles; quality means satisfying reasonable customer and user expectations again and again and again.
In the Japanese automotive revolution, quality control and quality assurance lead the way. They were key. There could be parallels now in software, if users and customers are ready to demand affordable software that simply works and works simply.
Certainly the economic situation and the rise of outsourcing is driving another possible parallel: The off-shore software houses, if they seize the moment, could lead the way, just as the Japanese took over the leadership of the auto industry in the 1980s.

Q6. How should a tester prepare oneself for the upward movement in management ladder as a test team lead, test manager and beyond? What are the essential qualities that they must develop?

There are many essential qualities and skills. A tester who aspires to management must understand the underlying business considerations. Technology by itself is not by the point. Technology makes business innovations and improvements possible. Technology solves problems. And technology involves costs and benefits, risks and opportunities. Management decisions revolve around such considerations.
So, a tester must grow the skills needed to effectively consider and manage these factors. Estimation and budgeting. Risk analysis. The pertinent business domains. The effective tester knows testing and technology, but transcends those areas to understand management.

Q7. Given that consumers are increasingly becoming conscious of quality, how do you see that impacting the software testing process? Or maybe consumers are not as demanding of quality as they should be?

Consumers are still not as demanding of software quality as they should be. Many users of software and systems assume that, if their computer screws up, they must be stupid. They say things like, ?Well, I wouldn?t have this problem if I knew how computers work.? Hogwash.
First, this statement is not true. I know how computers work, and I have plenty of problems with them. I call those problems ?bugs? rather than calling myself ?stupid.? Second, no one would ever make such an assumption about any other work-a-day implement or tool. Can you imagine someone saying, ?Stupid me, my car wouldn?t be stalling if I just understood internal combustion engines and could adjust my own timing belt?? Third, it is the job of those who build tools to make those tools fit for use. If the users find them clumsy or cumbersome, they should reject the tools, not insult themselves.
Sooner or later, as with cars, refrigerators, and washing-machines, computers will become common place. Then, as consumers did with other common-place appliances, tools, and conveniences, consumers will start to take them for granted. I think that day might be sooner than we think. At that point, consumer tolerance for lousy software will end.
I think we?re already seeing that in terms of IT software. Business customers?the CIOs, the IT managers, the project sponsors?are unwilling to tolerate big-budget projects that deliver poor quality, often late and sometimes not even at all. I think this intolerance of poor quality, along with a desire to save money, is part of what drives the outsource and off-shore boom. Perhaps consumers are not fair behind.
Companies?and professionals?who position themselves to thrive in a new environment of higher expectations for software will, I think, do well in the future, quite possibly the very near future.

Q8. Rex, you have written two books on software testing process management. What prompted you to write these books?

There have been books written on test design and testing itself, but not on test management. There have been books on software project management, but those books did not address test managers. So, my books are addressed to those previously-underserved professionals who are managing testing projects. Based on the fact that my first book has, in the last four years, sold over 15,000 copies around the world, I think the need is clearly there.

Q9. Why did you need to write two books? Why did you think one was not enough? How are the two books related? How should one read these books- Independently or one after the other? What is the overlap between two books?

Managing the Testing Process is about tools and techniques for test managers. Critical Testing Processes is about fitting those tools and techniques together into a successful test project, and fitting that test project into the larger project. They are distinct but related topics.
They overlap slightly, in that I can?t assume that everyone who reads Critical Testing Processes has already read Managing the Testing Process. So, I have to describe some of the tools covered in Managing the Testing Process, such as the test tracking spreadsheet when I?m talking about the test execution process.
If you?re planning on reading both, then you should read Managing the Testing Process first, then Critical Testing Processes.

Q10. What are the future trends in software testing that we are going to witness?

Clearly, international outsourcing is one key trend. The diverse set of tools for programmer testing will, I think, consolidate into tools for early testing in general, including static testing. Focusing test resources via risk mitigation is another trend.


Q11. Do you think outsourcing of testing is a good idea? What are the factors that prompt consideration of outsourcing?

Outsourcing is a good idea under many circumstances.

Certainly cost can be a factor. Geographical convenience can be a factor when the outsource test facility is close to the development facility. Specialized skills?including test expertise?in the outsource test facility can be a factor. Short-term spikes in work that can?t be handled in-house can also be a factor.

Q12. What are the main pain points of outsourcing?

It?s important to be very careful in planning and organizing distributed and outsource test efforts. Poorly planned and disorganized test efforts, run in-house, can sometimes succeed through brilliant and inspirational management and leadership. However, under such circumstances, an outsource project will fail.

Q13. What are the advantages of outsourcing?

Cost-saving, the access to specialized test professionals, and the ability to leverage strengths possessed by the outsource test facility.

Interview of Danny Faught

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=2)

Danny Faught is widely known in testing circles because of the various lists that he maintains, including the Test Tools List, Testing Contractors and Consultants List, and the Testing Courses List on http://www.testingfaqs.org.

He is owner of Tejas Software consulting http://www.tejasconsulting.com. In this interview he shares his ideas, passion and advice with the testers.

If you want to ask him more questions or you want to send any feedback on this interview, please reply to this post.

Q1. Danny, please tell us something about yourself, your background, your interests and things that you like to do in your free time. Please also tell something about Tejas Software Consulting.

First let me say that I've enjoyed reading the interviews on whatistesting.com and I'm honored to have the opportunity to share my thoughts with you.

I started as a software tester right after I got my bachelor's degree. It was really happenstance - I learned nothing whatsoever about testing in college. I was offered a job doing testing for supercomputer operating systems. This was by far the most interesting product I had the chance to work with at the time, so I thought I would try being a tester. And besides getting sidetracked doing process improvement for a while, I've been a tester ever since.

I'll never know if I would have gravitated toward testing if I had started out as a programmer instead. In a way, though, I did start out as a programmer because I had a co-op job as a programmer while I was in college.

I went into business in early 2001, the day after I was laid off by a consulting firm I was working for. Since then I've done a variety of things, including training, consulting, editing, publishing, and freelance writing. The business is a one-man show and I don't plan to hire any additional employees. You can learn more than you ever wanted to know about Tejas Software Consulting from my yearly wrap-up articles in the February/March issues of my newsletter (see http://tejasconsulting.com/newsletter/) or from the About Tejas Software Consulting web page (http://tejasconsulting.com/about/).

I have a personal interest that's a pretty good balance to the esoteric world I work in. I'm a bit of an "eco-freak," as those who've carefully read my newsletter might have figured out. I'm an avid composter, actually, I'm a Certified Texas Master Composter. My yard and my garden are organic. And I drive a hybrid electric car, a Toyota Prius. I could go on. This passion comes through in my work, where, for example, I always use recycled paper, and I've started asking others to do the same (so far unsuccessfully).

Another personal interest is homeschooling. My wife and I have been homeschooling our oldest daughter for almost a year, and we're loving it. I've started writing articles about the subject, and I teach a programming class for homeschool kids.

Q2. You maintain various lists such as on Test Tools, Testing Contractors and Consultants List etc. at http://testingfaqs.org. What drives you to maintain these lists?

Good question! I've maintained a variety of FAQs ever since I created the comp.software.testing FAQ. Usually I get started with this sort of project because I have a need to reference the information myself. For example, I also maintain a list of software-related organizations and other resources that are local to the North Texas area (http://tejasconsulting.com/resources/ntex.html). I took over the maintenance of the FAQs that are now on testingfaqs.org when Brian Marick said that he didn't want to do it any more. They had such a history, and were such an important part of the testing community, I couldn't stand to see them fall by the wayside.

I really enjoy the research required put these FAQs together, especially the Boneyard page, where I track historical information about companies, products, and services.

It also helps that I occasionally hear from someone who tells me how useful testingfaqs.org has been to them. That sort of feedback really makes my day. I'm embarrassed that I'm woefully behind on posting new entries that people have sent me. It's very important to me that I review the entries, make sure they're in the right category, and remove any superfluous hype. That takes a tremendous amount of time.

Q3. Do you think you have been getting enough support from people in this endeavor? What kind of support would you like to see for these lists from people?

I could certainly keep the site more up to date if I had help. I have had a few people help with things like the CGI scripting, and I've been starting to get a slow trickle of people who are pointing out new freeware tools. I think I could get more help if I could make it easier for people to help. I've thought about using a wiki, though it would be hard to maintain a consistent format on a wiki.

I'm very interested in hearing from people who would like to help, and we can be creative with the way they contribute. Of course, people can also contribute financially by buying a text ad or by sending a donation.

Q4.What is your opinion on the state of software testing today and where do you see it going?

I call myself a "Software Alchemist" because I think we're still in the dark ages of software. I believe it will be many decades before software development becomes a true engineering discipline like it should be. I'm not trying to be self-righteous in this pronouncement; I don't know all the answers either.

The average software tester is still fairly disconnected from the wealth of resources that are available. Most haven't been to a training course or university course that focuses on testing. Many haven't even seen a book about testing - they're still very hard to find in the brick and mortar bookstores. Test managers are in a similar position.

So testing often isn't much better than what common sense tells people to do. While companies are usually surviving in this mode, they could be much more efficient and produce much better products if they tapped into the accumulated knowledge that's out there.

It helps that there's a growing library of free information about testing on the web, but sorting through it to find what's relevant and what's worthwhile advice is difficult, especially for a beginner.

Where is it going? Looks like it's going to India. I'm getting an increasing number of queries from people in India, asking how to get started in testing. (One factor for this may be that my company name happens to have an interesting meaning in several Indian languages.) They seem to have an even more difficult time than average in getting access to resources.

I don't have a business background, so I won't try to guess what's going to happen to the industry.

Q5. What are the things a person who wants to choose testing as a career should do to prepare himself for it?

Do you think this question needs to be added to the testing FAQ?
I've had so many people ask this question that I've been setting their emails aside, planning to write an article on this topic to answer them all at once.

I'm going to write that article soon, but unfortunately a goblin deleted those emails, so I hope those who asked have joined my newsletter mailing list so they'll see the answer. I'm also working on a course that introduces people to testing as a career. The hard part is finding any reason to give them hope of finding an entry-level job.

Q6. What things should a novice tester do to enhance his/her testing skills for a better growth in the career?

Learn a general-purpose programming language if you don't know one already, especially a scripting language like Perl, Python, or Ruby, and also the language that the developers on your project are using. Start doing homebrew automation, automating small everyday tasks, even if test execution itself is still manual. There are things you can automate with only a few minutes or hours of effort, so you don't have to get anyone's approval to do it. People will start to ask, "Wow, how did you do that?"

Read books and articles about testing, network with people who do what you do, and attend conferences and workshops. Also learn consulting skills, even if you're not a consultant (see "The Secrets of Consulting" by Jerry Weinberg). It's easy to learn some nifty new way to do testing. It's much harder to judge whether the technique is really appropriate for your organization, and then to inject a new technique into the organization's culture.

Adopt a mentor, even if your company doesn't have a formal mentoring program. This might be somebody in your company, or someone on the other side of the world whom you've formed a relationship with.

Q7. Where do you see testing tools technology heading? Do you foresee some open-source testing tools which can give COTS tools a run for their money? How do you see your reviews of open testware tools helping the testing community?

The simpler commercial tools are probably feeling the heat from the open source community. For example, you don't see many companies trying to sell a standalone html link checker now, and I believe the availability of free link checking tools has had an influence. The commercial tools usually have several additional features added in order to make them marketable.

At the high end, the more complex types of tools probably aren't causing any commercial vendors to lose sleep yet. The commercial tools tend to have more features and are at least somewhat more reliable. I've looked at a few free load testing tools, for example. A number of expert users are utilizing these tools with good results and saving their companies tens of thousands of dollars in licensing fees. However, the tools are cranky and tend to be difficult to learn, so they're not ready for mainstream usage. Because people who do load testing have to be very experienced testers anyway, I think this is going to be the first type of complex test tool to get a foot hold. They're not just quite there yet.

Then there's homebrew automation, much of which is well below the radar of the commercial market. It's often easier to write a few lines of code to solve a specific automation problem than to find a relevant open source tool and learn how to use it. But open source also plays a big role here. The commercial vendors who have good external hooks in their tools and that don't use proprietary "vendorscript" languages are best-suited to integrate with a test team that's doing a lot of homebrew. But these teams may also be avoiding commercial tools entirely.

What I'm doing with Open Testware Reviews is finding the most promising open source and freeware tools, making people aware of them, and putting them through a grueling review. Rather than ride the wave of open source hype, I want to see what these tools can actually do and learn how difficult it will be for the average toolsmith to use them. I'm also hoping that as I share my findings with the project teams that maintain the tools, my work will help them improve the tools and make them more suitable for mainstream usage.

Q8. What is your advice to people who want to learn automated testing tools on their own? How can they do it without spending a lot of money?

That's a tough one. So many job postings demand years of experience with the big-dog tools. It's very difficult to get access to these tools without spending a lot of money on licenses. I've even seen a firm that joined a tool vendor's partner program and still couldn't get sufficient access to the tools for training purposes for its employees.

The obvious answer is to learn open source tools instead. Even better, use the tools to test another open source program, so you're doing real work rather than just playing. At least for me, I learn better when I have a real-world problem to solve. I use open source tools when I teach courses about test automation, because the participants can go back to their office and immediately start using the tool.

Unfortunately, many employers don't understand how experience with one example of a particular type of tool might enable you to easily learn how to do the same type of testing with a different tool. Maybe when an open source tool becomes as popular as its commercial counterparts, or if we get back to a situation where there are more testing jobs than candidates, open source tool experience will be a valuable addition to a resume. Certainly, there are some niche consulting firms that are having a go at marketing their skills with certain open source test tools.

Q9. You maintain a stress testing tool(stress-driver) on http://sourceforge.net/projects/stress-driver/. Can you please tell us something about this tool, how it can be useful and under what circumstances?

I originally designed this tool to drive stress tests for an operating system when I worked for Convex Computer Corporation. My team ended up using it for a wide variety of stress tests. After it was released under an open source license, I ported it to Linux and Windows and created a project for it on SourceForge. About 200 people have downloaded it so far.

The stress_driver is designed to run multiple invocations of a single test program. There are few restrictions on what the test program itself can do. I've used it to test filesystems, process management, threads, and I it used as one layer of a complex reliability test that simulated interactive user logins. You specify how many copies of the test programs to run, or a range to vary it within. You can let the individual tests run as long as they want to, or the stress_driver can stop them after a randomly determined time interval. There are many other features designed to make stress_driver flexible.

I can't guarantee that the tool will be useful for anything other than operating system testing since I haven't expanded its use beyond that context. But I think it's well-suited for a variety of situations where you have a single program that you want to run multiple copies of, with randomized variations. The test program can of course launch as complex a scenario as you want.

I haven't yet heard from anyone who has downloaded the tool, but the infrastructure is there on SourceForge if anyone wants to discuss any questions they have. Like any tool author, I'd love to hear how people have tried to use it.

Q10. What are top 5 testing books that you would recommend to people?

"Software Testing" by Ron Patton. This is a very well-written and gentle introduction to the subject. Those who already have a good grasp of testing can skip it.
"Lessons Learned in Software Testing" by Cem Kaner, James Bach, and Bret Pettichord. I've recently started recommending this one above the perennial favorite below. Very practical, up-to-date, and useful guidance.
"Testing Computer Software, 2nd edition" by Cem Kaner, Jack Falk, and Hung Nguyen. Generally recognized as the best and best-selling testing book.
"The Secrets of Consulting" by Jerry Weinberg. Testers need to be able to influence people in their organization the same way consultants do.
"Test-Driven Development" by Kent Beck. This is a growing trend among developers that stands to fundamentally affect the way software is developed and what testers' roles are.

Q11. If you were asked to name five persons who have made maximum contribution to testing, what names would you suggest?

This is actually a timely question. I'm writing a series of testing resource guides, and some of the early feedback I got was that I should add a "Who's Who" of testing. This is always a dangerous thing to do, because of the notable people who are invariably left off the list, but I'm going to give it a try.

It would take a lot of research for me to say with any confidence who has had the greatest impact in forming the collective wisdom about testing that has accumulated since the dawn of software. I'll just tell you five of the people who have had the biggest impact on me.

Boris Beizer is the grandfather of software testing. He was one of my first mentors. Many people find him difficult to like, but he nonetheless has had a big impact on the field.
Cem Kaner has done many things, from writing practical and influential books, to volunteering for the cause of software consumer protection, and now he's teaching a new generation of testers at Florida Tech.
Edward Miller facilitated my first face-to-face meetings with dozens of fascinating people in this field. His Quality Week conferences opened my eyes to many new ideas, and taught me the importance of meeting people.
Brian Marick has introduced refreshing new ideas, and has told more than one emperor that he has no clothes. I've often tried to emulate his pragmatic approach to evaluating testing techniques.
Many people still haven't heard of Jerry Weinberg. He prefers to have a profound impact on a few people rather than spreading his influence as thinly and as broadly as possible. He's certainly had a profound effect on me. But his books deserve more attention than they're getting. While he sometimes talks about software testing and practically everything he says can have some bearing on quality, what he really contributes is teaching us that all the issues are really people issues, not technology problems.
There are so many other people who have more recently affected my work and have contributed to the field as a whole. I started to list them, but the list just wouldn't stop growing, so I think I'd be safer to stop here for now. You'll have to wait for the Who's Who to see the rest.

If you were your own interviewer what questions would you like to ask yourself? They are:

1. What have you been up to lately?
I've been doing a lot of writing lately, and I dearly love doing it. But I've also finally gotten serious about convincing people to help me with writing Open Testware Reviews. So to some extent I'll take on the role of editor. I'm excited to be able to represent some different voices in Open Testware Reviews.

I've started to try to put together my first book. I really need to get a book published in order to look like a credible consultant. I have several ideas on the short list; I decided the first one to tackle is to basically take the FAQs I've been involved with over the years and put them in book form. It's frustrating seeing that pretty much all of the testing FAQs that are out there on the web now are incomplete and outdated. I can do much better than that. But I'm realizing that keeping this information up-to-date might be even harder in book form.

2. So how will you publish these lists of data that need to be updated?

I've put a lot thought into that. What I'd like to do is to publish a series of industry reports that are updated yearly. People who buy the reports will get updates for a year, and perhaps a discount on the next year's edition. There does need to be a dynamic element to it. I'm tentatively planning for the first one to include lists of conferences, professional organizations, online forums, and that Who's Who I mentioned earlier.

I'm including value-added information so that the reports aren't just lists of data. This will include cross-references, annotations, and commentary on how best to utilize the resources. The commentary may eventually grow into a book, but for now it looks like the book idea will have to wait.

Right now I'm building an international review panel that can help me make sure that the testing industry all across the world is well-represented in the reports.

3. What message do you want to leave testers with?

Testing is fun! We get to make a mess of the software and then someone else cleans up the mess. We get paid to do devious things on purpose. So have fun with it - now get back to work and break something.

Interview of Elfriede Dustin

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=4)

Elfriede Dustin is author of various books on testing - "Automated Software Testing," "Quality Web Systems," and her latest book "Effective Software Testing."

Her website http://www.effectivesoftwaretesting.com was created to create a community of software professionals in the Washington DC area.

If you want to ask her more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com

Q1. Elfriede please tell us something about yourself, your interests and things that you do in your free time.

Originally I am from Germany where I’ve received my business degree and then went on to get a Computer Science degree at KSU. In 1984 at one of my first jobs working at a US Government law center, we received a number of Wang computers and I lead the automation of the claims processes. Ever since then I have been hooked on computers, automation, and testing. Having lived in the suburbs of Washington, DC over a decade, I am now a naturalized American citizen. I enjoy spending any of my free time getting smarter in my chosen field of testing. Additionally, I enjoy spending time with my two daughters Jackie and Erika.

Q2. What prompted you to write the books that you wrote? What kind of response do you think these books have received?

In 1997 at a Rational users conference I presented the topic "Introducing Automated Testing to your Project." After the presentation I received uncountable repetitive inquiries from numerous sources throughout the world (even Belgium) about this same topic. It made me realize that there is a need for a book to put out this information and it prompted me to write the book "Automated Software Testing," together with my co-authors and former co-workers Jeff Rashka and John Paul. The books have been very well received, for example the book "Automated Software Testing" is in its 9th printing and has been translated into German, Russian, Japanese, Chinese, and is also available as a less expensive version for the testing audience in India.

Q3. What were your sources of experience that your book "Effective Software Testing" drew on?

"Effective Software Testing" is based on my many years of testing experience combined with the many years of software development experience of my former co-worker Douglas McDiarmid. The book also draws on the experiences of my former co-workers and co-authors Jeff Rashka, Program Manager, and John Paul, Developer and now CEO, as it also draws on the content of our books "Automated Software Testing" and "Quality Web Systems.”

Q4. Are you working on any other book?

I am in the process of writing a book on Security Testing for QA professionals, published by Symantec Press, written with two Symantec security experts. I feel there is a need for such a book that's specifically geared towards the testing and QA professionals. Not enough is known about how to approach security testing as a QA professional, often it's just a band-aid and too late in the development life-cycle.

My latest article Teamwork Tackles the Quality Goal was written for Software Test and Performance magazine, see http://www.stpmag.com for a free download. A summary of one of my five “Software Test and Performance conference” (see http://www.stpcon.com) presentations “Introducing Automated Software Testing” recently appeared in the SDMagazine newsletter. I am also preparing for a presentation at the ICSTest (see http://www.icstest.com) conference in Duesseldorf, Germany in April, while working full-time as an employee, i.e. an internal SQA consultant to GSS, Symantec. (Please note all opinions expressed here are my own and not those of my employer.)

Q5. Do you think there is a gap between what testing consultants and trainers teach and what the testers and test managers in the trenches face in their daily work? In what ways?

Yes, I definitely see a gap in what testing consultants and trainers teach and what testers and test managers face in the trenches. Theories are nice to have, but some theories are just not realistic and too time consuming or expensive to implement. In the trenches we constantly face ever shrinking budgets and schedules, and go-live dates and production deadlines which can’t be moved, and often the development schedule slips and cuts into the testing time.
In the trenches, so much depends on the developers’ skills. The most efficient and knowledgeable testers cannot succeed if developers implement bad software and/or ineffective software development life-cycles and processes are in place. If testing is the only quality phase implemented as part of a QA realm, it can at most be considered a band-aid often too late in the system development lifecycle to make much of a quality difference. Testing is only one piece of the Quality puzzle.
I don't see this here at Symantec, but I have seen it at other companies where there is still this general confusion by some management about what testers do. The one side of the management camp thinks that “anyone” can do the testing and no special skills are required and the other side of the management camp expects testers to “test quality into a system” and blames the tester if a defect makes it into production. When it comes to testing salaries and job requirements, generally a tester is paid less and technically less is expected of him. Generally there is still a gap between testers’ technical knowledge vs. the developer technical knowledge. This gap also rears itself in still little or no respect for the tester profession and testing still often being seen as the underdog and necessary evil. Based on this general mindset, I wrote my article Teamwork Tackles the Quality Goal on “integrating the “independent” testing team where I make the point that the tester has to be every bit as technical as the developer, working integrated with the development team, while maintaining the independent testing frame of mind. The testing profession has so much to offer, but so little about how it can really effectively be implemented is known or used.

Q6. Who in the testing community has influenced your thinking most?
Actually, my Computer Science professor Dr. Gustafson at Kansas State University has influenced my thinking about testing and quality in general. I took his graduate level course on "Quality software engineering processes" which peaked my interest in Quality and testing and I learned about great quality concepts and influential thinkers such as Tom DeMarco, Capers Jones, Boris Beizer, and Gerald Weinberg.

Q7. According to you what are top three things that one should learn/practice/be good at to be an effective and efficient software tester?

To be an efficient software tester the person really has to enjoy testing, have a knack for it, be analytical, detail oriented, and be versatile, such as a) be technically savvy, b) understand the business, and c) never stop learning and improving on the testing efforts

Q8. What do you think is the current state of automation? Where do you see automation heading towards? Do you think a paradigm shift is required in order to have more automation? Do you think there are other areas where tool vendors need to focus in addition to what they focus on right now?

Current state of automation. During a recent presentation at the Software Test and Performance conference (http://www.stpcon.com) I surveyed and learned that 50 percent of my audience had test automation tools that have become "shelf-ware." Those people are now tasked with resurrecting the automation tools and efforts, which can be an exercise in futility for many reasons. They will have to keep in mind whether the tool is still compatible with the technology being used, i.e. has the technology changed since the tool was purchased. Also, if the automation didn't succeed in the first go around did they note the reason for failure and have those "lessons learned" been evaluated so they wouldn't be repeated on the second automation attempt, etc.
This inofficial survey result is just another example that shows not much progress is being made with the use of vendor provided automated testing tools, an issue we've already discussed in our book "Automated Software Testing" published in 1999, "When inefficient automated testing implementation hasn't shown significant ROI, and budget pressures materialize, planned expenditures for test tool licenses and related tool support may be scratched. Often the tools end up as shelf-ware."

Q9. Where do you see automation heading towards?

Granted vendor provided tools continue to improve, vendors still have and will continue to have a difficult time to keep up with all the latest and greatest technologies and third party controls and widgets, which are usually the source for the biggest compatibility issues and user frustrations. People will continue to struggle with the vendor provided tools, others will turn to in-house developed automation efforts or to automation freeware. Here are some important tips before "jumping on the test automation bandwagon."
• Know the types of tools available along with the problem they are trying to solve. There is no one best tool on the market.
• Safeguard the integrity of the automated test scripts by using a configuration management tool to baseline them.
• Consider that automated testing is software development.
• Educate stakeholders about the test automation efforts, especially developers who need to understand the impact any code changes could have on it.
• Manage the expectations by not overselling test automation's return on investment.
• Do not let automated testing become a "side activity" to work on when time allows; ideally use technical testing talent whose sole focus is on test automation.
• Not all testers need to be "automators," since development expertise is required.
• Decide whether to buy or build a tool or whether to use freeware (or all of the above)

Q10. Do you think a paradigm shift is required in order to have more automation?

Unit test automation is the most effective defect detection/test automation technique. Too many companies still focus all of their test automation efforts on system testing without realizing that unit test automation can have the most ROI. Wishful thinking: Wouldn't it be nice if self-testable development languages existed, such as components are developed automated related unit tests were automatically self-generated.

Q11. In absence of licenses most people can’t learn using tools from most of the tool vendors. But employers generally want tools knowledge. How do you think testers can break this vicious circle and gain automation knowledge?

I think employers need to be educated to understand that someone with a development background can easily pick up any automated testing tool, no matter which one. When I hire automated testers and they can show extensive experience in one tool's scripting language or any scripting language for that matter, chances are they will be able to pick up and learn another tool's scripting language. Testers need to remember that automation is software development, and it is important to have experience in one or more programming or scripting languages.

Interview of Rick Craig

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=5)

Rick Craig is a well known name in testing circles because of his book "Systematic Software Testing" as well as the quality of his tutorials and presentation.

In this interview he shares his ideas and advice with the testers.

If you want to ask him more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com.

Q1. Rick, please tell us something about yourself, your background, your interests and things that you like to do in your free time.

I was born and raised in the Midwest and then moved to Annapolis, Maryland to attend the U.S. Naval Academy and then subsequently begin a career in the Marine Corps. I spent 12 years on active duty and then was recalled to active duty during the first Gulf War. I am now a Colonel in the Reserves.

Even though I am an artillery officer, I had the opportunity to attend the Marine Corps Computer Science School first as a student and then as an instructor. I got my first testing job around 1980. In 1982 I got the opportunity to work with a couple of contractors who were probably as close to testing experts as there were in those days. For 3 years they mentored me in testing, writing and presentation skills.

Then about that time Bill Perry invited me to do a key note at the First (Second?) International Testing Conference. Bill was a strong influence on my testing career at that point. I started working with Bill Hetzel and Dave Gelperin in the mid 80’s and they hired me to work for them fulltime in 1989 and I’m still with SQE.

I have a daughter, Crissy who is a Junior at the University of South Florida. I am the co-owner of a popular restaurant called Maddogs and Englishmen, which over the last 15 years has become an icon in Tampa Bay. (www.maddogs.com)

Q2. How did "Systematic Software Testing" happen? How much time did it take?

For years, the students in my seminars urged me to a write a book as a companion volume to my Test Management Course, but the problem was always finding the time…….. That problem was eased by asking my old friend Stefan Jaskiel, a technical writer to help me with the book. Even though almost all of the words in the book are mine, I doubt that I would ever have finished it without Stefan’s help and motivation.

The book took 2 months of dedicated work, plus a lot of work on airplanes over the course of a year. The book is also available in Chinese and will be out in Japanese this month.

Q3. Are there any incidents that might help future authors and that you would like to share?

Here are some tips that helped me write this first book:

1.Just do it!
2.Pick a time every day and work on the book even if it doesn’t seem like you’re making progress.
3.Get good reviewers. Listen to their input but at the end of the day, it’s your choice.
4.If you’re having trouble getting started, pick a coauthor.
5.Don’t try to make your book cover every topic under the sun………..

Q4. Are you working on any other book currently?

No, not right now. My travel and lecturing occupies most of my time.

Q5. What, in your experience, are the issues most testers face?

I think one of the major issues facing testers today (and every day in the past) is determining the value of the testing effort and determining when the task is done. I think that in some environments it is also difficult for testers to gain respect from developers and users.

Q6. Do you think there is a similarity in these issues across geographical locations or are the issues different?

I see very little difference across geographical regions (including other countries)

Q7. What, in your experience, are the areas of improvement for most testers and how can they improve?

Bill Hetzel and I did some research in the early 90’s and the one thing that I learned that stood out for me was that the best teams were the best because they did the basic things well. That is not to say they didn’t employ complex, sophisticated techniques—they did. But it was the basic stuff done well that ensured their success. I think today’s testers need to get better at estimating schedules and risk and learn some of the basic test design techniques that have been available for years.

Q8. Where do you stand between pure exploratory testing and pure scripted testing? Has your understanding changed over time? In what way?

Well, my book is called Systematic Software Testing which would lead you to believe that I am in the scripted camp. And indeed, I am in favor of creating test cases that demonstrate the users can perform those tasks that are necessary for them to be successful in their jobs. It really doesn’t matter how many bugs we find (and hopefully fix)if the users cannot do their jobs. On the other hand, finding and removing bugs can ultimately help the user perform their jobs in a more productive manner. I don’t think anyone questions the effectiveness of techniques like exploratory testing in identifying bugs. I would contend that good testers have done “exploratory-like” testing for years. That is to say that the result of one test leads the tester into the next test….. Experts like James and Cem have given exploratory techniques credibility and made it possible for mainstream testers to benefit from them.

Q9. How effective do you think buddy testing is? How cost-effective do you think it is?

Buddy testing, which is essentially working in teams and having the buddy create test scenarios before the code is written are very effective. Buddy testing, in effect, implements preventive testing at the unit level. Other techniques, like paired programming are also effective and enjoy much wider recognition.

Over the last 20 years or so, I’ve urged my clients and students to use buddy testing. I must admit that most organizations never receive enough buy-in to make it work. Literally, each programmer must learn twice as much code, which of course takes more time upfront. Those organizations that do try it though, are generally successful and in some cases greatly reduce the number of defects passing on to the test group and ultimately the users.

Q10. What is your take on certification of testers?

I think that certification is not necessarily a very good indicator of the skill or ability of a tester to do a good job. By the very nature of the training and certification testing, the focus is on terminology and “book learning”. Still without sounding too much like a consultant……I put myself firmly in the pro-certification camp. How so? If a company chooses to make certification a goal or challenge for their testers, it can be a motivator and offer a sense of accomplishment for the individual testers. Also, it can help a test manager standardize on the terminology used within the company and that alone is very valuable.

Q11. Is there any particular certification that you recommend?

I’m pretty excited about the ISTQB that is just now rolling out in the States. It has some leverage in Europe and is created by a team of experts and practitioners.

Q12. What are your thoughts in a single standard testing terminology? Why do you think one does not exist?

Well, testing was born of the need to validate software systems. The first testers tended to be developers and in some cases, users. Each group and each industry chose the words that made sense to them. When writing Systematic Software Testing, I spent quite a bit of time researching various terms and trying to come up with what was the most common usage. I can’t say that my efforts were always successful. I’m also not very optimistic that a common terminology will be obtained in the near future, although some progress is being made. I think, the internet, conferences, books, training programs, certification, etc. is gradually lessening this problem.

If you were your own interviewer what question(s) would you like to ask yourself? It is:

Q13. What is the one (ok two or three) skill that you look for most in a tester?

Good Communications skills. Creativity. Attention to detail…