Smarter Testing

Software Defect Reduction Top 10 List

2009-07-31T07:46:00.000-07:00

Software’s complexity and accelerated development schedules make avoiding defects difficult. These 10 techniques can help reduce the flaws in your code.

Barry Boehm, University of Southern California
Victor R. Basili, University of Maryland

Recently, a National Science Foundation grant enabled us to establish the Center for EmpiricallyBased Software Engineering. CeBase seeks to transform software engineering as much as possible from a fad-based practice to an engineering-based practice through derivation,organization, and dissemination of empirical data on software development and evolution phenomenology. The phrase “as much as possible” reflects the fact that software development must remain a people-intensive and continually changing field. We have found, however, thatresearchers have established objective and quantitative data, relationships, and predictivemodels that help software developers avoid predictable pitfalls and improve their ability to predict and control efficient software projects.

Here we describe developments in this area that have taken place since the publication of “Industrial Metrics Top 10 List” in 1987 (B. Boehm, IEEE Software, Sept. 1987, pp. 84-85). Given that CeBase places a high priority on software defect reduction, we think it is fitting to update that earlier article by providing the following Software Defect Reduction Top 10 List.

ONE
Finding and fixing a software problem after delivery is often 100 times more expensive than finding and fixing it during the requirements and design phase.

As Boehm observed in 1987, "This insight has been a major driver in focusing industrial software practice on thoroughrequirements analysis and design, on early verification and validation, and on up-front prototyping and simulation to avoid costly downstream fixes.” For this updated list, we have added the word “often” to reflect additional insights about this observation. One insight shows the cost-escalation factor for small, noncritical software systems to be more like 5:1 than 100:1. This ratio reveals that we can develop such systems more efficiently in a less formal, continuous prototype mode that still emphasizes getting things right early rather than late. Another insight reveals that good architectural practices can significantly reduce the cost-escalation factor even for large critical systems. Such practices reduce the cost of most fixes by confining them to small, well-encapsulated modules. A good example is the million-line CCPDS-R system described in Walker Royce’s book, Software Project Management (Addison-Wesley, 1998).

TWO
Current software projects spend about 40 to 50 percent of their effort on avoidable rework.

Such rework consists of effort spent fixing software difficulties that could have been discovered earlier and fixed less expensively or avoided altogether. By implication, then, some effort must consist of “unavoidable rework,” an observation that has gained increasing credibility with the growing realization that better user-interactive systems result from emergent processes. In such processes, the requirements emerge from prototyping and other multistakeholdershared learning activities, a departure from traditional reductionist processes that stipulate requirements in advance, then reduce them to practice via design and coding. Emergent processes indicate that changes to a system’s definition that make it more cost-effective should not be discouraged by classifying them as avoidable defects.

Reducing avoidable rework can provide significant improvements in software productivity. In our behavioral analysis of how software cost drivers affected effort for the Cocomo II model (B. Boehm et al., Software Cost Estimation with Cocomo II, Prentice Hall, 2000), we found that most of the effort savings generated by improving software process maturity, software architectures, and software risk management came from reductions in avoidable rework.

THREE
About 80 percent of avoidable rework comes from 20 percent of the defects.

That 80 percent value may be lower for smaller systems and higher for very large ones. Two major sources of avoidable rework involve hastily specified requirements and nominal-case design and development, in which late accommodation of off-nominal requirements causes major architecture, design, and code breakage. A tracking system for software-problem reports that records the effort to fix each defect lets you analyze the data fairly easily to determine and address additional major sources of rework.

FOUR
About 80 percent of the defects come from 20 percent of the modules and about half the modules are defect free.

Studies from different environments over many years have shown, with amazing consistency, that between 60 and 90 percent of the defects arise from 20 percent of the modules, with a median of about 80 percent. With equal consistency, nearly all defects cluster in about half the modules produced.

Obviously, then, identifying the characteristics of error-prone modules in a particular environment can prove worthwhile. A variety of context-dependent factors contribute to error-proneness. Some factors usually contribute to errorproneness regardless of context, however, including the level of data coupling and cohesion, size, complexity, and the amount of change to reused code.

FIVE
About 90 percent of the downtime comes from at most 10 percent of the defects.

Some defects disproportionately affect a system’s downtime and reliability. For example, an analysis of the software failure history of nine large IBM software products revealed that about 0.3 percent of the defects accounted for about 90 percent of the downtime. Thus, riskbased testing—including understanding a system’s operational profiles and emphasizing testing of high-risk scenarios—is clearly cost-effective.

SIX
Peer reviews catch 60 percent of the defects.

Given that finding and fixing most defects earlier in the project development cycle is more cost-effective than finding them later, we seek techniques that find defects as early as possible. Numerous studies confirm that peer review provides an effective technique that catches from 31 to 93 percent of the defects, with a median of around 60 percent. Thus, the 60 percent value cited in the 1987 column remains a reasonable estimate.

Peer reviews, analysis tools, and testing catch different classes of defects at different points in the development cycle.

Factors affecting the percentage of defects caught include the number and type of peer reviews performed, the size and complexity of the system, and the frequency of defects better caught by execution, such as concurrency and algorithm defects. Our studies have provided evidence that peer reviews, analysis tools, and testing catch different classes of defects at different points in the development cycle. We need further empirical research to help choose the best mixed strategy for defect-reduction investments.

SEVEN
Perspective-based reviews catch 35 percent more defects than nondirected reviews.

A scenario-based reading technique (V.R. Basili, “Evolving and Packaging Reading Technologies,” J. Systems and Software, vol. 38, no. 1, 1997, pp. 3-12) offers a set of formal procedures fordefect detection based on varying perspectives. The union of several perspectives into a single inspection offers broad yet focused coverage of the document being reviewed. This approach seeks to generate focused techniques aimed at specific defect-detection goals by takingadvantage of an organization’s existing defect history.

Scenario-based reading techniques have been applied in requirements, object-oriented design, and user interface inspections. Improvements in fault detection rates vary from 15 to 50 percent. Further, focused reading techniques facilitate training of inexperienced personnel, improve communication about the process, and foster continuous improvement.

EIGHT
Disciplined personal practices can reduce defect introduction rates by up to 75 percent.

Several disciplined personal processes have been introduced into practice. These include Harlan Mills’s Cleanroom software development process and Watts Humphrey’s Personal Software Process (PSP).
Data from the use of Cleanroom at NASA have shown 25 to 75 percent reductions in failure rates during testing. Use of Cleanroom also showed a reduction in rework effort so that only 5 percent of the fixes took more than an hour, whereas the standard process caused more than 60 percent of the fixes to take that long.

PSP’s strong focus on root-cause analysis of an individual’s software defects and overruns, and on developing personal checklists and practices to avoid future recurrence, has significantly reduced personal defect rates. Developers frequently enjoy defect reductions of 10:1 between exercises 1 and 10 in the PSP training course.

Effects at the project level are more scattered. They depend on factors such as the organization’s existing software maturity level and the staff’s and organization’s willingness to operate within a highly structured software culture. When you couple PSP with the strongly compatible Team Software Process (TSP), defect reduction rates can soar to factors of 10 or higher for an organization that operates at a modest maturity level. Results tend to be less spectacular if the organization already employs highly mature processes.

The June 2000 special issue of CrossTalk, “Keeping Time with PSP and TSP,” offers a good set of relevant discussions, including experience showing that adding PSP and TSP to a CMM Level 5 organization reduced acceptance test defects by about 50 percent overall, and reduced high-priority defects about 75 percent.

NINE
All other things being equal, it costs 50 percent more per source instruction to develop high-dependability software products than to develop low-dependability software products. However, the investment is more than worth it if the project involves significant operations and maintenance costs.

The analysis of 161 project data points for the Cocomo II model resulted in an added cost of 53 percent for its “required reliability” factor, while normalizing for the effects of 22 other factors. Does this mean that Philip Crosby’s landmark book, Quality Is Free (Mentor, 1980) had it all wrong? Maybe for some lowcriticality, short-lifetime software, but not for the most important cases.

First, in the Cocomo II maintenance model, low-dependability software costs about 50 percent per instruction more to maintain than to develop, whereas highdependability software costs about 15 percent less to maintain than to develop. For a typical life cycle cost distribution of 30 percent development and 70 percent maintenance, low-dependability software becomes about the same in cost per instruction as high-dependability software—again, assuming all other factors are equal.

Second, in the Cocomo II-related quality model, high-dependability software removes about four times as many defects as average-dependability software, which in turn removes about four times as many defects as low-dependability software. For example, consider an average-dependability system such as a commercial billing system, in which the operational cost of software defects—due to lost worker time, lost sales, added customer service costs, litigation costs, loss of repeat business, and so on—roughly equals life-cycle software development and maintenance costs. For such a system, the increased defect rate of using low-dependability software would make its ownership costs roughly three times higher than the ownership costs of high-dependability software.

TEN
About 40 to 50 percent of user programs contain nontrivial defects.

A 1987 study in this area (P.S. Brown and J.D. Gould, “An Experimental Study of People Creating Spreadsheets,” ACM Trans. Office Info. Sys., July 1987, pp. 258-272) found that 44 percent of 27 spreadsheet programs produced by experienced spreadsheet developers contained nontrivial defects—mostly errors in spreadsheet formulas. Yet the developers felt confident that they had produced accurate spreadsheets.

The creators of Web-programming facilities face the challenge of providing their tools with the equivalent of seat belts and air bags, along with safe-driving aids
and rules of the road.

Subsequent laboratory experiments have reported defective spreadsheet rates between 35 and 90 percent. The analysis of operational spreadsheets reveals defect rates between 21 and 26 percent; the lower rates probably stem from corrections already made during operation.

Now, and increasingly in the future, user programs will escalate from spreadsheets to Web scripting languages capable of sending agents into cyberspace to make deals for you. The ranks of “sorcerer’s apprentice” user-programmers will also swell rapidly, giving many who have little training or expertise in how to avoid or detect high-risk defects tremendous power to create high-risk defects.

One study for the Cocomo II book estimated that the US will have 55 million user-programmers by 2005. If we classify active Web-page developers as userprogrammers, this prediction appears to be on track. Thus, the creators of Web-programming facilities face the challenge of providing their tools with the equivalent of seat belts and air bags, along with safedriving aids and rules of the road. This software engineering research challenge is one of several identified by a National Science Foundation study, “Gaining Intellectual Control of Software Development,” which we recently summarized in Computer (May 2000, pp. 27-33).

Surely, our list can benefit from refinement and further empirical research on defect reduction. Much of the data we’ve reported, for example, fails to account for the interaction between many of the variables that, if known, could provide answers to questions like

• If I invest in peer reviewing, Cleanroom, and PSP, am I paying for the same defects to be removed three times?
• How much testing would this investment enable me to avoid?

We hope to involve the software community in expanding the Software Defect Reduction Top 10 List and other currently available data into a continually evolving, open source, Web-accessible
handbook of empirical results on software defect reduction strategies. We also plan to initiate counterpart handbooks for commercial-off-the-shelf-based systems and other emerging software areas.

We welcome your participation in this effort and urge you to visit the CeBase Web site at http://www.cebase.org for further information. You can also find an expanded version of this column at http://www.cebase.org/defectreduction/top10.

Barry Boehm is director of the University of Southern California Center for Software Engineering. Contact him at boehm@sunset.usc.edu.

Victor R. Basili is a professor in the Institute for Advanced Computer Studies and the Computer Science Department at the University of Maryland. Contact him at basili@cs.umd.edu.

Careers in Software Testing

2009-07-30T11:59:00.000-07:00

Software testing has broken free from the shackles of being just a part of software development process to a full fledged industry domain. If you have the ability to think out-of-the-box, can look at a situation from zillion different angles, can withstand work related stress and crack an application using stress tests then you can look at this domain as a sound career option. It not only provides tremendous opportunities for career growth but also decent remuneration

Rahul Sah and Isha Gakhar

Friday, May 01, 2009

http://pcquest.ciol.com/content/content_ITcareers/2009/109050101.asp

Those who say software testing as a career is inferior to software development need to get a sanity check done. Software testing is as important as software development activity. Do you know how much a simple software mistake can cost you? Rs 1 lakh, 2 lakhs, 3 lakhs...?? No clues?
$125 mn!!. Yes, in the year 1999 when NASA lost the $125 million Mars orbiter, a spacecraft that was meant to be launched in space to study Martian climate, weather, CO2, etc. This happened due to a flaw in its software.

Actually, there were two different teams involved in the mission-Lockheed Martin Engineering which built and developed Mars Orbiter for NASA and the other was the NASA flight team-both of which used their own units of measurement during the operation. Lockheed Martin engineering team used English units of measurement while NASA's team used the conventional metric system for a key spacecraft operation, which caused the navigation information to go haywire and the spacecraft crashed. Had proper procedures for the navigational software been followed, this irregularity might have been detected and the loss could have been averted. This is a classic example that shows the importance of having good software testing practices at place and the consequences one might have to face for not following them.

Arun Rao, Vice President, Global Human Resources, AppLabs

A Testing professional needs to understand the technicalities of databases, languages, GUI frameworks and OSes. He also needs to have analytical sense of mind and the ability to break open applications that seem to be perfect, and that how he can find loopholes in the applications. Today, organizations don't just want testers to have knowledge of testing but also require to have business domain knowledge, as well.

Mukesh Sharma, Founder & CEO, QA InfoTech

The demand for niche skills like SOA testers are on the rise. Also, there are lot of avenues in test automation areas - scripting skills in the tools languages like VB, Java and other scripting languages like Perl, Shell, Python etc. Technical resources with capabilities to evaluate automation tools, create automation framework and reusable components are on demand. There is always a demand for good performance testers who can analyze performance test results, identify the bottlenecks and recommend tuning techniques.

Today organizations have realized the importance of software testing and they do not want to face the embarrassment of their product or solution failing just because of a small mistake that gets overlooked.

Ramanan R V, Head of Global Delivery and Chief Software Architect, Hexaware Technologies

Software defects are so prevalent and detrimental that they cost the U.S. economy an estimated $59.5 billion annually -about 0.6% of the GDP, according to a study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST) in 2002.

So Quality assurance is a very decisive stage as testers should guarantee that all functional needs are met even when there's higher load.

Ananda rao Ladi, Vice President, MindTree

Specializations like security testing, SWOT testing, performance testing are the ones one can pursue in addition to create a lane for himself in the industry. There is a lot of need for testers to have industry domain specialization. For example, if you have been testing in the insurance world, you will be valued very high. Specializations with respect to type of testing or specialization with respect of the industry domain can be a good thing to pursue.

Unlike developers, testers are expected to know everything about the product and application. Therefore, testers can even become a domain specialist, or a business analyst or even a product manager.

Nowadays companies work at avoiding such defects from the start, so it is not just about detection of defects anymore. That's why they adopt standards like CMM, ISO, Six Sigma etc and taking all these developments as a cue, some IT companies have begin to leverage the power of good quality assurance practices, which has opened several new avenues in the field of software testing. Some companies have started positioning themselves as Independent Software Testing (IST) service providers who provide specialized software testing services to other organizations, to test their products or to device software testing practices for their processes.

Shreyas Merchant, PMP Associate Practice Director - Applications & Integration Services, Fujitsu Consulting India

Today clients are very cognizant of the cost implications of poorly tested products or solutions and strive to avoid this situation under all circumstances, since these directly result into financial as well as opportunity loss. Comprehensive and effective testing is the only �knight in shining armor� that could help businesses to plug these holes successfully. The only challenge is the current gap between demand-supply of Professional Testers which is widening across all geographies.

Manish Rathi, Head Delivery Management, Version 1.0/ New Venture Services, GlobalLogic

In software testing area any knowledge which is gained as part of courses can just act as a starting point; however the major evolution happens on the job. Testing pros should look for opportunities to get hands-on experience which will help them sharpen their software testing skills. Software testing is more about being applied and pragmatic rather than just following academic experience. My organization while assuming that pros have a good academic back ground looks for a good attitude & ability to work in fast-paced environment.

Looking at the pace of the recent developments happening in this field, there is no denying the fact that software testing is fast emerging as a lucrative career option; it enables rapid career growth, and has substantially moved away from the myth that if somebody can't make a career in development, then only does he settle down into testing.

Opportunities
Although testing is an integral part of SDLC, emphasis is given to keep it independent from development. In the past, testing was carried out after development was over. Today this has changed and the testing activity starts right from the requirements definition stage. In many organizations today, 'test driven development' is the norm. This increased focus on quality has greatly increased the scope of testing activity. Testing has become a serious career option for software engineers; in fact a fresher or some one with 2-3 years of experience, might eventually find testing to be the fastest way to gain domain expertise. This niche area created demand for testers and quality assurance professionals for myriad of opportunities. Careerwise and also remuneration wise, the growth opportunities for a professional in testing domain has increased. Testing is now compared at par with other IT services in terms of revenue for companies and also for the kind of remuneration and career growth it has on offer for a professional to enter this domain.

However, software testing should not be taken as an alternative to software development. And as a career option it is not just open for software professionals but also for professionals from other industry domains, as today software testing has not just been confined to application testing but to the whole process of development and also the need to adhere to quality norms. Because of that, testers do not have to look at the propriety of the software code, but have to look at all the scenarios that the software will be subjected to when it is deployed across production systems or when it will be deployed at the client's end. This not only requires software knowledge but also requires you to have to have the knowledge of the domain itself. For example, if you have to test a BFSI application, you not only have to check the UI or security or load but the actual requirements of the domain you are testing the application for. For eg, for the BFSI domain, you can use your knowhow of user requirements in banking and the various working procedures to create test scenarios for the application.

Harini Raghavan, Senior Software Test Development Engineer, Microsoft India Development Center

Understanding different types of testing (performance, stress, security, etc) as well as knowledge of a few testing techniques, experience in creating test plans and test cases, knowledge of different software development cycles are a plus for a fresher. If one possesses the right skill set with a good technical background the opportunities are plenty. At Microsoft, the developer to tester ratio is 1:1. About half of our annual hiring for Software Development Engineer in Test (SDET) comes through the campus recruiting pipeline.

Shashikant Mohite Chief Manging Director, SQTL

The knowledge gained in academia software testing is not sufficient as chances of the person going in depth to study a specific topic is rare. For example, topics like GUI testing, database testing, systems test design may not be taken up in their curriculum. Certifications for software testing help them in gaining additional knowledge about the subject and also in terms of employment. Certifications like CSTE (QAI), CSTP (IIST), ISTQB Foundation Level, ISEB Foundation Level are some of the recognized certifications recognized worldwide.

Skillsets required
The primary skill that one expects from a software tester, apart from computer science fundamentals, is good problem solving and analytical skills, ability to understand patterns in problems and creative thinking to fulfill requirements of a million customers. They should also have skills on test automation tools for regression automation and performance testing, database skills, analytical and lateral thinking skills, programming skills, knowledge of test management and test processes (including test planning, test design, test execution, defect management and testing metrics), knowledge of the platform used for developing the application (Microsoft, Mainframes, Java, etc). Software test engineers need to have knowledge of programming languages like C, C++, VB and scripting languages like VBscript, Shell, Python, Perl. These skills are in demand with the increasing demand for automation engineers. The technical skills should be supplemented with domain expertise also to form a holistic team for testing. Gaining prior exposure to usage of some of the prominent automation testing tools from companies like IBM, HP, Mercury etc can boost chances of employability especially for fresh graduates who aspire to become professional testers.

Nowadays domain knowledge is being preferred from software testers by the companies. Professionals coming from different industries like banking, Finance, Insurance, Retail, etc have strong understanding of business knowledge and business processes. These people can work as business analysts in software organizations. They can also help to validate business processes and provide valuable comments during software requirements and design phase and also devise requirements for the end user for an application or product.

The visual shows a typical career path for a fresher who can choose either Testing or QA has the growth path.

Automation, non-functional testing (performance testing, security testing, usability testing, localization testing) are niche areas in testing. One can learn these skills after getting expertise in the basic testing area. Test professionals can also learn new test case designs and optimization techniques to boost their career in this domain.

New trends in testing
New and emerging technologies such as Service Oriented Architecture (SOA), Software-as-a-Service (SaaS), Cloud Computing, virtualization and advent of mobile technologies are radically changing the trends in application testing. Some of the recent trends in software testing domain indicate an
upsurge in:
a) SOA Testing
b) Security testing
c) Testing in Cloud Computing environments
d) Tool based regression automation and performance testing
e) On-demand testing services
f) Risk-Based testing
g) VoIP application testing
h) Health care

Risk-based testing is a new concept. As software organizations face increased pressure to complete testing in available time, risk-based testing approach is followed. Due to increase in business complexity, testing scope has been steadily increasing, but at the same time the testing window remains the same or is shrinking. In risk-based testing, areas having major business risks are tested before others. This way, it is ensured that at least business critical functionality is tested properly.

Anthony D'souza, Director-Testing Business Unit,Tech Mahindra

�Although there is an economic slowdown, focus on software quality has not been reduced. In many cases there is a tremendous pressure on IT services providers to do more testing in lesser time. This is resulting in testing attracting some of the finest talent as we see many game-changing methodologies being introduced that are bringing greater efficiencies and provide opportunities to testing professionals. Product testing is a very big market, and many product companies spend upto 50% or more of product development costs on testing products.

This trend is a good one for software testers.

Professionals from these domains can become business analysts and quality testers by lending their expertise for creating test bases for applications. Having the domain experience helps an individual to understand user requirements and also to devise test scenarios for applications based on experience, thus going beyond the manual testing of a code for a given application.

Career path
Software testing professionals have many interesting options today to design their career path based on individual preferences. One can take a plunge into the world of business applications by moving into functional testing or keep in constant touch with technology by focusing on automation testing.

They also have the option to get into ground-breaking areas such as professional games testing and mobile device testing.

A fresher who enters the industry as a test engineer, can choose the testing line or else move towards the quality assurance line where he can become QA lead and then QA manager. A typical career path for a professional tester would unfold as a junior test engineer, test engineer, test analyst, test lead, QA manager, followed by program manager/COE head.

Certifications
Academic knowledge in software testing is the starting point which brings in more awareness of the available career options in IT industry. This needs to be further augmented with specialized trainings in different areas of testing based on individual preferences and market needs. Certifications in testing also play a major role in meeting the industry trends. A stamp of approval from a credible institute by means of certifications improves the confidence of the employers as well as clients on the testing professionals. There are several institutes such as, International Software Testing Qualifications Board (ISTQB), American Society For Quality (ASQ) and Quality Assurance International (QAI) that offer vendor neutral certifications. Tool specific certifications are available from OEMs such as IBM, HP, Borland, etc. Obtaining certifications like CSTE (Certified Software Test Engineer) from recognized institutions or from online certification site like Brainbench can help a professional get a competitive edge over others during recruitment.

There are institutions like Edista Testing Institute dedicated for Quality and Testing curriculum. Professionals who want to seek their career in software testing or want to enhance their understanding of various quality standards can gain certifications from these institutes. For freshers, the organizations themselves train them on various testing tools and quality standards. But, a test engineer who has already gained some experience can look forward for a specialized certification to jump-start his career.

Talking about the requirements in Indian IT industry for software testers, Pradeep C, CEO of Edista Testing Institute (ETI) says, Indian market itself requires 35,000 testers approximately to bridge the gap. And, that gap is continuously increasing to almost 1,650,000 in the year 2013, as per the projection. Thus a professional can safely bet on this niche domain for his career and having some domain experience behind can be of much demand by software testing organizations.

Interview of Rex Black

2009-07-29T00:58:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=8)

Rex Black is founder and principal consultant of Rex Black consulting services. He has written two books Managing the Testing Process and Critical Testing Processes. He has provided many templates for download on his site http://www.rexblackconsulting.com.

WhatIsTesting interviewed him recently. We hope that this interview will be useful for you.

Q1. Rex, please tell something about your background, your experience, and non-testing interests.

I have spent over 20 years in the software, hardware, and systems engineering business now. Most of that time has been as a tester and test manager, though I have also worked as a programmer and system administrator. I?ve worked with shrink-wrapped software for start-ups and big Information Technologies projects for banks. I?ve worked with the Department of Defense and with university research facilities. I?ve worked on PCs (starting back in the CP/M days), Linux, SGI, HP/UX and Solaris workstations, Unix, VMS, and AS/400 minicomputers, and even big mainframes.
In my varied experience, I?ve learned that, in testing, common themes run through projects that involve different technologies, different business problems, and different teams. The specific context matters, of course, but the commonality of problems?and solutions?is what strikes me.
When I?m not working, I try to spend time around the Texas Hill Country with my wife and daughters, out hunting with my dog, or fixing up our house.

Q2. What are your comments on the state of software testing in industry today?

Software testing?like programming?is in a state of intense change. We are seeing new technologies in both; testing tools and programming languages are evolving rapidly. We are seeing new project methodologies in both; quality risk analyses and solid unit testing are gaining ground. We are seeing new approaches to organizing projects in both; outsourcing is big for testing and development. As is the case in all industries throughout history, times of intense change are times of what the economist Schumpeter called ?creative destruction.? These changes will destroy many existing organizations, approaches, and accepted ideas, but those changes will also create amazing opportunities and growth for the educated, the clever, the disciplined, and the industrious.

Q3. Even after decades of practice software testing still remains a much misunderstood process in industry. Everybody agrees that it is important but nobody agrees on exactly how or what to be done. Your comments?

Indeed, there is a large gap between known best practices and typical practices in software testing. I believe this is true for software engineering in general. Think of how many projects are managed without even a cursory work-breakdown-structure. Read Brook?s The Mythical Man-Month: You?d think he was describing software engineering projects being done today, but he wrote that book based on experiences in the 1960s and early 1970s!
Education is part of the answer. People who want to prosper in this time of change will need to master the known best practices. There are plenty of books, articles, and courses available for those committed to doing so, in software testing in particular and in software engineering in general.
Just knowing the ideas is not enough. You must be clever enough to adapt these ideas to your circumstances. As I wrote in Critical Testing Processes, there is no such thing as the One Right Way to do anything. However, there are general ideas that clever people can fine-tune to fit their circumstances.
Discipline and hard work are also part of the answer. The lazy and the undisciplined cannot expect to survive in tough economic periods, particularly when major changes are occurring. Many of these best practices require patiently gathering data, incrementally improving, sticking with hard work, over long periods of time.

Q4. Do you think that software testing profession is maturing as an engineering discipline?

Yes. We are clearly seeing more people understanding the need for specialization in testing. We are seeing more books, educational offerings, and consulting services that specialize in testing. That said, we still have a long way to go, especially in terms of the gap between known best practices and typical practices that we spoke of earlier.

Q5. What is the future of software testing in your opinion?

It?s hard to say right now. I see software as in a position very similar to the automotive industry in the 1970s and 1980s. Up to that point, the US carmakers dominated the global market for automobiles, but, frankly, they built and sold low-quality cars. By applying ideas from people like Juran and Deming, the Japanese carmakers showed the world that it was possible to have high quality cars without high price tags. The Japanese helped people understand that quality does not mean luxury and ostentation, bells and whistles; quality means satisfying reasonable customer and user expectations again and again and again.
In the Japanese automotive revolution, quality control and quality assurance lead the way. They were key. There could be parallels now in software, if users and customers are ready to demand affordable software that simply works and works simply.
Certainly the economic situation and the rise of outsourcing is driving another possible parallel: The off-shore software houses, if they seize the moment, could lead the way, just as the Japanese took over the leadership of the auto industry in the 1980s.

Q6. How should a tester prepare oneself for the upward movement in management ladder as a test team lead, test manager and beyond? What are the essential qualities that they must develop?

There are many essential qualities and skills. A tester who aspires to management must understand the underlying business considerations. Technology by itself is not by the point. Technology makes business innovations and improvements possible. Technology solves problems. And technology involves costs and benefits, risks and opportunities. Management decisions revolve around such considerations.
So, a tester must grow the skills needed to effectively consider and manage these factors. Estimation and budgeting. Risk analysis. The pertinent business domains. The effective tester knows testing and technology, but transcends those areas to understand management.

Q7. Given that consumers are increasingly becoming conscious of quality, how do you see that impacting the software testing process? Or maybe consumers are not as demanding of quality as they should be?

Consumers are still not as demanding of software quality as they should be. Many users of software and systems assume that, if their computer screws up, they must be stupid. They say things like, ?Well, I wouldn?t have this problem if I knew how computers work.? Hogwash.
First, this statement is not true. I know how computers work, and I have plenty of problems with them. I call those problems ?bugs? rather than calling myself ?stupid.? Second, no one would ever make such an assumption about any other work-a-day implement or tool. Can you imagine someone saying, ?Stupid me, my car wouldn?t be stalling if I just understood internal combustion engines and could adjust my own timing belt?? Third, it is the job of those who build tools to make those tools fit for use. If the users find them clumsy or cumbersome, they should reject the tools, not insult themselves.
Sooner or later, as with cars, refrigerators, and washing-machines, computers will become common place. Then, as consumers did with other common-place appliances, tools, and conveniences, consumers will start to take them for granted. I think that day might be sooner than we think. At that point, consumer tolerance for lousy software will end.
I think we?re already seeing that in terms of IT software. Business customers?the CIOs, the IT managers, the project sponsors?are unwilling to tolerate big-budget projects that deliver poor quality, often late and sometimes not even at all. I think this intolerance of poor quality, along with a desire to save money, is part of what drives the outsource and off-shore boom. Perhaps consumers are not fair behind.
Companies?and professionals?who position themselves to thrive in a new environment of higher expectations for software will, I think, do well in the future, quite possibly the very near future.

Q8. Rex, you have written two books on software testing process management. What prompted you to write these books?

There have been books written on test design and testing itself, but not on test management. There have been books on software project management, but those books did not address test managers. So, my books are addressed to those previously-underserved professionals who are managing testing projects. Based on the fact that my first book has, in the last four years, sold over 15,000 copies around the world, I think the need is clearly there.

Q9. Why did you need to write two books? Why did you think one was not enough? How are the two books related? How should one read these books- Independently or one after the other? What is the overlap between two books?

Managing the Testing Process is about tools and techniques for test managers. Critical Testing Processes is about fitting those tools and techniques together into a successful test project, and fitting that test project into the larger project. They are distinct but related topics.
They overlap slightly, in that I can?t assume that everyone who reads Critical Testing Processes has already read Managing the Testing Process. So, I have to describe some of the tools covered in Managing the Testing Process, such as the test tracking spreadsheet when I?m talking about the test execution process.
If you?re planning on reading both, then you should read Managing the Testing Process first, then Critical Testing Processes.

Q10. What are the future trends in software testing that we are going to witness?

Clearly, international outsourcing is one key trend. The diverse set of tools for programmer testing will, I think, consolidate into tools for early testing in general, including static testing. Focusing test resources via risk mitigation is another trend.

Q11. Do you think outsourcing of testing is a good idea? What are the factors that prompt consideration of outsourcing?

Outsourcing is a good idea under many circumstances.

Certainly cost can be a factor. Geographical convenience can be a factor when the outsource test facility is close to the development facility. Specialized skills?including test expertise?in the outsource test facility can be a factor. Short-term spikes in work that can?t be handled in-house can also be a factor.

Q12. What are the main pain points of outsourcing?

It?s important to be very careful in planning and organizing distributed and outsource test efforts. Poorly planned and disorganized test efforts, run in-house, can sometimes succeed through brilliant and inspirational management and leadership. However, under such circumstances, an outsource project will fail.

Q13. What are the advantages of outsourcing?

Cost-saving, the access to specialized test professionals, and the ability to leverage strengths possessed by the outsource test facility.

Interview of Danny Faught

2009-07-29T00:42:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=2)

Danny Faught is widely known in testing circles because of the various lists that he maintains, including the Test Tools List, Testing Contractors and Consultants List, and the Testing Courses List on http://www.testingfaqs.org.

He is owner of Tejas Software consulting http://www.tejasconsulting.com. In this interview he shares his ideas, passion and advice with the testers.

If you want to ask him more questions or you want to send any feedback on this interview, please reply to this post.

Q1. Danny, please tell us something about yourself, your background, your interests and things that you like to do in your free time. Please also tell something about Tejas Software Consulting.

First let me say that I've enjoyed reading the interviews on whatistesting.com and I'm honored to have the opportunity to share my thoughts with you.

I started as a software tester right after I got my bachelor's degree. It was really happenstance - I learned nothing whatsoever about testing in college. I was offered a job doing testing for supercomputer operating systems. This was by far the most interesting product I had the chance to work with at the time, so I thought I would try being a tester. And besides getting sidetracked doing process improvement for a while, I've been a tester ever since.

I'll never know if I would have gravitated toward testing if I had started out as a programmer instead. In a way, though, I did start out as a programmer because I had a co-op job as a programmer while I was in college.

I went into business in early 2001, the day after I was laid off by a consulting firm I was working for. Since then I've done a variety of things, including training, consulting, editing, publishing, and freelance writing. The business is a one-man show and I don't plan to hire any additional employees. You can learn more than you ever wanted to know about Tejas Software Consulting from my yearly wrap-up articles in the February/March issues of my newsletter (see http://tejasconsulting.com/newsletter/) or from the About Tejas Software Consulting web page (http://tejasconsulting.com/about/).

I have a personal interest that's a pretty good balance to the esoteric world I work in. I'm a bit of an "eco-freak," as those who've carefully read my newsletter might have figured out. I'm an avid composter, actually, I'm a Certified Texas Master Composter. My yard and my garden are organic. And I drive a hybrid electric car, a Toyota Prius. I could go on. This passion comes through in my work, where, for example, I always use recycled paper, and I've started asking others to do the same (so far unsuccessfully).

Another personal interest is homeschooling. My wife and I have been homeschooling our oldest daughter for almost a year, and we're loving it. I've started writing articles about the subject, and I teach a programming class for homeschool kids.

Q2. You maintain various lists such as on Test Tools, Testing Contractors and Consultants List etc. at http://testingfaqs.org. What drives you to maintain these lists?

Good question! I've maintained a variety of FAQs ever since I created the comp.software.testing FAQ. Usually I get started with this sort of project because I have a need to reference the information myself. For example, I also maintain a list of software-related organizations and other resources that are local to the North Texas area (http://tejasconsulting.com/resources/ntex.html). I took over the maintenance of the FAQs that are now on testingfaqs.org when Brian Marick said that he didn't want to do it any more. They had such a history, and were such an important part of the testing community, I couldn't stand to see them fall by the wayside.

I really enjoy the research required put these FAQs together, especially the Boneyard page, where I track historical information about companies, products, and services.

It also helps that I occasionally hear from someone who tells me how useful testingfaqs.org has been to them. That sort of feedback really makes my day. I'm embarrassed that I'm woefully behind on posting new entries that people have sent me. It's very important to me that I review the entries, make sure they're in the right category, and remove any superfluous hype. That takes a tremendous amount of time.

Q3. Do you think you have been getting enough support from people in this endeavor? What kind of support would you like to see for these lists from people?

I could certainly keep the site more up to date if I had help. I have had a few people help with things like the CGI scripting, and I've been starting to get a slow trickle of people who are pointing out new freeware tools. I think I could get more help if I could make it easier for people to help. I've thought about using a wiki, though it would be hard to maintain a consistent format on a wiki.

I'm very interested in hearing from people who would like to help, and we can be creative with the way they contribute. Of course, people can also contribute financially by buying a text ad or by sending a donation.

Q4.What is your opinion on the state of software testing today and where do you see it going?

I call myself a "Software Alchemist" because I think we're still in the dark ages of software. I believe it will be many decades before software development becomes a true engineering discipline like it should be. I'm not trying to be self-righteous in this pronouncement; I don't know all the answers either.

The average software tester is still fairly disconnected from the wealth of resources that are available. Most haven't been to a training course or university course that focuses on testing. Many haven't even seen a book about testing - they're still very hard to find in the brick and mortar bookstores. Test managers are in a similar position.

So testing often isn't much better than what common sense tells people to do. While companies are usually surviving in this mode, they could be much more efficient and produce much better products if they tapped into the accumulated knowledge that's out there.

It helps that there's a growing library of free information about testing on the web, but sorting through it to find what's relevant and what's worthwhile advice is difficult, especially for a beginner.

Where is it going? Looks like it's going to India. I'm getting an increasing number of queries from people in India, asking how to get started in testing. (One factor for this may be that my company name happens to have an interesting meaning in several Indian languages.) They seem to have an even more difficult time than average in getting access to resources.

I don't have a business background, so I won't try to guess what's going to happen to the industry.

Q5. What are the things a person who wants to choose testing as a career should do to prepare himself for it?

Do you think this question needs to be added to the testing FAQ?
I've had so many people ask this question that I've been setting their emails aside, planning to write an article on this topic to answer them all at once.

I'm going to write that article soon, but unfortunately a goblin deleted those emails, so I hope those who asked have joined my newsletter mailing list so they'll see the answer. I'm also working on a course that introduces people to testing as a career. The hard part is finding any reason to give them hope of finding an entry-level job.

Q6. What things should a novice tester do to enhance his/her testing skills for a better growth in the career?

Learn a general-purpose programming language if you don't know one already, especially a scripting language like Perl, Python, or Ruby, and also the language that the developers on your project are using. Start doing homebrew automation, automating small everyday tasks, even if test execution itself is still manual. There are things you can automate with only a few minutes or hours of effort, so you don't have to get anyone's approval to do it. People will start to ask, "Wow, how did you do that?"

Read books and articles about testing, network with people who do what you do, and attend conferences and workshops. Also learn consulting skills, even if you're not a consultant (see "The Secrets of Consulting" by Jerry Weinberg). It's easy to learn some nifty new way to do testing. It's much harder to judge whether the technique is really appropriate for your organization, and then to inject a new technique into the organization's culture.

Adopt a mentor, even if your company doesn't have a formal mentoring program. This might be somebody in your company, or someone on the other side of the world whom you've formed a relationship with.

Q7. Where do you see testing tools technology heading? Do you foresee some open-source testing tools which can give COTS tools a run for their money? How do you see your reviews of open testware tools helping the testing community?

The simpler commercial tools are probably feeling the heat from the open source community. For example, you don't see many companies trying to sell a standalone html link checker now, and I believe the availability of free link checking tools has had an influence. The commercial tools usually have several additional features added in order to make them marketable.

At the high end, the more complex types of tools probably aren't causing any commercial vendors to lose sleep yet. The commercial tools tend to have more features and are at least somewhat more reliable. I've looked at a few free load testing tools, for example. A number of expert users are utilizing these tools with good results and saving their companies tens of thousands of dollars in licensing fees. However, the tools are cranky and tend to be difficult to learn, so they're not ready for mainstream usage. Because people who do load testing have to be very experienced testers anyway, I think this is going to be the first type of complex test tool to get a foot hold. They're not just quite there yet.

Then there's homebrew automation, much of which is well below the radar of the commercial market. It's often easier to write a few lines of code to solve a specific automation problem than to find a relevant open source tool and learn how to use it. But open source also plays a big role here. The commercial vendors who have good external hooks in their tools and that don't use proprietary "vendorscript" languages are best-suited to integrate with a test team that's doing a lot of homebrew. But these teams may also be avoiding commercial tools entirely.

What I'm doing with Open Testware Reviews is finding the most promising open source and freeware tools, making people aware of them, and putting them through a grueling review. Rather than ride the wave of open source hype, I want to see what these tools can actually do and learn how difficult it will be for the average toolsmith to use them. I'm also hoping that as I share my findings with the project teams that maintain the tools, my work will help them improve the tools and make them more suitable for mainstream usage.

Q8. What is your advice to people who want to learn automated testing tools on their own? How can they do it without spending a lot of money?

That's a tough one. So many job postings demand years of experience with the big-dog tools. It's very difficult to get access to these tools without spending a lot of money on licenses. I've even seen a firm that joined a tool vendor's partner program and still couldn't get sufficient access to the tools for training purposes for its employees.

The obvious answer is to learn open source tools instead. Even better, use the tools to test another open source program, so you're doing real work rather than just playing. At least for me, I learn better when I have a real-world problem to solve. I use open source tools when I teach courses about test automation, because the participants can go back to their office and immediately start using the tool.

Unfortunately, many employers don't understand how experience with one example of a particular type of tool might enable you to easily learn how to do the same type of testing with a different tool. Maybe when an open source tool becomes as popular as its commercial counterparts, or if we get back to a situation where there are more testing jobs than candidates, open source tool experience will be a valuable addition to a resume. Certainly, there are some niche consulting firms that are having a go at marketing their skills with certain open source test tools.

Q9. You maintain a stress testing tool(stress-driver) on http://sourceforge.net/projects/stress-driver/. Can you please tell us something about this tool, how it can be useful and under what circumstances?

I originally designed this tool to drive stress tests for an operating system when I worked for Convex Computer Corporation. My team ended up using it for a wide variety of stress tests. After it was released under an open source license, I ported it to Linux and Windows and created a project for it on SourceForge. About 200 people have downloaded it so far.

The stress_driver is designed to run multiple invocations of a single test program. There are few restrictions on what the test program itself can do. I've used it to test filesystems, process management, threads, and I it used as one layer of a complex reliability test that simulated interactive user logins. You specify how many copies of the test programs to run, or a range to vary it within. You can let the individual tests run as long as they want to, or the stress_driver can stop them after a randomly determined time interval. There are many other features designed to make stress_driver flexible.

I can't guarantee that the tool will be useful for anything other than operating system testing since I haven't expanded its use beyond that context. But I think it's well-suited for a variety of situations where you have a single program that you want to run multiple copies of, with randomized variations. The test program can of course launch as complex a scenario as you want.

I haven't yet heard from anyone who has downloaded the tool, but the infrastructure is there on SourceForge if anyone wants to discuss any questions they have. Like any tool author, I'd love to hear how people have tried to use it.

Q10. What are top 5 testing books that you would recommend to people?

"Software Testing" by Ron Patton. This is a very well-written and gentle introduction to the subject. Those who already have a good grasp of testing can skip it.
"Lessons Learned in Software Testing" by Cem Kaner, James Bach, and Bret Pettichord. I've recently started recommending this one above the perennial favorite below. Very practical, up-to-date, and useful guidance.
"Testing Computer Software, 2nd edition" by Cem Kaner, Jack Falk, and Hung Nguyen. Generally recognized as the best and best-selling testing book.
"The Secrets of Consulting" by Jerry Weinberg. Testers need to be able to influence people in their organization the same way consultants do.
"Test-Driven Development" by Kent Beck. This is a growing trend among developers that stands to fundamentally affect the way software is developed and what testers' roles are.

Q11. If you were asked to name five persons who have made maximum contribution to testing, what names would you suggest?

This is actually a timely question. I'm writing a series of testing resource guides, and some of the early feedback I got was that I should add a "Who's Who" of testing. This is always a dangerous thing to do, because of the notable people who are invariably left off the list, but I'm going to give it a try.

It would take a lot of research for me to say with any confidence who has had the greatest impact in forming the collective wisdom about testing that has accumulated since the dawn of software. I'll just tell you five of the people who have had the biggest impact on me.

Boris Beizer is the grandfather of software testing. He was one of my first mentors. Many people find him difficult to like, but he nonetheless has had a big impact on the field.
Cem Kaner has done many things, from writing practical and influential books, to volunteering for the cause of software consumer protection, and now he's teaching a new generation of testers at Florida Tech.
Edward Miller facilitated my first face-to-face meetings with dozens of fascinating people in this field. His Quality Week conferences opened my eyes to many new ideas, and taught me the importance of meeting people.
Brian Marick has introduced refreshing new ideas, and has told more than one emperor that he has no clothes. I've often tried to emulate his pragmatic approach to evaluating testing techniques.
Many people still haven't heard of Jerry Weinberg. He prefers to have a profound impact on a few people rather than spreading his influence as thinly and as broadly as possible. He's certainly had a profound effect on me. But his books deserve more attention than they're getting. While he sometimes talks about software testing and practically everything he says can have some bearing on quality, what he really contributes is teaching us that all the issues are really people issues, not technology problems.
There are so many other people who have more recently affected my work and have contributed to the field as a whole. I started to list them, but the list just wouldn't stop growing, so I think I'd be safer to stop here for now. You'll have to wait for the Who's Who to see the rest.

If you were your own interviewer what questions would you like to ask yourself? They are:

1. What have you been up to lately?
I've been doing a lot of writing lately, and I dearly love doing it. But I've also finally gotten serious about convincing people to help me with writing Open Testware Reviews. So to some extent I'll take on the role of editor. I'm excited to be able to represent some different voices in Open Testware Reviews.

I've started to try to put together my first book. I really need to get a book published in order to look like a credible consultant. I have several ideas on the short list; I decided the first one to tackle is to basically take the FAQs I've been involved with over the years and put them in book form. It's frustrating seeing that pretty much all of the testing FAQs that are out there on the web now are incomplete and outdated. I can do much better than that. But I'm realizing that keeping this information up-to-date might be even harder in book form.

2. So how will you publish these lists of data that need to be updated?

I've put a lot thought into that. What I'd like to do is to publish a series of industry reports that are updated yearly. People who buy the reports will get updates for a year, and perhaps a discount on the next year's edition. There does need to be a dynamic element to it. I'm tentatively planning for the first one to include lists of conferences, professional organizations, online forums, and that Who's Who I mentioned earlier.

I'm including value-added information so that the reports aren't just lists of data. This will include cross-references, annotations, and commentary on how best to utilize the resources. The commentary may eventually grow into a book, but for now it looks like the book idea will have to wait.

Right now I'm building an international review panel that can help me make sure that the testing industry all across the world is well-represented in the reports.

3. What message do you want to leave testers with?

Testing is fun! We get to make a mess of the software and then someone else cleans up the mess. We get paid to do devious things on purpose. So have fun with it - now get back to work and break something.

Interview of Elfriede Dustin

2009-07-29T00:35:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=4)

Elfriede Dustin is author of various books on testing - "Automated Software Testing," "Quality Web Systems," and her latest book "Effective Software Testing."

Her website http://www.effectivesoftwaretesting.com was created to create a community of software professionals in the Washington DC area.

If you want to ask her more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com

Q1. Elfriede please tell us something about yourself, your interests and things that you do in your free time.

Originally I am from Germany where I’ve received my business degree and then went on to get a Computer Science degree at KSU. In 1984 at one of my first jobs working at a US Government law center, we received a number of Wang computers and I lead the automation of the claims processes. Ever since then I have been hooked on computers, automation, and testing. Having lived in the suburbs of Washington, DC over a decade, I am now a naturalized American citizen. I enjoy spending any of my free time getting smarter in my chosen field of testing. Additionally, I enjoy spending time with my two daughters Jackie and Erika.

Q2. What prompted you to write the books that you wrote? What kind of response do you think these books have received?

In 1997 at a Rational users conference I presented the topic "Introducing Automated Testing to your Project." After the presentation I received uncountable repetitive inquiries from numerous sources throughout the world (even Belgium) about this same topic. It made me realize that there is a need for a book to put out this information and it prompted me to write the book "Automated Software Testing," together with my co-authors and former co-workers Jeff Rashka and John Paul. The books have been very well received, for example the book "Automated Software Testing" is in its 9th printing and has been translated into German, Russian, Japanese, Chinese, and is also available as a less expensive version for the testing audience in India.

Q3. What were your sources of experience that your book "Effective Software Testing" drew on?

"Effective Software Testing" is based on my many years of testing experience combined with the many years of software development experience of my former co-worker Douglas McDiarmid. The book also draws on the experiences of my former co-workers and co-authors Jeff Rashka, Program Manager, and John Paul, Developer and now CEO, as it also draws on the content of our books "Automated Software Testing" and "Quality Web Systems.”

Q4. Are you working on any other book?

I am in the process of writing a book on Security Testing for QA professionals, published by Symantec Press, written with two Symantec security experts. I feel there is a need for such a book that's specifically geared towards the testing and QA professionals. Not enough is known about how to approach security testing as a QA professional, often it's just a band-aid and too late in the development life-cycle.

My latest article Teamwork Tackles the Quality Goal was written for Software Test and Performance magazine, see http://www.stpmag.com for a free download. A summary of one of my five “Software Test and Performance conference” (see http://www.stpcon.com) presentations “Introducing Automated Software Testing” recently appeared in the SDMagazine newsletter. I am also preparing for a presentation at the ICSTest (see http://www.icstest.com) conference in Duesseldorf, Germany in April, while working full-time as an employee, i.e. an internal SQA consultant to GSS, Symantec. (Please note all opinions expressed here are my own and not those of my employer.)

Q5. Do you think there is a gap between what testing consultants and trainers teach and what the testers and test managers in the trenches face in their daily work? In what ways?

Yes, I definitely see a gap in what testing consultants and trainers teach and what testers and test managers face in the trenches. Theories are nice to have, but some theories are just not realistic and too time consuming or expensive to implement. In the trenches we constantly face ever shrinking budgets and schedules, and go-live dates and production deadlines which can’t be moved, and often the development schedule slips and cuts into the testing time.
In the trenches, so much depends on the developers’ skills. The most efficient and knowledgeable testers cannot succeed if developers implement bad software and/or ineffective software development life-cycles and processes are in place. If testing is the only quality phase implemented as part of a QA realm, it can at most be considered a band-aid often too late in the system development lifecycle to make much of a quality difference. Testing is only one piece of the Quality puzzle.
I don't see this here at Symantec, but I have seen it at other companies where there is still this general confusion by some management about what testers do. The one side of the management camp thinks that “anyone” can do the testing and no special skills are required and the other side of the management camp expects testers to “test quality into a system” and blames the tester if a defect makes it into production. When it comes to testing salaries and job requirements, generally a tester is paid less and technically less is expected of him. Generally there is still a gap between testers’ technical knowledge vs. the developer technical knowledge. This gap also rears itself in still little or no respect for the tester profession and testing still often being seen as the underdog and necessary evil. Based on this general mindset, I wrote my article Teamwork Tackles the Quality Goal on “integrating the “independent” testing team where I make the point that the tester has to be every bit as technical as the developer, working integrated with the development team, while maintaining the independent testing frame of mind. The testing profession has so much to offer, but so little about how it can really effectively be implemented is known or used.

Q6. Who in the testing community has influenced your thinking most?
Actually, my Computer Science professor Dr. Gustafson at Kansas State University has influenced my thinking about testing and quality in general. I took his graduate level course on "Quality software engineering processes" which peaked my interest in Quality and testing and I learned about great quality concepts and influential thinkers such as Tom DeMarco, Capers Jones, Boris Beizer, and Gerald Weinberg.

Q7. According to you what are top three things that one should learn/practice/be good at to be an effective and efficient software tester?

To be an efficient software tester the person really has to enjoy testing, have a knack for it, be analytical, detail oriented, and be versatile, such as a) be technically savvy, b) understand the business, and c) never stop learning and improving on the testing efforts

Q8. What do you think is the current state of automation? Where do you see automation heading towards? Do you think a paradigm shift is required in order to have more automation? Do you think there are other areas where tool vendors need to focus in addition to what they focus on right now?

Current state of automation. During a recent presentation at the Software Test and Performance conference (http://www.stpcon.com) I surveyed and learned that 50 percent of my audience had test automation tools that have become "shelf-ware." Those people are now tasked with resurrecting the automation tools and efforts, which can be an exercise in futility for many reasons. They will have to keep in mind whether the tool is still compatible with the technology being used, i.e. has the technology changed since the tool was purchased. Also, if the automation didn't succeed in the first go around did they note the reason for failure and have those "lessons learned" been evaluated so they wouldn't be repeated on the second automation attempt, etc.
This inofficial survey result is just another example that shows not much progress is being made with the use of vendor provided automated testing tools, an issue we've already discussed in our book "Automated Software Testing" published in 1999, "When inefficient automated testing implementation hasn't shown significant ROI, and budget pressures materialize, planned expenditures for test tool licenses and related tool support may be scratched. Often the tools end up as shelf-ware."

Q9. Where do you see automation heading towards?

Granted vendor provided tools continue to improve, vendors still have and will continue to have a difficult time to keep up with all the latest and greatest technologies and third party controls and widgets, which are usually the source for the biggest compatibility issues and user frustrations. People will continue to struggle with the vendor provided tools, others will turn to in-house developed automation efforts or to automation freeware. Here are some important tips before "jumping on the test automation bandwagon."
• Know the types of tools available along with the problem they are trying to solve. There is no one best tool on the market.
• Safeguard the integrity of the automated test scripts by using a configuration management tool to baseline them.
• Consider that automated testing is software development.
• Educate stakeholders about the test automation efforts, especially developers who need to understand the impact any code changes could have on it.
• Manage the expectations by not overselling test automation's return on investment.
• Do not let automated testing become a "side activity" to work on when time allows; ideally use technical testing talent whose sole focus is on test automation.
• Not all testers need to be "automators," since development expertise is required.
• Decide whether to buy or build a tool or whether to use freeware (or all of the above)

Q10. Do you think a paradigm shift is required in order to have more automation?

Unit test automation is the most effective defect detection/test automation technique. Too many companies still focus all of their test automation efforts on system testing without realizing that unit test automation can have the most ROI. Wishful thinking: Wouldn't it be nice if self-testable development languages existed, such as components are developed automated related unit tests were automatically self-generated.

Q11. In absence of licenses most people can’t learn using tools from most of the tool vendors. But employers generally want tools knowledge. How do you think testers can break this vicious circle and gain automation knowledge?

I think employers need to be educated to understand that someone with a development background can easily pick up any automated testing tool, no matter which one. When I hire automated testers and they can show extensive experience in one tool's scripting language or any scripting language for that matter, chances are they will be able to pick up and learn another tool's scripting language. Testers need to remember that automation is software development, and it is important to have experience in one or more programming or scripting languages.

Interview of Rick Craig

2009-07-29T00:12:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=5)

Rick Craig is a well known name in testing circles because of his book "Systematic Software Testing" as well as the quality of his tutorials and presentation.

In this interview he shares his ideas and advice with the testers.

If you want to ask him more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com.

Q1. Rick, please tell us something about yourself, your background, your interests and things that you like to do in your free time.

I was born and raised in the Midwest and then moved to Annapolis, Maryland to attend the U.S. Naval Academy and then subsequently begin a career in the Marine Corps. I spent 12 years on active duty and then was recalled to active duty during the first Gulf War. I am now a Colonel in the Reserves.

Even though I am an artillery officer, I had the opportunity to attend the Marine Corps Computer Science School first as a student and then as an instructor. I got my first testing job around 1980. In 1982 I got the opportunity to work with a couple of contractors who were probably as close to testing experts as there were in those days. For 3 years they mentored me in testing, writing and presentation skills.

Then about that time Bill Perry invited me to do a key note at the First (Second?) International Testing Conference. Bill was a strong influence on my testing career at that point. I started working with Bill Hetzel and Dave Gelperin in the mid 80’s and they hired me to work for them fulltime in 1989 and I’m still with SQE.

I have a daughter, Crissy who is a Junior at the University of South Florida. I am the co-owner of a popular restaurant called Maddogs and Englishmen, which over the last 15 years has become an icon in Tampa Bay. (www.maddogs.com)

Q2. How did "Systematic Software Testing" happen? How much time did it take?

For years, the students in my seminars urged me to a write a book as a companion volume to my Test Management Course, but the problem was always finding the time…….. That problem was eased by asking my old friend Stefan Jaskiel, a technical writer to help me with the book. Even though almost all of the words in the book are mine, I doubt that I would ever have finished it without Stefan’s help and motivation.

The book took 2 months of dedicated work, plus a lot of work on airplanes over the course of a year. The book is also available in Chinese and will be out in Japanese this month.

Q3. Are there any incidents that might help future authors and that you would like to share?

Here are some tips that helped me write this first book:

1.Just do it!
2.Pick a time every day and work on the book even if it doesn’t seem like you’re making progress.
3.Get good reviewers. Listen to their input but at the end of the day, it’s your choice.
4.If you’re having trouble getting started, pick a coauthor.
5.Don’t try to make your book cover every topic under the sun………..

Q4. Are you working on any other book currently?

No, not right now. My travel and lecturing occupies most of my time.

Q5. What, in your experience, are the issues most testers face?

I think one of the major issues facing testers today (and every day in the past) is determining the value of the testing effort and determining when the task is done. I think that in some environments it is also difficult for testers to gain respect from developers and users.

Q6. Do you think there is a similarity in these issues across geographical locations or are the issues different?

I see very little difference across geographical regions (including other countries)

Q7. What, in your experience, are the areas of improvement for most testers and how can they improve?

Bill Hetzel and I did some research in the early 90’s and the one thing that I learned that stood out for me was that the best teams were the best because they did the basic things well. That is not to say they didn’t employ complex, sophisticated techniques—they did. But it was the basic stuff done well that ensured their success. I think today’s testers need to get better at estimating schedules and risk and learn some of the basic test design techniques that have been available for years.

Q8. Where do you stand between pure exploratory testing and pure scripted testing? Has your understanding changed over time? In what way?

Well, my book is called Systematic Software Testing which would lead you to believe that I am in the scripted camp. And indeed, I am in favor of creating test cases that demonstrate the users can perform those tasks that are necessary for them to be successful in their jobs. It really doesn’t matter how many bugs we find (and hopefully fix)if the users cannot do their jobs. On the other hand, finding and removing bugs can ultimately help the user perform their jobs in a more productive manner. I don’t think anyone questions the effectiveness of techniques like exploratory testing in identifying bugs. I would contend that good testers have done “exploratory-like” testing for years. That is to say that the result of one test leads the tester into the next test….. Experts like James and Cem have given exploratory techniques credibility and made it possible for mainstream testers to benefit from them.

Q9. How effective do you think buddy testing is? How cost-effective do you think it is?

Buddy testing, which is essentially working in teams and having the buddy create test scenarios before the code is written are very effective. Buddy testing, in effect, implements preventive testing at the unit level. Other techniques, like paired programming are also effective and enjoy much wider recognition.

Over the last 20 years or so, I’ve urged my clients and students to use buddy testing. I must admit that most organizations never receive enough buy-in to make it work. Literally, each programmer must learn twice as much code, which of course takes more time upfront. Those organizations that do try it though, are generally successful and in some cases greatly reduce the number of defects passing on to the test group and ultimately the users.

Q10. What is your take on certification of testers?

I think that certification is not necessarily a very good indicator of the skill or ability of a tester to do a good job. By the very nature of the training and certification testing, the focus is on terminology and “book learning”. Still without sounding too much like a consultant……I put myself firmly in the pro-certification camp. How so? If a company chooses to make certification a goal or challenge for their testers, it can be a motivator and offer a sense of accomplishment for the individual testers. Also, it can help a test manager standardize on the terminology used within the company and that alone is very valuable.

Q11. Is there any particular certification that you recommend?

I’m pretty excited about the ISTQB that is just now rolling out in the States. It has some leverage in Europe and is created by a team of experts and practitioners.

Q12. What are your thoughts in a single standard testing terminology? Why do you think one does not exist?

Well, testing was born of the need to validate software systems. The first testers tended to be developers and in some cases, users. Each group and each industry chose the words that made sense to them. When writing Systematic Software Testing, I spent quite a bit of time researching various terms and trying to come up with what was the most common usage. I can’t say that my efforts were always successful. I’m also not very optimistic that a common terminology will be obtained in the near future, although some progress is being made. I think, the internet, conferences, books, training programs, certification, etc. is gradually lessening this problem.

If you were your own interviewer what question(s) would you like to ask yourself? It is:

Q13. What is the one (ok two or three) skill that you look for most in a tester?

Good Communications skills. Creativity. Attention to detail…

Interview of Scott Barber

2009-07-29T00:04:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=6)

Scott Barber is the person behind www.perftestplus.com, co-founder of WOPR (the Workshop On Performance and Reliability) and a prolific writer. In this interview he explains what performance testing is, how one should go about learning it, various resources that can help you and many more things.

If you want to ask him more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com.

1. Scott, please tell us something about yourself.

I was born in Woodbury, NJ USA in 1971 and grew up in a typical American small town. As a child I was very active in school, sports and Boy Scouts. Growing up, the goals I had for myself in adulthood included:

Getting to travel and “see the world”
Get a good college education
Get out of small-town America
Be recognized as a success in my career (as with most young people in the USA my specific career choice changed many times as I grew up)
Be a father who is active in his children’s lives and activities.
Design and build my own home.
I left “small-town America” to attend Virginia Tech on a U.S. Army ROTC Scholarship to pursue a degree in Civil Engineering. After four years, I completed a Bachelor’s of Science in Civil Engineering and started my 4-year active duty commitment in the U.S. Army as a Second Lieutenant (2LT). I later went back to school at night to earn a Master’s Degree in Information Technology. My career has taken a rather wandering, but in retrospect an extremely beneficial, path to finally land in the area of Software Testing.

I spent 4 successful, but difficult years in the U.S. Army culminating as a Company Commander for the 101st Airborne Division, Air Assault Headquarters and Headquarters Company approximately 8 years before I was technically qualified to hold the position. As a junior Captain at the end of my 4 years, I was recruited into a government consulting company to be an Information Engineer on a team that was designing and building a system to replace the computer system I most often complained about while I was on active duty. I held this position for about 1½ years while becoming increasingly more technical before I went to a small private software development company with the intent of becoming an Oracle Database Administrator, but actually ended up spending 6 months as a Configuration Manager. During that time, I learned a ton about Databases, Configuration Management, development and networking. Most importantly, I learned that none of those were what I wanted to do day in and day out. 6 months later, that company was bought by a huge, publicly traded conglomerate that resulted in a significant reduction in pay, leading me to seek a new job. This is when I was recruited into a Performance Test Consulting job by an old college friend where I spent the next 4 years. This time as a consultant is where most of my performance testing experience comes from. After 4 years of consulting, my family and I decided it was time to move. As it turned out, we moved to Florida about the same time the consulting company went through a major re-organization leading me to re-think what I wanted to do next in my career. After a long and difficult 4 months of soul searching, I ended up taking a wonderful position as a Systems Test Engineer and Software Test Manager at my current company AuthenTec, Inc.

As far as my childhood goals are concerned, I have had a chance to travel more than I ever thought I would, but have plenty left to do. I achieved my educational goals, but have a passive interest in earning a Doctorate degree someday. After living in a wide variety of places, I have finally settled in a place that feel like “home” and am very content with my career progression so far.

I have two young boys who are the greatest joy in my life. As I write this, it is with thoughts of my older boy starting school tomorrow morning and my youngest learning to talk. I guess that means that the only childhood goal I have left is to design and build that house. Luckily, I continue adding goals, so I’m far from bored.

2. You specialize in performance testing. What drew you to it? How did you get started, and what led you to start writing about it?

One Sunday evening I got a call from my manager…

“Scott, be at the Marriott Hotel Conference Room at 8am tomorrow. Our CEO has an announcement to make.”

Being a good employee, I did as I was told and went to the meeting. As it turns out, our software development company was “merging” with a huge media company. We were going to become their “development branch”. As a result, we were giving up our bonuses, overtime, and our base pay was being reduced, but we were going to get stock options.

As one might imagine, I was less than enthusiastic about this. I called my friend of 10 years before I even left the parking lot of the hotel. I told him the circumstances and asked him if he’d help me update my resume, to which he responded:

“Dude, that sucks. Send me your resume I’ll take a look, but, we need performance engineers. You’d be a perfect fit.”

I said “Performance engineer? What’s that?!?”

He replied “Don’t worry, you’ll like it.”

He was right, I’ve loved every minute of it! Since then I have dedicated my professional career to this field. It has been a wonderful ride so far.

After a few years, people started approaching me to write about what I do. I never though of myself as much of a writer, so I hesitated and stalled. (Mind you, I was the Civil Engineer in college who avoided writing classes at ALL COSTS. When they finally told me that if I didn’t take a writing course I wouldn’t graduate, I went to the smallest, least reputable, community college I could find, took the course and transferred it!) Eventually, I agreed to create the Performance Engineering section of my company’s Best Practices. Some folks thought it was good and started selling me to other companies to write their procedures. Finally, my boss talked me into trying my hand at writing articles. I really didn’t think anyone would read them, but I wrote a few anyway. As it turns out, people did read them, and liked them.

3. What fascinates you in performance testing?

Most simply, I am fascinated by the diversity of it. As my meandering career path let me to this field, I had a lot of “false starts” in other branches of software development, I learned that regardless of my degree of success in those more narrow branches, I found myself becoming easily bored. As a technical tester, specifically as a performance tester of multi-user distributed systems, I get to apply aspects of all of those branches and more in new and unique ways on every project I do. This diversity from project to project is also what led me to embrace the Context-Driven School of software Testing and ultimately become friends with such software testing thought leaders as Cem Kaner and James Bach.

4. If one wants to gain expertise in performance testing what all should one do? What are the tools, books and sites that can be helpful?

There is very little consolidated information out there about the type of performance testing that most organizations need. There is no doubt that my site (http://www.perfetsplus.com) and articles are one of the most complete references for “Practical Performance Testing”. However, it is by no means complete. Some of the other places I have learned valuable lessons that I have applied to performance testing are:

Everything I can find written by Alberto Savoia
Lessons Learned in Software Testing, John Wiley & Sons, 2002 by Cem Kaner, James Bach and Brett Pettichord.
Testing Computer Software (2nd Ed.), International Thomson Computer Press, 1993 by Cem Kaner, Jack Falk, & Hung Quoc Nguyen.
Select Chapters from The Web Testing Handbook, STQE Publishers, 2001 by Steve Splaine and Stefan P. Jaskiel.
Research published by performance tool vendors.
Research published by testing and research organizations such as Nielsen//NetRatings, Keynote Systems, Inc., and IEEE
Online forums and magazines such as StickyMinds.com, QAForums.com and stpmag.com and performancetester.com.
Performance Engineering of Software Systems, Addison-Wesley, 1990 by Connie U. Smith, Ph.D.
Scaling for E-Business: technologies, models, performance, and capacity planning, Prentice Hall, 2000 by Daniel A. Menascé, Ph.D.
Speed Up Your Site, New Riders, 2003 by Andrew B. King
Anything by Cem Kaner, James Bach, Jerry Weinburg and/or Edward Tufte.
These days, I continue learning through networking with experts, being a technical editor for new performance related publications, beta testing tools, hosting and/or attending workshops and conferences and assisting with research being led by other experts and students in the field.
To really grow your career as a performance tester, I believe there are several things you can do that will gain you largest benefit.

Become a “Mid-Level Everything” – Developer, DBA, Network Admin, Systems Admin, Architect, Business Analyst, etc.
Become an expert in the following – your tools, performance analysis, relevant aspects of operational research and statistics, consolidating, interpreting, sorting, and graphically presenting complex information in an intuitive format.
Become skilled/knowledgeable in the following areas – group facilitation, product management and training.
Find a mentor. There are several exceptional performance testers out there who believe in Jerry Weinburg’s consulting principle “Give your best work away for free.” Befriending one (or more) of them to ask questions of, share ideas and brainstorm with is amazingly useful… to everyone involved.

5. What is your opinion about the state of performance testing tools? Are there some open source or free tools available?

Most tools on the market aren’t actually Performance Testing tools, they are Load Generation tools. No matter what the tool vendors claim, they do not make performance testing simple, they do not analyze results, and they do not pinpoint performance issues. They are virtually all grossly overpriced, and never believe the demo. That said, I have a favorite tool that I support heavily… with a long list of caveats. My favorite is my favorite because it allows me to create realistic loads by writing custom functions and procedures in C – rather than trying to guess how to make the silly GUI buttons duplicate the workload I have constructed.

The best OpenSource tool for general web based load generation is OpenSTA. As long as you are only testing websites, it is competitive with the major (pay) tools in terms of actual ability to generate load and collect data. There are plenty of other free niche tools for specific platforms and protocols, but they are only useful to small groups of people doing very specific types of testing.

6. How do you combine performance testing with load and stress testing

Actually, I see Performance Testing as a superset of testing that includes load and stress testing. I often refer to it as “Performance Related Testing”. In general, load testing is testing your application under expected load conditions to determine the overall performance. Stress testing is testing your application under unexpected or extreme conditions to determine failure modes so you can protect against them. In my presentation “Introduction to Performance Testing; The Who, What, Where, When and Why.” (available here) I describe various components of Performance Testing, and the process in general.

7. What is WOPR? How did it happen? Who all are involved in it and what is your role?

WOPR is the Workshop On Performance and Reliability (http://www.performance-workshop.org). Ross Collard and I co-founded this workshop about a year ago with the purpose of bringing together both Performance Testing Experts and “Enthusiastic Beginners” in a forum where we can share our experiences together in a format that encourages questions, critical thinking and learning. So far, we have had three very successful workshops and plan to continue having them in the spring and fall of every year. You can find out more from the website. The workshop is limited to 20 people per meeting and is by invitation only, but you can apply for an invitation by filling out the form on the website.

8. How is performance testing for consumer applications different from enterprise/server based applications?

There is actually less difference between consumer applications and enterprise/server based applications than there can be between any two individual applications in the same category. The real difference centers around performance expectations. One of the keys to effective performance testing is the ability to collect and classify requirements and then develop tests that verify those requirements.

9. Where do you think performance testing fits in the testing life cycle? Should it be done early or should it be done late? What are the pros and cons and how do you decide?

There are actually at least 3 separate, but highly related, disciplines related to performance that should ideally be included in the Software Development Lifecycle – Software Performance Engineering (SPE), Performance Testing (what we have been talking about) and Capacity Planning. The only book I am aware of that addresses these three discipline as part of an integrated process is Improving .NET Application Performance and Scalability, Microsoft Press, 2004. In this book, Senior Author J.D. Meier does a fantastic job of taking the reader through the entire process. The only drawback is that much of the book is extremely Microsoft/.NET specific.

Performance Test planning should actually begin before the architecture has been determined or any code has been written. It is especially important to finalize performance testing planning before any machines have been configured for testing and the plan should continue to be revised as necessary until the application has reached it’s target load in production several times. Very few organizations do this. Most organizations create a plan shortly before load generation is scheduled to commence and stick to the letter of the plan regardless of the actual results or detected issues. By working this way, much of the value that can be gained through performance testing is lost.

The bottom line is this, Performance Testing late in the process alone, as most organizations do, can be highly effective, but only if all identified issues are relatively easy to resolve. Saving the testing for the end of the development cycle can be disastrous if the identified issues point back to poor architectural or design decisions.

10. You have written a lot of papers on performance testing. Do you plan to write a book on this subject?

I have written quite a number of papers, presentations and tried to tie them all together on my website. When I started doing performance testing, I found very little written on the topic and had to learn through trial and error, seeking out mentors and applying principles from other disciplines. I became frustrated with the lack of valuable information available and found many others who were looking to improve their performance testing. As a result I tried to share what I learned through these mediums so that others will have an easier time getting started than I did.

I have tentative plans for two books. The first is simply a consolidation and expansion of my existing articles. There are two main reasons that I would like to do this. First, I’d like to retrofit the articles to be even more generally applicable, include examples and code samples for various tools since they were initially written on contract for IBM Rational. The second reason is because many readers have requested hard copies, which is unfortunately prohibitively expensive on a case-by-case basis.

While there are some very good books on performance tuning, and other performance related topics and some very good chapters in various testing books about performance testing, there is no single “good” book for practical performance testing lessons. Writing a book is much more difficult than writing a book’s worth of articles, but I keep hoping that I’ll find the time, energy, resources and support to get it done in the not too distant future. While I certainly don’t think I know everything there is to know about performance testing, I do think sharing what I do know would be helpful to many performance testers both now and in the future.

11. What are the things you are currently working on?

Outside of the books and some “clean-up” articles, I have actually moved in a new direction with my new job. I am now in charge of testing fingerprint sensors and their associated software (API, drivers, etc) on both PC and embedded platforms. This is extremely challenging as the testing is a blend of four very different industries (Hardware, Software, Embedded/RealTime and Biometrics) that, to the best of my knowledge, has never been written about and as far as I can tell has rarely, if ever, been tried. The knowledge I have gained as a Performance Tester/Analyst has been extremely helpful in the endeavor, as has my relationships with various industry experts such as Cem Kaner, James Bach, and Alan Jorgensen, not to mention my Embedded Software Development Instructor friends at MIT.

Additionally, I am trying to take my User Community Modeling Language (UCML™) to the next level. It was created specifically to model Performance Workloads and the data associated to those workloads, but I have found it to be personally useful in modeling other application characteristics by simply adding a few symbols and thinking about it from a slightly different perspective. Soon, I hope to publish UCML™ v2.0 to see what other folks think.

Interview of Johanna Rothman

2009-07-28T23:55:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=9)

Johanna Rothman, author of Hiring Technical People and numerous articles, is a well known name in the world of testing. You can visit Johanna's website at www.jrothman.com.

WhatIsTesting interviewed her recently. We hope that this interview will be useful for you.

Q1. Johanna, please tell us something about you, your background and experience. What are your interests and hobbies?

I was born in New Bedford Massachusetts. I graduated from the University of Vermont in 1977 with a BA in English Literature and a BS in Computer Science. I graduated from Boston University in 1985 with a MS in Systems Engineering (Software Engineering concentration). I've been working professionally in the software field since 1976 (I had some part-time developer jobs before I graduated from college). I was a developer (and small project manager) until 1985. I was a tester from 1985-1988. I started managing large projects and people in 1988. When I'm not working, I read voraciously, cook, and drive my kids where they need to go (I think the driving is an American thing.)

Q2. Your email signature reads "Speaker, Author, Consultant -Managing Product Development." What all do you do?

I speak professionally at conferences and privately so that I can spread the word. Seriously, I find that speaking is a large part of my business. I can reach many people when I speak, so I seek out conferences to explain my pragmatic approach to software project management. I write (books, many articles, blogs) so that I can explore the issues around software projects. Both speaking and writing are one-to-many relationships. When I'm lucky, someone writes back and I can develop a relationship with that person. I consult when people want a one-on-one relationship with me. In my consulting business, I perform assessments (not audits) to help people understand what they really do and ways they can perform the work better. I coach people and project managers, including executives. I help projects start on track and stay on track. I facilitate project retrospectives. I do anything that helps people see where they've been, and help facilitate their choice about whether to continue working like that.

Q3. What percentage of your total "management" consulting time is spent in "managing testing"? Do you specialize in testing management?

Every so often, I take on interim management positions. About half the time those positions have been VPs of Engineering, and the other half has been some form of test/quality management. I don't specialize in any one kind of management -- because of my experience, I'm well qualified to perform either type of position.

Q4. What do you have to say about the present state of test management in most organizations?

I speak and write for test managers as much as speak and write for project managers. I find test management to be the hardest position in a typical company. Too often, the position is underfunded, understaffed, and the management position is staffed by people who are ignorant of how testing can work to provide significant value to the organizations. In many ways, it's easy to be a development manager: no one says "let's just cut the development time in half" and really expect the same result. They may cut development time, but they realize they're not going to get the product they want. On the other hand, too many senior management teams think nothing of cutting the testing, and still assuming they will get a quality product for their efforts. When I see this, I say, "Huh??" Test managers need all the help they can get.

Q5. What do you think are the career paths that exist for the testing professionals, vis-is both the management and technical ladders?

I've articulated four areas of technical skill (in my blog and in my book): functional skills, domain expertise, tools/technology, and industry expertise. The more a person gains functional skill and domain expertise, the less limit there is to that person's career. (I believe in dual ladders for technical and management staff.) Too many testers stop growing their technical functional skills. Too many testers are unable or uninterested in growing their domain expertise. The more you learn about your craft and how to apply that craft to the product, the more valuable you will be. In my experience, I found that generalization I had as a tester was great preparation for work as a project manager, program manager, and eventually a people manager.

Q6. What should testing professionals do to prepare themselves for these paths?

Read about testing. Apply different test techniques to the product. Organize the testing. Create a project dashboard. Learn the internals of the product. Write code (if you have the technical skills), so you understand what the developers are up against. Learn how to facilitate project meetings. Negotiate the schedule. Keep doing all of this, and you'll see if you want to be more technical as a tester, more into project management, or more into people management.

Q7. You are writing a book Hiring Technical People . What is this book about? What is the target audience for this book?

The book is about how to hire technical people, realizing that people are much more than their technical skills. Too often, I've heard managers say something like, "We need another developer with Java and Unix experience." Well, that's great, but people vary greatly in their approaches to their work, and just another Java programmer is probably not what the hiring manager wants. I've applied how I perform requirements to analyzing the job and writing a job description. Then I explain about 20 different ways to find candidates, how to sort through the resumes, how to ask questions and what to listen for in the answers, how to make a job offer and start a person out right in the job. I have appendices for specifics about developers, project managers, testers, writers, tech support staff and technical managers. The book is targeted to anyone who in involved in hiring technical people, from hiring managers to people who participate on the interview team.

Q8. When is the book going to be available? Dorset House is publishing the book.

I *think* it might be available in January, but we may slip into February.

Q9. What is the main challenge in hiring good technical people?

Creating auditions is the hardest thing, because each audition is customized for the particular environment.

Q10. What are chief qualities that one should keep in mind for hiring technical people?

I find the best technical people *for the projects I typically work on* are adaptable. They see problems as issues to be solved not insurmountable obstacles. Everyone needs to organize their work in some fashion, although I'm not particular about how they organize their work. I prefer people who have high initiative and take responsibility for their work, so I know that they will take on all the work they can and they know when the work will be complete.

Q11. How does one train people to be good interviewers?

Practice! Seriously, first people have to know how to interview. They have to learn the difference between behavior-description questions and hypothetical questions and closed questions. They have to learn how to reframe the irrelevant questions to learn what they are interested in about the candidate. Then they need to practice. When I teach interviewing, I first explain the different kinds of questions, and then have people develop questions. Then I have them practice in groups of three, so they have an observer to tell them what actually happened in the interview. I find that the feedback step from observer to interviewer is necessary for effective learning.

Q12. Testing sometimes entails endless hours of repetitive regression testing. How do you keep the really good testers motivated in such situations?

First, I try hard to organize the testing so there isn't much repetition. (I typically test from underneath the GUI as much as possible.) That tells the testers that you care about not having them perform work that a computer could do better and faster. When I'm involved with agile projects, I find it easier to test from scenarios and avoid most of the repetition (and on agile projects, it's much easier to develop automation).

Q13. What are the three top skills a testing manager should acquire or hone to become a really successful manager?

First, make sure you like dealing with people. Test managers deal with people all across the organization. Second, develop your influencing and negotiation skills. Third, learn how to organize and report on the testing so your executives understand the value of your testing to the organization. Hmm, that may be more than three

Q14. You have written extensively on project retrospectives. Would you like to share some of the key points?

If you don't perform project retrospectives, you don't increase the value of your work to your organization. If you don't continually increase the value of your work, you are destined to become a commodity, or worse, irrelevant. One more thing: about projects: There is no one right way to organize a project. You choose a lifecycle, the practices (process) for each project based on who all your customers are and what they need.

If you were your own interviewer what questions would you like to ask yourself? They are:

1. What's the most important action to take for project success? Make sure the project manager understands how to manage a project like this one. If the project manager doesn't have experience with this kind of a project, the project manager will need time to make some mistakes.

2. What one technique is most valuable in software projects? Estimating the work to be done. If we under-estimate, we're stuck trying to meet impossible dates. If we over-estimate, our project doesn't appear to have enough value.

Interview of Srinivasan Desikan

2009-07-28T22:50:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=228)

WIT – Srinivasan, please tell us something about yourself, your background, your interests and things that you like to do in your free time?

I come from a middle class family and completed my schooling in a government school. After schooling, I was keen on becoming a Chemistry graduate, and had no idea about other fields such as Computer Science. But with my father’s insistence, and the encouragement from my college principal, I enrolled for Computer Science. This decision is important because it was the first time Computer Science was introduced as a subject at graduation level. I secured the 1st place when I graduated in 1986.

I completed my post-graduation from Pondicherry Engineering College, and was one among the many students who formed the first batch to graduate from this college. I was “pushed” into software testing in 1989, while many of my classmates ventured into supposedly more lucrative roles as programmers and analysts.

While software testing was initially never a key interest for me, I learned soon enough that testing does provide a quick passport to the IT industry. This interest of mine in software testing enabled me to perform successfully in different roles such as Test Engineer, Automation Engineer, Test Manager, Test Architect, Worldwide Director, Head of Testing and now Master Technologist in testing.

Coming to the question on free time – free time is like white elephant for many of the people on earth. A day has only 24 hours and a week has only 7 days. I try to keep a calendar, in which 5 days are completely dedicated for the organization I work for, 1 day is for teaching at a college about what I learnt from industry and 1 day is dedicated exclusively for time with family. But I don’t want to say that I teach only during my free time. This would imply that teaching is a second priority for me, which is really not the case. Rather I would say that I spend my weekends constructively in colleges for two way benefits. Helps me to understand how the book authored by me is used and helps the college in getting industry perspective from my involvement.

WIT - Could you tell us something about your book "Software Testing – “Principles and Practices". What inspired you to write this book? How is this book different or unique?

When I started teaching software testing in colleges in late 1990’s, there were not enough books or college curriculum in this area. In particular, there were no books published from India that was affordable to poor students. This gave my friend and I the motivation to publish a book that is rich in content and affordable for everyone. Regular teaching on this subject left us with plenty of materials and audio & video recordings and we decided to convert them in to a book. Today, this book is one of the best sellers in its category and is recognized by more than 47 universities across the world. More than the copies sold, it is really motivating to get hundreds of emails from students, professors and professionals on how helpful this book has been for them.

WIT - Do you have plans to write another book?
Definitely No - at least not in the near future. The job of a writer is to ensure that readers are benefited by the content of the book. I would like to spend my efforts in popularizing the topic of Software Testing to more and more colleges and organizations with the book that I have already published, rather write another book.

In my efforts in the future, if I find any avenues or topics that are not covered by books in the market, then perhaps I would think of writing another book.

WIT - What in your opinion is considered to be "passionate" about software testing, that you spend your weekends giving guest/free lectures at universities, colleges and companies to promote the subject? Which subjects do you concentrate on? What sort of audience normally attends these? Do you think you have been getting enough support from people in this Endeavour?

Being passionate is always contagious factor. My passion for software testing is also that way and I get my constant energy from my colleagues, friends, students and teaching staff. In my opinion, there is lot to learn in this subject and that is possible only if I concentrate full time on it. This is the reason I teach and give lectures only on software testing. My regular audience includes professionals from the IT industry, students and professors. Last year, I visited 40 colleges, 5-6 IT companies and 5-6 conferences which give an idea on the number of invites I get to speak. Of course, I could always accommodate more if I have more time at hand. Wherever I go, I get more than 100% support from all quarters. This just proves that being passionate is contagious.

WIT - What is your advice to people who want to learn testing on their own? How can they do it without spending a lot of money?

There are many books on software testing that can give you the desired start. If you are a student, pick up a testing project and use the lab resources to learn the topic in a practical way. If you are a professional, then make use of the people around you to enquire about testing related resources that are available in your organization. Focus on practical testing aspects that can be implemented in a small way in your college or organization. Please make use of Web resources, however with a caution. Online information could have been written by people who are less experienced in this field.

Certification is good to test your testing knowledge but always enroll with authentic certification authorities. There are many certification boards which offer affordable certifications by separating the teaching and exam sessions. If you are confident that you can study on your own, and only appear for an exam, then that would make the certification more affordable.

WIT – The co-author of your book is Gopalaswamy Ramesh. Could you tell us something about him?

Both Ramesh and myself come from product organizations and are practitioners of testing. I met him in one of the conference and amazed to find converging goals in us, but with exploding ideas. A combined experience of 45+ years is what we tried to put in our book. Both of us have the passion to teach. He has gone one step ahead, and resigned from his position of Senior Director at Oracle to spend more time in teaching and consulting. Now he teaches project management, software maintenance and software testing at colleges.

WIT - Do you blog?

I don’t write blog but I do read blogs that are dedicated to testing. What is depressing is that many of the blogs project individuals, their strong views or business needs. Testing is a subject that needs a different perspective or treatment. I would like to see neutral blogs in the future that are more genuine in exploring the topic of Software Testing.

WIT - What is your opinion on the state of software testing today and where do you see it going?

Today, software testing is used to judge the quality of a product and is a penultimate phase in product release. Growing popularity of Agile methodologies and techniques such as test driven development, may take testing to be the frontrunner for defining quality requirements of product in future. Changes happen not only due to technology advancements. The business environment and an unpredictable economy also bring it’s own set of changes. Economy of testing, low-cost testing, defect prevention will play more significant roles in the future. However, the biggest challenge in these changes revolves around people. In the past, knowing just the concepts of testing was good enough to be a test engineer. Today, test engineers are expected to be good in both testing and development as automation and understanding of products, require both skills. With all these changes in technology and the economy, test engineers will be required to understand not only roles & skills such as Development & testing, but also business environment and what constitute product quality for customers.

WIT - What are the things a person who wants to choose testing as a career should do to prepare himself for it?

The concepts, principles, and the purpose of testing should be understood first. Testing doesn’t exist alone and its main purpose is to ensure quality of product and profitability of the organization. As I said earlier, change is constant and someone should adapt himself/herself to those changes and keep learning. A career in testing is possible only if test engineers protect their jobs during tough times like the current situation. There is adequate growth in competency areas in testing around the world and the jobs are going to be only for the deserving best. This requires constant learning and foreseeing the changes in the world of software testing. Testing is a career which should be chosen by choice and not by chance.

WIT - What things should a novice tester do to enhance his/her testing skills for a better growth in the career?

There is no place for novices in testing and there is no one called novice tester unless we have novice developers and novice managers. By nature, all people are good testers by their own analytical skills and using those inherent skills in unearthing defects takes them to the first level. Learning concepts from software testing books, through certifications and understanding of how testing is done in other matured industries (such as automobile industry) will create a better understanding on the purpose of testing and can take the testers to next level.

WIT - What are top 5 testing books that you would recommend to people?

Being an author of a best selling book it is unfair to ask me to recommend top 5 testing books. I would leave this question to the readers of my book to judge. But personally, I would recommend our book “Software testing – principles and practices” for readers because of our involvement with both academia and industry and whatever we have written is our practical experience on testing and the intent of the book is to share that experience both good and bad.

WIT - If you were asked to name four persons who have made maximum contribution to testing in India, what names would you suggest?

Prakash Mutalik : For founding one of the very first companies dedicated to software testing in India.
Vikram Shah: For showing what management support can do to testing and for bringing one of the very first system test labs in Bangalore for product testing.
Ramesh Gopalaswamy (my co-author): For promoting software testing in academia across India.
Vipul Kochar: For founding the Indian Testing Board and for making software testing certifications familiar across India.

WIT - What are the most important lessons that you think every tester should learn, either from your experience or own experience?

Lesson 1: Learn to fail personally yet make the product successful. Organizations and individuals that depend on testing to just find and fix problems many times fail due to increased R&D cost and market window time lost. Test engineers who prevent defects proactively don’t get any visibility and hence they are perceived as failed but they are my real heroes for product success.

Lesson 2: Testing on its own produces nothing and always it is part of a bigger purpose to make product successful. Making testing successful alone doesn’t make a product successful.

Lesson 3: The job of testers is to ensure testing is eradicated. If any tester repeats the same activity again and again, then it is an indication of decreasing product quality and lack of growth for that individual.

Disclaimer: All views expressed in this interview are my own and not that of my employer – Srinivasan Desikan

Interview of Michael Bolton

2009-07-28T21:11:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=10)

Michael Bolton is an independent test consultant. He provides testing related training and mentoring, frequently contributes to discussions on various mailing lists, writes a newsletter on testing and plays mandolin. WhatIsTesting interviewed him recently to discuss various testing topics under the sun including testing education, testing certification, CMM and things that he is currently working on. You can visit his site http://www.developsense.com.
If you want to ask him more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com.

1. Michael, please tell us something about your self, the path your career has taken and things that you do in your spare time.

Testing and computer stuff in general are my second career, actually. In high school, I was on a math-and-science track, but in my last couple of years I had a couple of great English teachers and one math teacher that really didn't inspire me at all, so I switched focus. At University of Toronto, I worked on English major and a Theatre (that's the way we spell it here in Canada) minor. After the kind of post-post-secondary jobs that you'd expect from a liberal arts major--washing dishes at a comedy club was one--I became a theatre stage manager, specializing in touring shows for kids. You wouldn't believe how that prepares one for dealing with software projects.

I worked for Quarterdeck from 1990 to 1998, the first four years in Toronto and the latter four as a program manager in Los Angeles. (I was born in New York City to Canadian parents, so I'm a citizen of both countries and relocating was easier for me than it would be for most people.) Quarterdeck made DESQview, a multitasker for DOS, and QEMM, a memory manager. Those products were pretty successful, and since they were technically excellent and interacted with so many other products, Quarterdeck was a fantastically educational place to work. As a friend said, if you didn't learn five really interesting new things, it was a slow day. On top of DESQview and QEMM, in the early 90s we had DESQview/X, the only complete X Window implementation for DOS, where we turned DOS and Windows into X Clients. Most people didn't understand the advantages of that sort of thing then (and the rest have forgotten about them now), and it took a lot of horsepower to run effectively--you needed 64MB of RAM and one of those newfangled Pentium chips. So DESQview/X didn't do well commercially, but it was brilliant technically, and it did lead Quarterdeck into the Web very early on. That was seen to be a Good Thing, because as Windows got memory management, QEMM's market niche eroded. Quarterdeck was the first company to licence Mosaic for commercial release. Our developers rewrote it into a wonderful product, but we released it too late to make a difference in the Netscape/Explorer space. We released a bunch of other Internet tools soon after that, but things were pretty noisy in the marketplace by then. I was managing QEMM and CleanSweep and some other utility software at the time, but Quarterdeck was a small enough company that we could all get exposed to everything--the strengths, which were mostly technical and the weaknesses, which were mostly business problems. Quarterdeck grew far too quickly by buying other companies with Internet-based paper, which led to the company's demise. I point out with some pride that Quarterdeck, as an Internet company, flamed out in 1998, fully three years before it became fashionable to do that. Quarterdeck was always ahead of its time. (laughs)

One of the best things about Quarterdeck, for me, was that it introduced me to Cem Kaner. Program managers at Quarterdeck were essentially the chief quality advocates for each product. I was promoted from testing to be the second program manager there, and everyone after that came out of the testing group too. Cem came to teach his Black Box Testing course after I had moved on from the testing department, but I was lucky enough to be able to sit in on the course. He was able to take all of the things that I had been thinking about in terms of testing and quality and put them into contexts that made sense to me. Cem was--and still is--extraordinarily generous with advice and mentorship, and he also pointed me towards Jerry Weinberg and James Bach, who are also huge influences on my work and my thinking.

So now I'm an independent consultant, and have been for six years. I test software, and I teach other people how to test software, and I write about doing both. Most of my writing eventually makes it to my Web site, www.developsense.com.

That was the career path part. For fun, I play mandolin at Irish sessions around Toronto (and in other cities when I visit), I work on trying to keep up with repairs and reconstruction of our family home, Pandora's House, and I like bicycling--Toronto's a wonderful city for that. I enjoy cooking, and reading about food and cooking--those things have taught me a lot about testing, too. But the most fun and the most educational thing in my life these days is my first daughter Ariel, three weeks old today.

2.You publish a testing newsletter. Can you please tell us something about it, why did you start it and what are your plans regarding its future?

I started writing the newsletter mostly because I wanted to spread around some ideas about testing and about the world in general that I think are important. Although I do get the occasional flash of insight myself, most of the ideas I'd like to spread are from other people--so I'm really trying to be a curator of things that I've found to be interesting or useful. Other people write blogs for the same purpose. I wanted to do something a little more formal than that; a schedule drives me to write in a more regular way than a blog would. It's also a way of marketing my services, and, I hope, adding to my credibility. I'll keep doing it as long as I get positive feedback, and as long as I keep learning things as part of the exercise. If you want to learn about something, in fact, I recommend writing about it.

3. To which school of testing such as Process oriented, exploratory testing etc. do you belong? Or are you unhappy with this classification?

I'm perfectly happy--and proud--to be a member of the Context-Driven School of Software Testing; our manifesto is at www.context-driven-testing.com. The principal tenet is that there are no practises or techniques that can be called "best"; there are things that might be more or less useful or appropriate depending on the context in which they're used. The whole idea is to do things that serve the testing mission and that serve the project. We testers already have plenty to do; we don't have time for practises that /don't/ serve the project.

I'm a reformed process-oholic. Quarterdeck was pretty typical of hot, publicly-held technology companies in the mid-to-late nineties; product release schedules driven by the need for quarterly results, lots of mergers and acquisitions, lots of rapid expansion and catastrophic layoffs. A bunch of us hoped that formal processes would help to foster a bit of coherence. Some processes did help us in the development department to get organized to some degree, which wasn't a bad thing. But general systems thinking will tell you what came next: the processes helped us to deal with an increasingly absurd number of products and projects to the point where, as soon as we were able to cope, the business piled even more projects on.

So we tried to get the business people to follow processes too. We wanted them to follow /our/ processes. Robert Heinlein, I think, said "Never try to teach a pig to sing; it wastes your time and annoys the pig." No, I'm not calling the business people pigs--far from it--but I am saying that some of them had a set of priorities and imperatives that baffled and frustrated a lot of the technologists, and vice versa. To the extent that we could get good work done--and we did do some very good work, under heavy pressure and tight deadlines--it always came down to negotiation and collaboration between people on both sides with skills, understanding, and good will. Rules or guidelines that got in the way would get pitched if we could collaborate to make the project happen. The big lessons there were that people, not processes, saved projects on a regular basis; that processes might be great or terrible, depending on the context in which we chose to use them; that processes and documentation have to serve the needs of project and the business; and that neither people nor processes can save a company that's too deeply in the mire. In order to get something done, you need motivated people. If you need to dig your way out of a hole, your processes won't ever pick up the shovel.

4. What is your take on quality certifications such as ISO and CMM? How do you see them being used or abused, especially in relation to testing?

I mostly see them as being abused, again mostly because some believe that processes, rather than people, are most important to the success of an organization. I recently worked on a project that incorporated software from a company that claimed CMM Level 5. I suspect that at one stage, one project or one department managed to garner that assessment from somebody, and now the whole company claims it. The software was terrible, just rotten. Fixes were shoddy in all kinds of ways; half or more of them failed.

In the early days, the ISO 9000 series of standards were designed for manufacturing contexts--repeated production of large volumes of tangible stuff--and I think they're more appropriate there. Software development is not that orderly; it's a creative, intellectual process, not a deterministic, manufacturing process. As for the CMM, its original context was for defense-related development projects, where human lives are at stake, and where time, money, and manpower are all available in quantity.

Are CMM practises appropriate for every software development context? Certainly not; no way. No single process model can fit every organization, and it's inappropriate to think so, in my view. To the extent that the CMM is ever appropriate, it's tailored for patient, affluent kinds of environments; I think CMM processes would ruin most companies that had to respond quickly to conditions in the commercial software market or the Web world. The trouble is that, to some influential people who are working in a chaotic circumstance, the CMM can have a seductive appeal. Those people will try to impose processes on their organizations--they'll teach many pigs to sing--but what if you need a critter to help you dig up truffles? The singing would be interesting, but it wouldn't help to find more truffles.

Jerry Weinberg defines quality as "value to some person", and he goes on to note that the next question is "who is that person"? The next step from there is to note that decisions about quality are political decisions. The CMM clearly has value to some people. So some questions, if you're interested in the CMM for your organization: Would it bring value to your organization? At what cost? Do you have the political power to see it through?

Let me give a thought experiment: there's a restaurant in Toronto called Susur. The chef, Susur Lee, is a brilliant, creative artist, a world-famous chef. The meals are very expensive, but they're exquisite and innovative. Pretty much the menu changes every night, so the cooks have to be well-trained, knowledgeable, and flexible. There's a much smaller and less expensive restaurant a few doors down--not as exciting, but still really nice and a lot cheaper. They have a standard menu and some specials every night. Just down the street from there, there's a McDonald's. Where are you going to eat? What factors go into your decision? Now ask: of the three, which restaurant uses processes that are closest to the CMM style?

5. What is your opinion on various tester certification schemes and examinations?

Do any of the certification schemes or examinations involve sitting a tester in front of a piece of software, watching while she actively investigates it, and making assessments about her thought processes? Or do the exams ask you to fill in the blanks or check boxes, using terms that appear in someone's Body of Knowledge? That is, do the certifications test your ability to think and to perform well as a tester, or do they merely test your memory? I have a very good memory, so I can assure you that having a good memory can make you appear to be much more clever than you really are.

I've spoken to a bunch of certified testers who tell me things like, "Well, I sat the test, and it was mostly multiple choice; I knew the answers that they wanted, so I put those down even when I didn't agree with them." That kind of conversation led me to describe certification recently: You go to a room in a hotel, you pay a couple hundred dollars to someone you've never met, you stay with them for a couple of hours, you say a bunch of things you don't mean, and at the end they tell you "Thanks--you were great." That a business model with a long pedigree, don't you think?

As a hiring manager, I would want to certify my employees by my own criteria--resumes, letters of reference, interviews, auditions--and not by criteria set down by someone else. On the other hand, if you're applying for a job, some certification or another might be important to the hiring manager. If the manager wanted a certification that I didn't find credible, I don't think I'd want to work for that kind of manager anyway.

6. Can you please suggest how testers should educate themselves in all matters pertaining to testing?

I'd say that the first thing is to educate yourself in all matters that interest you, whether they pertain to testing or not. I don't think that testing is a body of knowledge so much as a point of view, or a way of thinking: that is, thinking critically. I should point out that I mean thinking as a literary critic would, trying to comprehend something, rather than merely dissing it. A tester at some point should learn to think scientifically, too. I have some video of Richard Feynman summing it up: "When we want to discover a new law of nature, we form a hypothesis or a /guess/. Then we make a prediction and perform an experiment. If the hypothesis doesn't fit with the results of the experiment, then the hypothesis is /wrong/--it doesn't matter how beautiful it is, or how elegant, or who wrote it up: if it doesn't fit with observation, then the hypothesis is wrong." If you're tester, people will regularly approach you with a piece of software, and a theory that it works correctly in all circumstances. If you're a tester, a huge part of the job is to come up with experiments that challenge that theory, and to use observations to drive new theories.

I mentioned that I was interested in cooking. Testers who like to cook would do well to read Cook's Illustrated Magazine, which takes a very scientific, testing-oriented approach to cooking. In each article, the authors tell you what they were striving for in terms of taste and texture and presentation, and if they say that they table cream is better than the whipping cream in a given recipe, it's because they tested it.

I think testing and humour something important in common. Jonathan Miller (a British fellow who has been a comedian and doctor and TV presenter and stage director) once gave a lecture that I saw, in which he said that humour allows us to change our perspective on things, "to alter our categories", as he put it. Comedians are iconoclasts, they rely on jarring conventional views of things, and they also listen closely to the way people express themselves. Those are important skills for testers. When we see or hear things in a new light, we laugh, and I think we can learn from that new perspective. A lot of the people I admire are funny. George Carlin, the American standup comedian, is a great example of someone who observes things and relates them to us in new ways. The Europeans would love him; it's a shame he's not better known outside North America. (You can find out more about him athttp://www.georgecarlin.com; I should give a strong caution to those who don't like strong language, though.) Richard Feynman, one of the great physicists of the 20th century, was a very funny guy. On the other hand, Newton was famously humourless, so you never know.

Similarly, testing is all about looking at things from as many perspectives as you can. That's a perfect place for me to pass on some advice from Jerry Weinberg, which is to think of at least three possible explanations for something that you've observed. If you can do that reliably, you're on your way to becoming a first-class tester.

Talk a lot with colleagues and peers. Swap stories, exchange techniques. Sign up for my newsletter (send mail to addme@developsense.com). Go to conferences and local testing groups; start one if you can't find one. For reading--read Kaner, Bach, and especially Jerry Weinberg. Oh, and Feynman.

9. What's your favourite testing joke?

Someone once posted a sign on a laser at CalTech that said, "Please do not look directly into laser with remaining eye."

7. Have you evevr been involved with testing in the XP world? Would you like to share your experiences with us?

I haven't been involved with much testing in the XP world--except that at Quarterdeck, our best developers were doing many XP-like things a full decade before Kent Beck codified XP. I loved that environment. I sometimes see some religiosity in the XP mailing lists, which worries me. But XP's values--collaboration, communication, courage, refactoring, simplicity, rapid and continuous feedback--those things are great. I seek them out, whether they're called XP or not.

8. What are the things you are currently working on?

Heh, heh. Marketing. For me, marketing is much harder than testing. I need people to introduce me to people and organizations that find testing harder than marketing. I think there are more of them than there are of us.
(laughs) That means writing the newsletter, working on the Web site, and so forth. Doing interviews for whatistesting.com. (laughs)

I'm preparing and offering presentations to the testing and development trade shows. I'm working with James Bach on a couple of workshops for Jerry Weinberg's Amplifying Your Effectiveness (AYE) conference (which is a fantastic and unusual conferences; check it out at http://www.ayeconference.com), and I'm trying to contribute to James' Rapid Software Testing course. Continuing to develop really good exercises seems to be the biggest challenge there. I'm also in the process of qualifying to teach Elisabeth Hendrickson's Creative Software Testing course.

I try to write at least a little every day, mostly on testing and quality issues. Since software is so pervasive, I think we need a wider awareness of testing's role, so I'd like to publish articles in places that don't usually talk about such issues.

And then there's that aforementioned daughter.

10. Any big ideas?

Some things seem to be percolating these days. One is that we're trying to automate large parts of our world, particularly the business world, with the goal of making things more efficient (note that efficient is usually just a code word for "cheaper" in modern-day business parlance). We're trying to replace expensive people with cheap machines, or with cheaper people aided by cheap machines. Most customers have needs that are "exceptional" to some degree; customers are individuals, and they have individual needs and preferences and circumstances. However, computers are pretty stupid and have no imagination, and processes are rarely designed to respond well to exceptional circumstances. When we force humans to follow business processes that are being modelled and controlled by poorly-designed, buggy software, will that really serve the customers? Are customers going to enjoy the experience?

The testing world contains an instance of that trend. There's a drive to push automated testing, but I'm not sure that the people behind that push really understand testing OR automation. As I said earlier, testing isn't an activity as much as it's a way of thinking. A human can observe all kinds of problems that a machine cannot. A human can change her behaviour based on observation and judgement; a machine cannot. Machines can do repetitive, deterministic tasks really quickly and accurately; so let's make sure we put automation to those tasks and those tasks only. XP and Agile Processes emphasize automation for unit testing, and that seems to be a good thing, as long as we continue to refine and develop the tests to reflect what the software has to do. But I think that automation shows its limits as it gets closer to modelling the end-user's task--for one thing, people rarely choose to automate the kinds of mistakes that people can quite reasonably be expected to make from time to time. So let's automate the low-level and high-volume stuff, but let's emphasize the value of the human capacity for thinking and inventing and observing and reporting things that matter.

That may be tough, because I frankly don't think businesses are very good at measuring or even considering cost and value if there's not a direct cash figure that can be calculated easily. Outsourcing is a great example: outsourcing is usually less expensive if you look only at the cost of labour. However, many companies today have no assets to speak of, other than intellectual ones--the real assets are in the minds and experience of their staff. Economics suggests that we can't put an accurate value on intellectual capital; there aren't good ways to measure its worth. That's not the same thing as intellectual capital having /no/ value, but without a yardstick, nobody bothers measuring, and the value implicitly gets set to zero. And yet it seems pretty clear to me that software companies that don't place value on knowledge, or that don't treat their people well are vulnerable losing their most valuable assets. Companies that work hard to retain their employees, to keep their minds engaged, to keep their skills sharp, and to keep knowledge in-house will do correspondingly better.

That's why, although I offer testing services, I prefer to provide training or mentorship. I can do expert testing for an organization, but in that case, a lot of what the organization /could/ learn leaves with me. If I train people, the knowledge stays with the organization, and what the people learn from their own subsequent testing stays with them and with the organization. Quarterdeck in its best days used and valued the intellects of its employees, and I can assure you that there's nothing more stimulating than that in the workplace. I hope I can pass on ways for other people to have that experience.

Interview of Brian Marick

2009-07-28T21:02:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=11)

Brian Marick is the author of "The Craft of Software Testing", editor of STQE, and Chair of Agile Alliance board among other things. We interviewed him recently and asked him questions about his current work, book that he plans to write, Agile manifesto, testing patterns and some more.
If you want to ask him more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com.

1. Brian, please tell us about something about yourself, your interests, your hobbies and what do you do in your free time

I graduated from the University of Illinois in 1981, wanting nothing more than to be a programmer. In my first job, my first boss told me "We don't really know anything about you, so we'll put you in testing where at least you can't do any harm." (How wrong he was!) After that, I vacillated between programmer and tester for years, finally settling on testing. I became an independent consultant in 1992. In recent years, I've become highly associated with the Agile software movement (I'm chair of the Agile Alliance board and webmaster). Perhaps that means I'm again vacillating between programmer and tester, but I prefer to think that it's time for the two roles to be much more blended than they have been.

I don't have any free time to speak of. Seriously.

2. You wrote "The craft of software testing" in 1995. This book is different from most of the testing books because it targets developers or testers who do structural testing. What made you write that book?

Why do you think there are so few books on the topic?
At that time, it seemed to me that all testing books left big gaps for the reader to fill. They didn't work through their examples in enough detail. I wanted to give that detail. And at that time there were two contending camps of academic testing: one stressed testing based on faults in code (mutation testing and allied techniques) and the other stressed programs as formal mathematical objects that you could relatively mechanically derive tests from. I favored the first camp, but in a less academic way. The core idea of the book is that testers learn by experiencing bugs and cataloging ways to find them.

I don't particularly care for the book, by the way. The core idea is good, but the presentation is lousy. And its assumptions about the programming process is quaint in this day of test-driven design.

3. Do you have plans to write another book?

I've just started one, to be titled _Driving Projects with Examples: a Handbook for Agile Teams_. Ten years ago, it would have been called something like _System Testing_. That "testing" isn't even in the title indicates that we've come a good way. The drafts of the chapters are available at http://www.exampler.com/book.

4. You are doing PhD. Can you please share the subject with us? How do you think this is going to impact the software development and testing as we understand today.

I gave up. I am likely to contribute to an edited sociology volume this year with a paper called "Rule Following as Manglish Behavior", which casts test-driven design and refactoring as an example of what the sociologist Andy Pickering calls "the dance of agency". There's no way I could get that kind of stuff past an engineering PhD committee, and what would I want a sociology degree for?

5. Would you like to share some of interesting experiences of being a technical editor of STQE magazine?

I'm afraid that there aren't really any. It's a rewarding job, especially when I take a new author and help her get her first article down, but it doesn't make for stories.

6. How did Agile Manifesto happen and Where do you think it stands today? What is your assessment of the impact agile alliance has made?

It happened because there were a bunch of people who had devised similar methodologies that were starkly different from the orthodoxy.

It was Bob Martin, I believe, who decided that some of these people should get together and hash out what their methodologies had in common. What was the core that made Scrum and XP feel so compatible, whereas even a stripped-down Rational Unified Process felt just... different? The Manifesto was our best statement of that core.

As for the impact? I'm writing this on the plane back from the Agile Development Conference. Do you realize what it's like to talk to people who love their job? Who feel like _at_ _last_ they're allowed to produce at their peak? Without the Agile Manifesto, many people's jobs would be worse.

7. How do you think that large organizations can adopt agile methods and what are the hurdles that many of these organizations see? How do you think these organizations can be educated about it?

I'm not the best person to ask this. I usually work bottom-up rather than top-down.

So what are some of the differences between big companies and small companies? Some projects will be larger, and they're more likely to be distributed. There are likely to be more layers of hierarchy. They might have more of an atmosphere of formality. They exist because of economies of scale that capitalize on uniform processes. People can specialize more.

It seems to me that those things make agile processes less appealing, sometimes for good reasons, sometimes for bad.

- Agile's sweet spot is the small collocated team. People have had success with larger teams and distributed teams, but it's probably not the place you want to start. Now, there's no reason not to do small projects agilely, but the desire for uniform processes (perhaps in an Infinitely Tailorable Form) argues against that.

- Agile seeks to put the people who produce the code in direct contact with the people who pay for it. The larger the organization, the more traditional intermediaries who are ready for a political struggle.

- It's so messy and unsophisticated. 3x5 cards everywhere! Crude charts drawn on walls! Stuff piled everywhere, right out in the open, instead of hidden away behind cubicle walls. A scorn of Powerpoint status reports. No project tracking tools that produce arbitrary reports at the click of button.

- Agile projects favor generalists over specialists. That breeds resistance from specialists.

But there are other hurdles that don't (maybe) depend on company size:

- Project managers are often a hurdle. Agile projects represent a shift from directing other people to being directed by them. The best agile project managers serve their team by protecting them from distraction and by removing obstacles from their path. For example, if the team wants an open workspace, it's the project manager's job to make it happen, no matter what the Facilities Department says. Since a lot of project managers are good at directing people, it's not surprising they resist such a big shift.

- The message of an agile project is "You *will* be able to ship an X on the date you choose for the amount you choose to spend. On that day, it will be the most valuable X we could have produced for you. But you can't know what that X is today." That drives some people crazy. They want to know everything: scope, time, and budget. I think they're fooling themselves - but they don't, and they want to keep trying for all three.

Right now, agile is for the visionary business people, ones who believe it's actually possible for software not to disappoint. It will move into the mainstream as the majority talks to enough visionaries to gain confidence that it can work for them (and it will not work for all of them).

8. What is the motive behind http://www.testingcraft.com? Do you think it is receiving enough support from the community. Especially for the techniques and projects? If you would some help, what kind of help would it be?

I don't think I can describe the motivation better than in the motivation page: http://www.testingcraft.com/motivation.html. In short, expertise isn't all that much about grand unifying theories; it's about knowing a wealth of small techniques and having a tacit understanding of when to apply them. The site was a place for testers to share techniques. But they didn't. I don't really know why. It probably has something to do with the way very few testers test in their spare time.

I've moved on from that project. I'm content to let it stagnate.

9. What open source testing tools you are working on? What open source testing tools would you like to see being developed?

Most of my work these days is on Ward Cunningham's Fit
(http://fit.c2.com). But I don't do that much tool work these days.

When I'm not wearing my consulting hat, I'm mostly experimenting with different styles of blending testing and programming. I do that by building code test-first.

In many ways, we live in a golden age of open source test tools. That's driven by the fact that many programmers on agile projects love testing so much that they are building testing tools for us.

What we need most isn't tools. We need testers who are able to use scripting languages. My favorite is Ruby. (http://www.rubycentral.com/book/) Such testers will be able to use tools more effectively, and do great chunks of their job more effectively.

I have some examples of using Ruby: http://www.testing.com/writings/behind-the-screens.pdf
http://www.testing.com/writings/bypassing-the-gui.pdf

11. How do you see the play between testing by developers and testing by testers?

My personal thought process is that once unit testing and its automation becomes a norm, higher level tests would be created by combining these unit tests the same way programs are created from units and testers will have to acquire those skills. But 'black box testing' as we know is not going to go away and testers who have domain knowledge, user interface knowledge etc.
Except for the agile programmers, I haven't seen any particular
evidence that programmers are testing any more than they ever did. And it's now the common wisdom that the "unit testing" that programmers in XP (say) do is not primarily testing. It's instead a way of working through the design and smoothing the act of programming. I expect that to continue and to be extended to system tests (whole product tests, end-to-end tests). Such "business-facing" examples will increasingly play the role that requirements are supposed to. But I don't find it that useful to think of them as tests (though it is awfully nice that they tell programmers when they've changed something and caused unintended consequences - so they are also "regression" tests).

Of course, those examples are made by people, so they'll be incomplete, wrong, etc. There's still a need to critique work after it's been completed. I hope that exploratory testing really comes into its own in agile projects - my experience is that programmers love it, and they appreciate the testing techniques that make exploratory testing more than just poking at keys.

But there's an interesting twist here too, as well. In an agile project, the backlog of stories (descriptions of desirable business value) can grow at any time as people get ideas. I see exploratory testing as not just about finding bugs, but also about getting ideas for new features. Elisabeth Hendrickson and I did a tutorial on exploratory testing this past week. But instead of having the attendees test software, we had them design a game, exploratorily playtest the game, revise the game, and then test it again. If I recall correctly, the attendees said that the first set of tests were mainly about finding problems, but the second set both found bugs and also led to cool new ideas. I want teams doing exploratory testing to have the the attitude that they're in it to find both.

So it seems that testing, in the conventional sense, is getting squeezed out. And I think it is. That's not to say that the people who are testers now have no place on an agile project. Good testers are skilled at translating between the technology domain and the business domain. They're better at coming up with telling examples than programmers or business people. They also seem to be better at noticing when the business experts become "captured" by one of the interest groups that cares about the project. They seem better at maintaining objectivity about the product even when part of the team. All these are traits that testers can bring to the team (partly by spreading them). So the *good* testers will fit well in the agile team, though not doing the same thing they do today.

What they'll do, I think and hope, is collaborate very closely with the business experts and programmers to get a first set of guiding examples running quickly in an iteration. The programmers can then launch into making those examples work while the testers and business experts collaborate to generate more tests (bringing in the programmers as needed). The goal is that the programmers should never lack for an example to guide them and give them confidence. As soon as they finish one, the next one should be ready.

There is, however, a set of testers who will be doing the same thing they do today, I think. They are those testers with a specialized technical skill, especially ones whose tests don't lend themselves to the kind of up-front automation that I've been talking about. I'm thinking of configuration testers, security testers, performance testers, usability testers, and the like. What are often called "ilities". What Cem Kaner calls the "para-functional requirements".

I have a longish list of ideas on this theme here: http://www.testing.com/cgi-bin/blog/2004/05/26#directions-toc
There's an extended example of this kind of interaction in the introduction to my new book. It's at http://www.exampler.com/book/introduction.html.

12. What are your views on the testing patterns? What kind of favor have they found in testers? What future do you see for them?
I did start up an effort to get people together to create testing patterns. It didn't work out well. We didn't get much. I think we just didn't have enough of the right people. The agile movement grew out of the patterns movement in a lot of ways, and the programmers-who-test tend to be pattern-aware, so it's likely we'll get some patterns from that direction. (There are some, for example, in Kent Beck's _Test-driven Development by Example_, and there were mock object patterns at last year's PLoP.) So we shall see what comes up in the years ahead.

Interview of James Bach

2009-07-27T23:52:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=3)

James Bach, co-author of Lessons Learned in Software Testing , is a respected name in the world of testing because of his new and unconventional ideas. You can visit his website at http://www.satisfice.com.

WhatIsTesting interviewed him recently. We hope that this interview will be useful for you.

Q1. James, please tell us something about you, your background and experience. What are your interests other than testing and what do you do with your free time?

I was born in the state of Iowa, USA. I spent my formative years mainly living in Vermont, a quiet, rural place.

I had a terrible time in school, after fifth grade or so. I had read in the U.S. Constitution that "Neither slavery nor involuntary servitude, except as a punishment for crime whereof the party shall have been duly convicted, shall exist within the United States, or any place subject to their jurisdiction." So, I decided that meant I was not required to do homework. My rejection of homework grew into a general rejection of all educational bureacracy and presumptuous authority. It all seemed absurd to me. When I was 16, in 1982, my father (who wrote a book called Jonathan Livingston Seagull about the power of going your own way in life) urged me to drop out of school entirely. So, I did, and continued education in my own fashion.

I got a job at a computer store-- where in six months I believe I did not successfully sell a single thing to anybody-- but soon was hired away to write video games on Apples and Commodore 64's. A few years later I was hired at Apple Computer to be a test manager. I believe I was the youngest manager at Apple, at the time. I had no management or testing background, but that didn't turn out to be much of a problem. Few people there knew much about testing, anyway. I decided to dedicate myself to becoming a testing expert, and spent many hours in a coffee shop across the street from work, reading textbook after textbook, trying to discern the secret of testing.

So, I have an anti-establishment pedigree. I don't feel obligated to remain confined to any one discipline. This has advantages and disadvantages. I was never part of the system that indoctrinates people into the belief that some guru somewhere has found a simple recipe for developing software. I believe there are no such gurus. There are no simple recipes, unless you count the simple recipes that require great skill to fulfill, such as "use your mind."

I have reverence for two things, the scientific method and the happiness of innocent people. In everything else I am an energetic skeptic.

Q2. Your Web Site is named satisfice. What is the significance of this name as well as the words 'Epistemology for the rest of us' used on your site?

To satisfice is to find a solution that is good enough. The word was coined by Herbert Simon, who won a Nobel prize for his investigations into how organizations make real decisions. Complex decisions, such as whether a product is ready to ship, are not made with perfect rationality, but through that foggy process Simon called bounded rationality. My business and all my work is founded on that idea: How can I make an excellent decision, even though I don't have all the facts and even though I'm imperfect as a thinker? That's what interests me and my clients.

Epistemology is the philosophy of how we know what we know. Science is founded on epistemology. "Epistemology for the rest of us" is a play on Apple Computer's old slogan "computers for the rest of us." Like Apple, I am trying to bring something out of the realm of specialists and into common everyday use. That's where I'm going with my philosophy of testing and software development.

Q3. James, this may be a short question with a long answer. Can you describe your association with context driven school of software testing. How did it all start? How did it mature? Please tell us something about the history and the future that you see.

"Context-driven testing" is a term coined over dinner, on November 21, 1999, by Cem Kaner, Brian Marick, and I, while we were attending the eighth meeting of the Los Altos Workshops on Software Testing. Two weeks later, we established the context-driven testing discussion list on Yahoogroups. In 2001, Cem, Bret Pettichord, and I formulated seven principles of context-driven testing and published the first self-described context-driven testing book, Lessons Learned in Software Testing.

Although we didn't coin the term until '99, what we call the context-driven testing community emerged as an identifiable phenomenon with the advent of the LAWST meetings in 1997. That was a watershed event. LAWST stands for the Los Altos Workshops on Software Testing. They are non-commercial, facilitated, weekend-long discussions where small groups of peers come together to share testing experience reports. Cem Kaner and Brian Lawrence founded the first LAWST. There have been almost forty meetings of LAWST or LAWST-like spinoffs with acronyms too numerous to list, here.

In a LAWST meeting, no one who presents an idea is protected from cross-examination and criticism. The critical process of LAWST requires each attendee to learn how to discuss the underlying reasons for their choices, and the context-specific variables involved. People who came to LAWST looking for "best practices" were often disappointed to find more questions than answers, and more controversy than consensus. Many people attended a few meetings and then dropped out, but the ones who continued to meet gained skill in the art of test methodology. So, LAWST and its sibling peer conferences are largely responsible for the rise a new sensibility that says any practice might be a good practice, and any practice might be a bad practice, depending on the context-- a context that is always dominated by the people who populate it.

Context-driven testing, then, is not a technique, it is a professional attitude toward techniques. To "do" context-driven testing is to establish a conscious, explainable, self-critical relationship between your test process and the context in which you are testing. A context-driven tester desires to be fully responsible for his own process of testing.

I see the future of context-driven testing developing slowly as like-minded scientific thinkers come together in small peer conferences and try to write insightful articles about about how to think about testing. I don't think there will ever be more than a small percentage of testers in the world who seriously consider themselves context-driven, just as there are very few ham radio enthusiasts compared to casual cell phone users. In the far future (30 years or so) I believe the values of context-driven testing will supplant the other schools of testing and be recognized as simply "skilled testing." The context-driven label will become unnecessary, because educated testers will consider it obviously true that context dominates practice.

Q4. There is another piece of history that we would like you to shed some light on. Please tell us something about exploratory testing and your association with it. Where do you think exploratory testing is heading towards? How does context driven school embrace exploratory testing?

Exploratory testing is a particular family of test practices defined as "simultaneous learning, test design, and test execution." It can be thought of as a mental martial art. The term was coined (in the software field, anyway) by Cem Kaner in the late eighties, but he didn't write very much about it, and few people used the term. I picked up the term in 1994 after trying and failing to convince people that ad hoc testing is a teachable, sophisticated form of testing. Ad hoc does not mean "sloppy", but most people seem to think it does, so I figured exploratory testing would be a better term for my purposes. Today it's a relatively well known term, and Cem and I are known for our work in this area. Elisabeth Hendrickson has also done some striking work with it, as has James Whittaker. Not many other people are actively working on expanding and describing the practice, which seems to mirror the situation with exploratory work in other fields.

Exploration is by nature difficult to turn into a simple concrete recipe, so many managers are frightened of it. I think exploratory testing is a foundational approach to testing. In other words, I can't accept that a tester is skilled unless he can do exploratory testing well, and it's the first skill that I teach to a new tester who works with me. The basic phenomenon of exploratory thinking has been studied in the contexts of medicine, engineering, philosophy of science, economics, theorem proving, game theory, creative thinking, art, navigation, education, and artificial intelligence. And that list is just from the books I happen to have read.

Exploration is a core part of agile software development. No person can claim to be educated about testing, in my opinion, and yet maintain the belief that exploration is just a luxury to be done only when we finish all our scripted tests. So I find it hard to take seriously those authors and consultants to continue to talk as if exploratory testing is an undisciplined, unteachable, unmanageable process. I hope they will realize that it's not 1985 any more and the craft has moved on.

Q5. What is the place of scripted testing in context driven school? How have your views on scripted testing changed over time?

The context-driven school has absolutely no preferences for any practice over any other practice, except inasmuch as a practice violates one of the seven principles of the school as given at http://www.context-driven-testing.com That means scripted testing has the same place every other family of practices has: a context-driven tester strives to understand the dynamics of the scripted testing and how it might be the solution to some interesting problem.

A context-driven tester applies those forms of testing that fit the situation at hand. As a context-driven tester, for instance, I'm aware of nine specific reasons why I might want to repeat a test, and that implies some form of scripting. Regardless of repetition, there are other possible motivations for scripting. Still, I would say that most testing on a typical complex test project should not be very scripted if your goal is to find important bugs. Exploratory testing tends to find more problems more quickly.

Scripted testing means you create the test before you perform it, and you don't change it while performing it or as a consequence of performing it. This is sort of the opposite of exploratory testing, but I find that most testing includes some elements of scripting and exploration at the same time-- in fact, the realization that most testing combines both styles is the biggest way my thinking about it has changed over the last few years.

Q6. What is your advice to the companies that are stuck with the 'old-world-model' of testing? What should they do that results in better software?
My advice is pay attention. Learn to watch what is really going on. Be self-critical.

So many companies pursue test approaches based on theories of productivity and quality that fall apart the moment they open their eyes and notice that, for instance, 85% of bugs (a popular estimate in my experiences with clients) are found through exploratory means. Or the moment they notice that the test documents they create aren't actually helping to improve testing, and rather draw a lot of time and energy away from the actual business of testing. Many people assume that new testers will benefit from reading old test documentation, but watch new testers and you find this is almost completely untrue. Testers learn through exploration and by being involved in projects with other people.

My other quick advice is to read books by Gerald M. Weinberg. Particularly his Quality Software Management series.

Q7. How did 'lessons learned in software testing' happen?

Cem Kaner and I were at a peer conference dedicated to creating "patterns" of software testing, and we became frustrated with the stilted and unhelpful formats we were being asked to use to communicate wisdom. So, Cem had the idea of writing a simple set of ideas, each with a compelling and memorable label, that would comprise our pattern set. We drafted Bret Pettichord into the project and got to work.

At first, like all writing projects I've been a part of, not much happened. But then Cem negotiated a publishing contract and soon lit a fire under Bret and I to get it done. We got together at my lab and brainstormed a list of 522 testing ideas we wished someone had shared with us when we were new to the craft. We divided these into chapters and assigned them to each other.

Writing the book was a lot like writing a bunch of emails to ourselves. The design concept of the book is to provide a busy tester with a book he can benefit from without having to read the whole thing in a particular order.

Q8. What are the most important lessons that you think every tester should learn, either from other's experience or own experience?

Here it is: Testing Is In Your Head. That is, everything important about testing is a consequence of how you think, what you know, what you see, and what you remember. The human part of testing is fundamental, and cannot be automated out of the process.

Most testing books, I now believe, are written by people who don't know very much about testing. They have experience, and so they think they know what it is. But the problem is that testing is a very subtle craft. Like psychology, it evades simple descriptions and analyses. So, I guess another lesson is Most Testing Books Don't Tell The Truth About Testing.

Therefore, learn these skills:

How to observe what's going on
How to use the scientific method
How to think systemically
How to deal with people
How to be a part of a self-critical community

Q9. Are you working on any other book?

I periodically contribute to Kaner's next edition of Testing Computer Software, and I'm idly working toward a book on how to give yourself an unconventional education.

Q10. What do you mean by "Sleeping Giant" when you refer to Indian Testers in your blog? How do you think they can '"wake up"?

I mean that I see no significant obstacle to India becoming an internationally recognized powerhouse of skilled testing, whenever people in India decide they want to do that.

Before I visited India, I thought there were educational or cultural barriers to becoming skilled testers. I was wrong about that. Please, testers of India, accept my apology.

I don't know what it takes to wake people up. I wonder how it's possible that I wake up each morning. One moment I'm asleep, the next I'm awake. All I can say is that the people I dealt with at the two Bangalore companies I taught started my class with what seemed to be a dull and limited view of testing. But as the class progressed, I noticed a remarkable improvement in their ability to solve the puzzles I set for them. I also noticed, in those particular classes, a lot of energy and enthusiasm to learn about testing. I guess what it may take is for influential people within the Indian technical community to see the possibility of a competitive advantage in becoming viewed as leaders in the worldwide testing community, instead of followers. Right now, I know of no one who speaks of India as leading in any aspect of the intellectual practices of testing. That will continue to be true until at least of few of you start showing up online and participating in the various communities, not as silent lurkers, and not as people pushing outdated ideas from the eighties (there are already lots of people doing that) but as practitioners contributing to the cutting edge of testing thinking.

That cutting edge, I believe, has to do with agility, skill-based process improvement, testing heuristics and psychologically sophisticated theories of testing.
If you were your own interviewer what questions would you like to ask yourself? They are:
1. What is it you spend most of your time doing? I am on the road, most of the time, teaching rapid software testing and risk-based software testing classes and attending peer conferences. Occasionally I work on a court case as an expert witness or trial consultant .
2. What would you most like to be doing? I wish I was doing more testing. I get tired talking about it all the time. The most fun I've had on a project was when I was hired to test Windows XP Embedded as part of the Microsoft antitrust case, in 2002. It's a crying shame that I am not allowed to write about what I discovered during that testing process, or what methods I used. Maybe someday I'll be released from my confidentiality agreement.

Interview of Bj Rollison

2009-07-27T23:30:00.000-07:00

This interview is conducted by WhatIsTesting forum (http://www.whatistesting.com/forum/phpBB3/viewtopic.php?f=35&t=229)

WIT – BJ, please tell us something about yourself, your interests, your hobbies and what do you do in your free time?

BJ - I am a relatively private person regarding my personal life, but regarding my interests or hobbies I think most people know that I am an avid sailor. I taught scuba diving for 20 years, but now only dive occasionally; mostly to change the zincs on my sailboat. I also enjoy surfing and windsurfing, and even raced windsurfers in Japan for 2 years. A few years ago I took up the sport of fly-fishing, and during the winter months I enjoy downhill skiing and the challenge of not breaking my legs.. When I am not in or on the water, I enjoy organic vegetable gardening with my lovely daughter.

WIT - Please tell us something about your new book "How we test software in Microsoft"

BJ - We wrote the book because people outside of Microsoft kept asking - How do we test software at Microsoft?
This is not really an easy question to answer because Microsoft has several divisions with multiple product lines, and each product line pretty much has autonomy over how it ships its product to our customers.

The book attempts to illustrate the career path of SDETs at Microsoft, some professional skills and techniques that we teach to new SDETs at the company to help improve our testing efficiency and effectiveness, and also discusses some of our best practices that many of our groups have found useful. We tried to differ from most other books on testing by sprinkling real-life stories and experiences throughout the book to make it more enjoyable to read. Overall we are pretty pleased with the many positive reviews and feedback we have gotten, and we certainly hope that more testers read the book and find value in our story.

WIT - Do you have plans to write another book?

BJ - I thought about this for a long time. Writing a book is very time consuming. But, when I put together a program at the University of Washington for test automation I realized there are few books on practical application of well-designed, robust test automation. Most books on programming focus on developing applications, and the few books on test automation mostly focus on strategy. So, I am currently writing a series of books on automation practices using the C# programming language because it is easy to learn and very effective for everything from API testing to Win32 clients, web clients, web services, etc.. The goal is to make these books for practitioners filled with practical code samples, and suggestions for designing and developing robust automated test cases. The first book will focus on automated API testing, followed by GUI automation of Win32 clients, managed code clients, etc. I am trying to keep them down to about 150 pages and I hope to complete the first book in the series by the end of the year.

WIT - You were in India last year for the test2008.in conference. Please tell us your experiences about the conference and your interaction with Indian testers?

BJ - The Test2008 conference in India was pretty remarkable, and I was quite impressed with how the organizers planned and conducted workshops throughout the major cities and then gathered folks for the technical presentations at Delhi. I have travelled to India to work at Microsoft’s facilities in Hyderabad several times in the past. I am always impressed by many of the testers I meet in India. Although many Indians initially appear to be rather reserved, I find them to be quite passionate and willing to share their perspectives and viewpoints after building a relationship of mutual respect.

WIT – Please tell us something about your blog - I. M. Testy and your website TestingMentor

BJ - My blog is a venue to share some of my thoughts about testing, and the testing discipline. I thank the many folks who visit my blog, and I am sure they read several other blogs as I do. So, while there are probably a few folks who would love to read about my battles with slugs in my vegetable gardens I think most readers go to my blog to read my thoughts about our profession, and hopefully learn a trick or two they might find useful in their jobs.

The TestingMentor website is simply a way for me to distribute some job aids and tools that I have developed over the years. I am a big proponent of stochastic test data, and when I first began talking about it outside of Microsoft many people asked me if there were any available tools. There were a few tools, but few had the capabilities that are necessary in today’s world. For example, there are a plethora of string generators available, but most of them are only capable of generating ASCII or ANSI characters, and the ones that can generate Unicode characters are incapable of generating strings with surrogate pair characters. So, I decided to design and develop to test data generation tools and a few other utilities and make them freely available to testers to use in either manual or automated testing. Right now I am working on an automation library TestDataGenerator that will wrap many of my existing and new data generators into a single DLL for use in automated test cases.

WIT - From your work, it appears that you are promoting different perspectives of testing. What are these different perspectives?

BJ - I promote software testing as a professional discipline. The practice of software testing is difficult and requires a variety of skills and knowledge. Software testing is not simply about finding bugs, and many businesses realize that bug quantity does not equal product quality. Personally, I view our profession similar to medical doctors. Doctors spend years studying physiology, chemistry, biology, anatomy, and a myriad of other subjects, and they continue to learn new ideas and skills throughout their careers. Likewise I believe that professional software testers should have in-depth knowledge of the systems they are working on. Of course we know the really great doctors are not only experts in the science of their profession, they also have a great bed-side manner with their patients. Similarly I think great testers should also understand customer values but from a different perspective. To paraphrase Bill Gates, ‘testers should understand how our customer’s use the software and understand the bugs the customers run into, but the role of the tester is fundamentally different than that of the project/program manager or UX designer.’ In my opinion, many people in the industry have bought into the notion that behavioural testing of software from a black box only approach in an attempt to emulate the end-user customer needs is good enough. What I have come to realize over the years, and many companies also understand, is that relying primarily on behavioural testing approaches is very costly to the business, and is no longer adequately meeting customer’s quality expectations as the complexity of the systems and expectations of the customers increase. Essentially, I try to promote a pragmatic perspective that encourages testers to constantly improve their technical understanding of the systems they are working on, and to enrich their professional knowledge by continuing to study and learn about the testing discipline.

WIT - What is your perspective on Open Source Testing tools?

BJ - I recently read “Implementing Automated Software Testing” by Elfriede Dustin, Thom Garrett, and Bernie Gauf. I highly recommend this book, and I whole-heartedly agree with their statement “Taking advantage of freely available open source components or free/shareware that provides the required features allows for reuse and ease of additional feature integration, saving time, and money.” Microsoft is also starting to adopt a more open policy and many previously internal tools as well as external contributions have begun to populate thehttp://www.codeplex.com/ open source project community. However, I also realize that many companies have strong policies about using open source tools and since some of my test data generation tools are used by such companies I don’t add them to an open source community, but make them freely available for everyone to use. However, I am also currently collaborating with colleagues on two different projects that are on CodePlex, and I expect we will see a proliferation of testing tools and professionals from across the world collaborating on those tools in the open source communities in the near future.

WIT - What is your opinion about the “Schools of Testing”? To which School of testing such as Process oriented, exploratory testing etc. do you belong? Or are you unhappy with this classification?

BJ - Well I have blogged about this in the past at End the segregation of the four schools of testing and Schools of Testing Revisited

These days, I try not to dwell on such trivialities or meaningless debates that have absolutely no impact on the vast majority of professional testers in the industry or the advancement of our profession.

Although I do suspect Bret Pettichord intended the concept of ‘schools’ to provide insight into different approaches to testing that co-exist within any project (much like successful teams include people with different personality types), in my opinion the whole “schools of testing” notion has become a ridiculous attempt to segregate the discipline by some fictitious philosophical doctrine.

WIT – What do you feel about Software Testing Certifications? How do you see them being used or abused, especially in relation to testing?

BJ - I think many people misunderstand certifications. People can “game the system” for many different types of certifications issued in various professional disciplines, and software testing certifications are no different. But, I think most people are overlooking personal responsibility for one’s own education and personal development. Simply showing up for some class and passing an exam is simply a formality of many certifications. It is how the individual applies themselves to the learning experience, and acquires new skills and knowledge that differentiates a person who is truly committed to a discipline versus a hobbyist or someone who simply wants to collect wallpaper

Currently, I think many companies are using software testing certifications for 2 purposes. First, similar to a college degree is illustrates to the company that you took time and passed the exams to become certified. For some companies a certification is a flag that “puts the foot in the door,” and could be the difference between a resume that is reviewed more carefully versus one that becomes circular file (trash bin) fodder. All companies use various types’ mechanisms to flag resumes, and software certifications are simply one way a company might choose to identify potential candidates. Secondly, the certification process provides a common professional jargon within the workplace. For example, when I speak to someone who is certified about a particular testing technique such as equivalence class partitioning I don’t have to spend the first 30 minutes of the conversation explaining how the principles of mathematical set theory is applied in software testing which makes the overall conversation much more productive.

WIT – Can you tell us something about Microsoft's Engineering Excellence group? What work do you do there?

BJ - The Engineering Excellence group was established in 2003 by Bill Gates with a mission to revitalize our engineering teams through strategic leadership and community building, technical career development, and identification of best practices around the company to improve our internal practices and processes. The team is mostly staffed with senior engineers who have been at the company for several years working on several different projects, and who are recognized as subject matter experts in one or more technical areas by their peer group at Microsoft.

As a Test Architect in the group I am primarily responsible for designing, developing and teaching technical courses for the 6000+ testers at Microsoft. I and others on our team also collaborate with divisional leadership teams, and provide internal consulting on the adaptation of best practices into project cycles. More recently, my role includes building community both inside and outside of Microsoft, consulting with premier customers on our internal practices, and partnering with universities to incorporate adoption of software testing curriculum in computer science programs. Although I don’t directly work on any specific product, I think I have one of the best jobs at Microsoft because I get to meet so many brilliant and talented people from across the company every day, and I get to work on challenging projects that will ultimately impact how we design, build and ship world class software.

WIT - What, in your experience are the areas of improvement for most testers and how can they improve?

BJ - Design! Many testers I’ve met are very good at breaking software or finding bugs in software. But really, finding bugs in software is really not that hard! My 7 year old daughter finds bugs in software…and let me tell you…she lets me know when she does. However, software testing is much more than bug hunting. Software testing is a creative and intellectually challenging discipline that requires a myriad of skills. The value of a software tester is not in his/her knack to find bugs, but in his/her aptitude to understand the systems they are working on and his/her ability to design effective test cases that evaluate tenet and non-tenet attributes of the product to provide important information to the project’s management team. In my experience, many of the test cases I’ve reviewed have no clearly defined purpose, are too restrictive in their execution, or are so vague they cannot be handed off for another party to execute. Test cases are records that may be required for several reasons such legal or governmental requirements. For software that has a protracted shelf-life, software maintenance requires repeatability to prevent regressions in service environments and effective test cases that are reused in the maintenance cycle and the next release can reduce overall production costs. Whatever the reason for test cases, designing effective tests is a critical skill that professional testers should master as they progress in their career. But, one thing is for certain; the less a tester understands the inner workings of the systems they are evaluating, and the external needs or values that are driving the project the less the tester’s ability to design effective tests from a variety of perspectives. Unfortunately, there is no simple solution to learning how to design effective test cases. I personally study various testing methodologies, I study how bugs can manifest themselves in different ways and look for common root causes, and I study various design patterns. I also review a lot of test cases from across the company, provide feedback, and help redesign test cases so they are more comprehensive.

WIT – What big changes do you see in the industry and what are the potential impact on software testers?

BJ - The proliferation of Agile concepts in software development lifecycles shouldn’t be a surprise to anyone who is paying attention in the industry, and many testing teams are feverishly trying to figure out how to wedge their existing testing strategies into an agile lifecycle. If the primary testing strategy has relied on exploratory or behavioural testing to find bugs many teams are discovering that this testing approach is simply a bottleneck in the process. This is especially true on projects with close connections to the customer where the customer is providing constant feedback during the process, and the iterations between releases to the customer is relatively short. The agile environment requires testers to work much more closely with the developers. Rather than an adversarial relationship testers and developers will work in partnership and find way to identify problems sooner in the lifecycle. Testers can partner with developers to strengthen unit tests, participate in code reviews and inspections, and better understand where to focus their ‘testing attention’ based on a more in-depth analysis of the system. There is also a tremendous amount of work throughout the industry focusing on static and dynamic analysis tools. We will see an increase in tools designed to identify defects sooner, and also help developers prevent certain classes of defects. I also suspect we will see greater use of coverage analysis and path analysis tools by testers to identify weak or under-tested areas of the code and design tests to help reduce risk in those areas. I also think we will probably start seeing tools that reveal properties and class details without the source code similar to reflection in managed code that can help testers conceive of new or different tests. Essentially, the testers of the future will require a much more technical skill set and greater understanding of the systems they are working on. Of course, another impetus of change is the downturn in the economy is the fact that many companies are demanding more with less; meaning those companies are demanding testers who are able to engage in upstream defect detection and prevention practices, design and develop effective automation, collect and analyze important information (including customer feedback) and incorporate that information to improve processes and practices, and who are perceived as valuable assets to the company.

My QA Books

2009-07-27T10:21:00.000-07:00

(last updated on July 28, 2009)

1. Effective Software Testing: 50 Specific Ways To Improve Your Testing - Elfriede Dustin - 2002
2. A Practical Guide to Testing Object-Oriented Software - D. McGregor, David A. Sykes - 2001
3. A Practitioner's Guide to Software Test Design
4. A Testers Guide to Dot NET Programming
5. Advanced Software Testing Vol1
6. Advanced Software Testing Vol2
7. Agile Software Development - Quality Assurance
8. Automated Web Testing Toolkit
9. Component-Based Testing - Towards Continuous Integration In Software Engineering
10. Critical Testing Processes
11. Debugging
12. Effective GUI Test Automation - Developing an Automated GUI Testing Tool
13. Effective Software Test Automation - Developing An Automated Testing Tool
14. Exploiting Software - How to Break Code
15. Fatigue Testing And Analysis - Theory And Practice
16. Foundations of Software Testing - ISTQB
17. Getting Started Tutorial Silk Test 2006
18. Google Hacking for Penetration Testers
19. Hack Attacks Testing How To Conduct Your Own Security Audit
20. Hack I.T. Security Through Penetration Testing
21. How We Test Software at Microsoft
22. Integrated Approach to Web Performance Testing - A Practitioner's Guide
23. ISEB Software Testing Presentation
24. ISO 9001-2000 Quality Management System Design
25. Java Testing and Design From Unit Testing to Automated Web Tests
26. JUnit Recipes Practical Methods for Programmer Testing
27. Measuring Information Systems Delivery Quality
28. Methods for Testing and Evaluating Survey Questionnaires
29. Metrics and Models in Software Quality Engineering
30. mySAP Tool Bag for Performance Tuning and Stress Testing
31. Penetration Testing and Network Defense
32. Performance Testing Microsoft .NET Web Applications
33. Practical Guide to Software Quality Management, 2ed
34. Practical Software Testing
35. Pragmatic Unit Testing In C# With Nunit
36. Pragmatic Unit Testing in Java with JUnit
37. Process Quality Assurance For Uml-Based Projects
38. Quality Control - Pareto
39. Quality Of Service For Internet Multimedia
40. Software Engineering Quality Practices
41. Software Process Quality - Management and Control
42. Software Quality Assurance - A Student Introduction
43. Software Test Automation
44. Software Testing - A Craftman's Approach
45. Software Testing - Testing Across the Entire SDLC
46. Software Testing 2nd
47. Software Testing and Analysis - Process, Principles and Techniques
48. Software Testing and Continuous Quality Improvement
49. Software Testing Fundamentals Methods and Metrics
50. Software Testing Techniques, Second Edition
51. Statistical Quality Control
52. Systematic Software Testing
53. Test and Analysis of Web Services
54. Test Driven Development A J2EE Example
55. Test-Driven Development By Example
56. Test-Driven Development in Microsoft NET
57. TestGoal Result-Driven
58. Testing And Quality Assurance For Component-Based Software
59. Testing Applications On The Web
60. Testing Dot NET Application Blocks
61. Testing Embedded Software
62. Testing Extreme Programming
63. Testing of Communicating Systems
64. Testing Quality Assurance and Quantifiable Improvement
65. Testing Web Security - Assessing the Security of Web Sites and Applications
66. The Art of Application Performance Testing
67. The Art of Software Testing, Second Edition
68. The Certified Quality Engineer Handbook , 3rd Ed
69. The Economic Impacts of Inadequate Infrastructure for Software Testing
70. The Testing Network
71. The Web Testing Companion The Insider's Guide to Efficient and Effective Tests
72. UML - Testing Products
73. Unit Test Frameworks
74. Unit Testing in Java, How Tests Drive the Code
75. Web Security Testing Cookbook
76. xUnit Test Patterns Refactoring Test Code

Index

2009-07-25T23:34:00.000-07:00

Agile Development

1. Agile Software Development Video

Agile Testing

1. Agile Testing Video

Techniques

1. Test Standard Format of a Domain Name

Agile Testing Video

2009-07-25T12:20:00.001-07:00

Agile Testing - Sogeti

Agile Testing

Agile Software Development Video

2009-07-25T12:11:00.000-07:00

Agile Software Development

Why does Agile Software Development pay?

Agile Development

On the Art of Agile Development

Transitioning to Agile Methodology in the Enterprise - Part 1

Transitioning to Agile Methodology in the Enterprise - Part 2

Transitioning to Agile Methodology in the Enterprise - Part 3

Transitioning to Agile Methodology in the Enterprise - Part 4

Transitioning to Agile Methodology in the Enterprise - Part 5

Transitioning to Agile Methodology in the Enterprise - Part 6

Agile is not just a method - Willem Van Den Ende

Are you Agile or Are You Fragile?

Agile Development Teams: Scope and Scale with Mike Cohn

The Road from Project Manager to Agile Coach - 1 of 2

The Road from Project Manager to Agile Coach - 2 of 2

Bay XP Meeting Part 1: Agile Estimation, Mike Cohn

Bay XP Meeting Part 2: Agile Estimation, Mike Cohn

Agile Retrospectives: Making Good Teams Great!

Øredev 2008 - Agile - Agile tales of creative customer collaboration

Øredev 2008 - Agile - 'Wouldn't it be cool if...' - Managing a game development team

Øredev 2008 - Agile - Lean for Agile Managers

Øredev 2008 - Agile - Hitch Hikers Guide to Management

Øredev 2008 - Agile - Agile Fine-Tuning

Øredev 2008 - Agile - Handling Agile Processes successfully as a Developer

Øredev 2008 - Agile - Agile Planning Beyond the Next Iteration

Scrum Basics

Scrum et al.

Scrum Tuning: Lessons learned from Scrum implementation

Øredev 2008 - Agile - Scrum @ large, managing 100 people and more

Øredev 2008 - Agile - The Scrum Alliance and The IT&T SIG ? Building a Bridge over the Waterfall

Øredev 2008 - Agile - Applying Lean Thinking to the Software Development Process

Øredev 2008 - Agile - Scrum Shock Therapy

Agile Development with Microsoft Visual Studio

Controlling Complexity in Agile and Traditional Development

Why Agile Teams Fail

Security in Agile Development

Agile Primer

Agile Software

PMBOK vs Agile: Sifting Reality from Myth

Test Standard Format of a Domain Name

2009-07-25T11:54:00.000-07:00

As a tester, we usually meet testing requirements for standard format of any domain name such as email textbox in registration form. There are many cases you have to list out for testing and definitely it takes a lot of time. However, if you know the standard format of a domain name clearly, you can get the work done easier then.

In terms of testing, I think we should focus on what related to our work. To be simple, domain name is the name to identify device such as computer on the net. It contains three parts

1. Top level domain names (TLD):
Every domain name ends with it. TLD includes either one of generic names (three or more characters) or a two character territory code.
Generic TLD are .com, .edu, .travel, etc. (see full list at http://en.wikipedia.org/wiki/Generic_top-level_domain)
Two character territory code are .au, .us, etc. (see full list at http://en.wikipedia.org/wiki/Country_code_top-level_domain)

2. Second level domain names (SLD):
These are names placed directly left to TLD, separated by a dot (mail.yahoo.com - yahoo is the second level domain name). SLD usually has been chosen based on name of the company, product or service.

3. Other level domain names:
Domain name of third level or higher levels are placed directly left to SLD, still separated by a dot. Third level domain names are often used for the specific host such as ftp, mail or identification purpose. There is no limitation for the number levels of domains, e.g. xxx.xxxxx.yyyyyy.123.456.789.yahoo.com).

You can see more detail of a domain name at http://en.wikipedia.org/wiki/Domain_name

Actually, developer tends to write code to validate email using Regular Expression. See below

string pattern=@"(\w[-._\w]*\w@\w[-._\w]*\w\.\w{2,3})"
System.Text.RegularExpressions.Match match =
Regex.Match(txtEmail.Text.Trim(), pattern, RegexOptions.IgnoreCase);

if(match.Success)
MessageBox.Show("Success");
else
MessageBox.Show("Fail");

From the above code, you can see a bug if user enters the valid email such as abc@bug.aero since the Regular Expression validates the TLD as 2 or 3 characters only. In this case, developer should have used this Regular Expression (\w[-._\w]*\w@\w[-._\w]*\w\.\w{2,6})(there is the TLD .museum)

With the general above information, we can create a set of testing cases as the following (suggestion only):

1. Test names out of the list of Generic TLD
2. Test names out of the list of Two Character Territory Code
3. Test names which contain two TLDs or TLD placed at the second level, not at the end.
4. Test names with incorrect format, not separating by a dot
5. Test the number levels of domains
6. Test other level domain names which contain special character such as @, #, etc.