Testing is the process of evaluating a system or its component(s) with the intent to find whether it satisfies the specified requirements or not. In simple words, testing is executing a system in order to identify any gaps, errors, or missing requirements in contrary to the actual requirements.
According to ANSI/IEEE 1059 standard, Testing can be defined as – A process of analyzing a software item to detect the differences between existing and required conditions (that is defects/errors/bugs) and to evaluate the features of the software item.
Who does Testing?
It depends on the process and the associated stakeholders of the project(s). In the IT industry, large companies have a team with responsibilities to evaluate the developed software in context of the given requirements. Moreover, developers also conduct testing which is called Unit Testing. In most cases, the following professionals are involved in testing a system within their respective capacities −
- Software Tester
- Software Developer
- Project Lead/Manager
- End User
Different companies have different designations for people who test the software on the basis of their experience and knowledge such as Software Tester, Software Quality Assurance Engineer, QA Analyst, etc.
It is not possible to test the software at any time during its cycle. The next two sections state when testing should be started and when to end it during the SDLC.
When to Start Testing?
An early start to testing reduces the cost and time to rework and produce error-free software that is delivered to the client. However in Software Development Life Cycle (SDLC), testing can be started from the Requirements Gathering phase and continued till the deployment of the software.
It also depends on the development model that is being used. For example, in the Waterfall model, formal testing is conducted in the testing phase; but in the incremental model, testing is performed at the end of every increment/iteration and the whole application is tested at the end.
Testing is done in different forms at every phase of SDLC −
- During the requirement gathering phase, the analysis and verification of requirements are also considered as testing.
- Reviewing the design in the design phase with the intent to improve the design is also considered as testing.
- Testing performed by a developer on completion of the code is also categorized as testing.
When to Stop Testing?
It is difficult to determine when to stop testing, as testing is a never-ending process and no one can claim that a software is 100% tested. The following aspects are to be considered for stopping the testing process −
- Testing Deadlines
- Completion of test case execution
- Completion of functional and code coverage to a certain point
- Bug rate falls below a certain level and no high-priority bugs are identified
- Management decision
Verification & Validation
These two terms are very confusing for most people, who use them interchangeably. The following table highlights the differences between verification and validation.
|1||Verification addresses the concern: “Are you building it right?”||Validation addresses the concern: “Are you building the right thing?”|
|2||Ensures that the software system meets all the functionality.||Ensures that the functionalities meet the intended behavior.|
|3||Verification takes place first and includes the checking for documentation, code, etc.||Validation occurs after verification and mainly involves the checking of the overall product.|
|4||Done by developers.||Done by testers.|
|5||It has static activities, as it includes collecting reviews, walkthroughs, and inspections to verify a software.||It has dynamic activities, as it includes executing the software against the requirements.|
|6||It is an objective process and no subjective decision should be needed to verify a software.||It is a subjective process and involves subjective decisions on how well a software works.|
Manual testing includes testing a software manually, i.e., without using any automated tool or any script. In this type, the tester takes over the role of an end-user and tests the software to identify any unexpected behavior or bug. There are different stages for manual testing such as unit testing, integration testing, system testing, and user acceptance testing.
Testers use test plans, test cases, or test scenarios to test a software to ensure the completeness of testing. Manual testing also includes exploratory testing, as testers explore the software to identify errors in it.
Automation testing, which is also known as Test Automation, is when the tester writes scripts and uses another software to test the product. This process involves automation of a manual process. Automation Testing is used to re-run the test scenarios that were performed manually, quickly, and repeatedly.
Apart from regression testing, automation testing is also used to test the application from load, performance, and stress point of view. It increases the test coverage, improves accuracy, and saves time and money in comparison to manual testing.
What to Automate?
It is not possible to automate everything in a software. The areas at which a user can make transactions such as the login form or registration forms, any area where large number of users can access the software simultaneously should be automated.
Furthermore, all GUI items, connections with databases, field validations, etc. can be efficiently tested by automating the manual process.
When to Automate?
Test Automation should be used by considering the following aspects of a software −
- Large and critical projects
- Projects that require testing the same areas frequently
- Requirements not changing frequently
- Accessing the application for load and performance with many virtual users
- Stable software with respect to manual testing
- Availability of time
How to Automate?
Automation is done by using a supportive computer language like VB scripting and an automated software application. There are many tools available that can be used to write automation scripts. Before mentioning the tools, let us identify the process that can be used to automate the testing process −
- Identifying areas within a software for automation
- Selection of appropriate tool for test automation
- Writing test scripts
- Development of test suits
- Execution of scripts
- Create result reports
- Identify any potential bug or performance issues
Software Testing Tools
The following tools can be used for automation testing −
- HP Quick Test Professional
- IBM Rational Functional Tester
- Testing Anywhere
- Visual Studio Test Professional
There are different methods that can be used for software testing. This chapter briefly describes the methods available.
The technique of testing without having any knowledge of the interior workings of the application is called black-box testing. The tester is oblivious to the system architecture and does not have access to the source code. Typically, while performing a black-box test, a tester will interact with the system’s user interface by providing inputs and examining outputs without knowing how and where the inputs are worked upon.
The following table lists the advantages and disadvantages of black-box testing.
|Well suited and efficient for large code segments.||Limited coverage, since only a selected number of test scenarios is actually performed.|
|Code access is not required.||Inefficient testing, due to the fact that the tester only has limited knowledge about an application.|
|Clearly separates user’s perspective from the developer’s perspective through visibly defined roles.||Blind coverage, since the tester cannot target specific code segments or errorprone areas.|
|Large numbers of moderately skilled testers can test the application with no knowledge of implementation, programming language, or operating systems.||The test cases are difficult to design.|
White-box testing is the detailed investigation of internal logic and structure of the code. White-box testing is also called glass testing or open-box testing. In order to perform white-box testing on an application, a tester needs to know the internal workings of the code.
The tester needs to have a look inside the source code and find out which unit/chunk of the code is behaving inappropriately.
The following table lists the advantages and disadvantages of white-box testing.
|As the tester has knowledge of the source code, it becomes very easy to find out which type of data can help in testing the application effectively.||Due to the fact that a skilled tester is needed to perform white-box testing, the costs are increased.|
|It helps in optimizing the code.||Sometimes it is impossible to look into every nook and corner to find out hidden errors that may create problems, as many paths will go untested.|
|Extra lines of code can be removed which can bring in hidden defects.||It is difficult to maintain white-box testing, as it requires specialized tools like code analyzers and debugging tools.|
|Due to the tester’s knowledge about the code, maximum coverage is attained during test scenario writing.|
Grey-box testing is a technique to test the application with having a limited knowledge of the internal workings of an application. In software testing, the phrase the more you know, the better carries a lot of weight while testing an application.
Mastering the domain of a system always gives the tester an edge over someone with limited domain knowledge. Unlike black-box testing, where the tester only tests the application’s user interface; in grey-box testing, the tester has access to design documents and the database. Having this knowledge, a tester can prepare better test data and test scenarios while making a test plan.
|Offers combined benefits of black-box and white-box testing wherever possible.||Since the access to source code is not available, the ability to go over the code and test coverage is limited.|
|Grey box testers don’t rely on the source code; instead they rely on interface definition and functional specifications.||The tests can be redundant if the software designer has already run a test case.|
|Based on the limited information available, a grey-box tester can design excellent test scenarios especially around communication protocols and data type handling.||Testing every possible input stream is unrealistic because it would take an unreasonable amount of time; therefore, many program paths will go untested.|
|The test is done from the point of view of the user and not the designer.|
A Comparison of Testing Methods
The following table lists the points that differentiate black-box testing, grey-box testing, and white-box testing.
|Black-Box Testing||Grey-Box Testing||White-Box Testing|
|The internal workings of an application need not be known.||The tester has limited knowledge of the internal workings of the application.||Tester has full knowledge of the internal workings of the application.|
|Also known as closed-box testing, data-driven testing, or functional testing.||Also known as translucent testing, as the tester has limited knowledge of the insides of the application.||Also known as clear-box testing, structural testing, or code-based testing.|
|Performed by end-users and also by testers and developers.||Performed by end-users and also by testers and developers.||Normally done by testers and developers.|
|Testing is based on external expectations – Internal behavior of the application is unknown.||Testing is done on the basis of high-level database diagrams and data flow diagrams.||Internal workings are fully known and the tester can design test data accordingly.|
|It is exhaustive and the least time-consuming.||Partly time-consuming and exhaustive.||The most exhaustive and time-consuming type of testing.|
|Not suited for algorithm testing.||Not suited for algorithm testing.||Suited for algorithm testing.|
|This can only be done by trial-and-error method.||Data domains and internal boundaries can be tested, if known.||Data domains and internal boundaries can be better tested.|