How to Get Started with Test Automation

Test process:

Automation must be considered as a project in its own right and, as with manual testing, must follow a testing process with the following main stages: planning, design, execution, observation and anomaly management.

The main purpose of test automation is to reduce test execution time between correction cycles, and to check complex elements that are difficult to perform manually (e.g. code complexity, % CPU used, branch coverage, performance testing, etc.).

It is essential to devote time to planning, identifying test scenarios, requirements and objectives. Then, the design of test cases must be meticulous, taking into account different combinations of data and possible execution paths. After test execution, it is imperative to evaluate the results obtained, to check whether they correspond to expectations and to detect any false-positives and false-negatives. Finally, any anomalies must be corrected to improve the quality of the software under test. So, an automated test must be planned, designed, evaluated after execution, and corrected if it does not match the expected results.

1. False positives
A false positive occurs when a test reports a defect that does not actually exist in the test object.
In other words, a test fails when it should pass.
Unreliable and unstable tests are often prone to false positives.

2. False negatives
A false negative occurs when the test fails to detect a defect actually present in the test object.
This usually reflects a design flaw.
False negatives are more dangerous than false positives, as they can miss critical anomalies.

To remember :

Planning -> Design -> Execution -> Observation -> Correction

• A mature development cycle will reduce “planning and design” time.
• Automated testing reduces execution time”.
• Better quality will quickly reduce “correction” time.

Key testing activities :

Here's a set of activities that can't be covered by automation and must be considered when drawing up a test plan. Note that these are also the main responsibilities of the test team.

Define test objectives:

• Participate in the business requirements review (ADR - Advanced Design Review)
• Generate business requirements list
• Identify test objectives
• Evaluate work plan
• Organize test team

Develop test plan :

• Build test matrix
• Write the test plan
• Define test environments
• Participate in requirements review (SSR - Software Specification Review)
• Generate software requirements list

Develop test cases:

• Write test procedures
• Finalize traceability matrix
• Write unit test proofs
• Write test schedule
• Participate in design review (PDR - Preliminary Design Review)
• Verify installation and/or install environments
• Initiate user guide
• Initiate training guide

Run tests:

• Receive training on the software product
• Finalize test preparation
• Receive unit test proofs
• Participate in Test Readiness Review (TRR)
• Execute integrated tests (IT)

Report results:

• Complete test log
• Produce progress report
• Produce metrics
• Generate test report (TI)
• Perform Module Acceptance (TI)
• Execute system tests (TS)
• Document TS

Manage anomalies :

• Compile anomalies
• Assign severity and priority
• Perform regression tests
• Put into production (deployment)

Other testing activities :

• Acceptance test follow-up
• Health testing
• Post-implementation follow-up
• Automation (generating/coding test scripts)

Benefits of test automation

In addition to reducing test execution time between correction cycles, and checking complex elements that are difficult to perform manually, automation brings other indirect benefits:

Expeditious processing: while testers are checking test cases and updating, the next test can be run in parallel by the tool.

Focus on validating results: testers can spend more time validating test results.

Repetition: basic scenarios are always run in the same way.

Consistency: execution and reporting are constant across different runs.

There are, however, disadvantages that are not insignificant. Test automation often fails because :

• High tool and license costs.
• Lack of return on investment (ROI).
• Programming and configuration time.
• Updating test script programming when the application is constantly evolving (maintenance problems).
• The tool does not come without a reorganization of work (test methodology).
• Only tests are run, not analyzed.
• Results not understood.
• False positives and false negatives.
• Full-time resources are needed for automation.
• You need to set aside time to carry out automation; it's a project in itself.

The tools must enable testers to carry out their work more efficiently, and not hinder their productivity.

Careful selection of tools is essential, and must be governed by rules of use to maximize the chances of success.

A tool does not solve the problem of product quality!

Automation is a process that adds value by improving efficiency and productivity. However, the return on investment (ROI) only really becomes noticeable once you've reached around 25% automation and are reusing test cases. Before that point, the benefits may be less obvious. In other words, it takes a certain level of automation for the benefits to become fully apparent.

Although test automation is the biggest trend in software testing today, focusing your strategy solely on automation will not guarantee a flawless quality assurance process.

By integrating various methods into your testing strategy, you'll be able to cover more ground, including untested source code, and better identify unexpected behaviors in your system or software product.

Exploratory tests should be included in your testing strategy, as they verify the effectiveness of your current tests. They uncover parts of your code or system that affect performance and usability, or cause interruptions to normal operation. They limit the accumulation of bugs or technical debt.

Automated tests should be :

• Easy to run
• Simple to write
• Simple to maintain

Automating gradually pays off in the long term.

The choice is between coding to develop or coding to automate.

If you can't perform manual tests properly, forget about automation.

Which test cases should be automated?

You can't automate every test scenario. For one thing, it's impossible to consider everything; for another, it's simply not necessary. What's more, if you use test automation without thinking, it can lead to higher costs and failures. Careful automation planning is essential for success

For automation tests, the cases must be :

Constantly repeating

• If you test a function only once or twice, it should not be automated.
• Select functions that require continuous verification and testing.
• The test cases selected for automation should form the basis of your regression testing strategy.

Associated with high risk

• Risk analyses can show the most vulnerable and important parts of a system.
• Risks associated with the human factor can be automated.
  These may be routine tasks, with a high volume of data or pages with heavy web traffic, or a payment process.

With a wide variety of data:

• In a type of testing such as regression, testers have to deal with a huge amount of data, as this requires verifying existing functions of a system after code changes. Testing the code manually would take months, so automation is an excellent solution for regression testing.

Efficient:

• Automation testing is expensive, and ROI is an excellent way of assessing your efforts and the values of automation in particular cases. ROI includes measures such as:
  - The cost of software testing tools and new employee training,
  - Overall testing time,
  - The number of bugs and the time required to eliminate them.
• With the ROI calculation, you'll know which testing activities you need to automate to achieve long-term benefits.

Conclusion

Automation in the software world has become a powerful lever for increasing the efficiency, precision and speed of development processes. By automating repetitive and complex tasks, teams can concentrate on higher value-added activities, reducing human error and accelerating delivery cycles. However, to maximize its benefits, automation must be judiciously integrated into an overall test and development strategy, taking account of project specifics and user needs. Used properly, it becomes an essential driver for innovation and software quality.

To find out more :