Monitoring Code Coverage in Quality Assurance Systems

Why Use Code Coverage Statistics in QA Systems?

A developer needs code coverage measurements in the development system to see whether the tests that he or she is developing are adequate.

A quality assurance manager, working in an integration or consolidation system, needs code coverage statistics as well.

For the QA manager, the code coverage statistics offered by the Coverage Analyzer provide a reliable indicator as to whether the testing in the QA system is adequate or not.

Quality assurance, after all, is not just concerned with whether tests run successfully. A quality assurance manager also has to be able to judge whether the test suite adequately tests the software under development. Too often, developers are happy to provide only trivial unit tests and integration tests that are sure to succeed. A QA manager must be able to detect such cases. A QA manager must also have a data basis for requiring more thorough testing from developers.

Depending on the level of QA staffing and the organization of testing in a development organization, a QA manager may have other ways to validate tests in addition to code coverage. The manager may, for example, help developers with test design. In this case, the QA manager is able to judge the quality of the tests directly - the completeness of path replication, the quality of data-flow testing.

But often the ABAP code coverage statistics are the only source of information that a QA manager has on the quality of testing. The code coverage statistics from the QA system tell the QA manager whether the test suite is likely to be doing a good job of testing the software. The coverage standards set in the Coverage Analyzer allow QA and Development to agree on code coverage standards for validating tests. And code coverage results below the standards give the QA manager the leverage to require developers to improve their tests.

Prerequisites

• To use code coverage to validate testing, you will need a separate QA system for doing your testing.

  SAP has always recommended separate development and test systems. But often development and quality assurance work in a single system.

  If you wish to use code coverage in quality assurance, then you will need to provide a separate ABAP System for testing.

  The reason: The coverage analyzer resets code coverage statistics automatically whenever a processing block is changed. You will not be able to collect these statistics in a development system because the programs under test are likely to change frequently. And the code coverage statistics will continually be reset.

• With your developers, agree on standards for code coverage.

  Code coverage standards determine when the Coverage Analyzer marks a program as 'adequately tested with respect to code coverage.' You can set these standards in the Coverage Analyzer.

  Without agreement on code coverage standards, you and your developers will never be able to agree on whether a program has been adequately tested. You will have less leverage for getting improvements in unit, component, and integration testing.

  The code coverage requirements that you set and whether you apply them to procedure, branch, or statement coverage depend on many factors. For example, your standards will differ depending on whether developers are extending existing code or writing new, how much effort you are willing to allocate to re-factoring of old code for testability, and more.

  Though the Coverage Analyzer comes with default code coverage standards, SAP makes no recommendations on good testing standards. For suggested code coverage standards, please see the extensive literature available in the Internet and elsewhere.

• Optionally, you can maintain personnel and organizational data on your developers using standard human resources functionality. Typically, this data is maintained in a central system and exported into the systems that need it.

  The Coverage Analyzer uses this personnel data (in table CRMAHR) to structure the reports it generates in the Global Display. For example, in the package and author views, the Coverage Analyzer structures the data according to your organizational hierarchy.

  You do not need to use this organizational data. But you may find that the Coverage Analyzer reports have added value if you add the data.

Organizing Code Coverage Monitoring in a QA System

The test cycle in a QA test system is usually determined by the regular transport of new code into the system. Perhaps weekly, perhaps more or less frequently, code released from development systems is transported into the system, which is often also the consolidation or integration system.

When new code is imported, the cycle of regression testing begins anew in order to verify that no damage has been done to existing functionality. For completed functions, component and integration testing can start.

The same transport rhythm governs best practices for monitoring code coverage as part of the testing process. The following sequence of activities has proven to be sensible for organizing code coverage monitoring in a QA system:

1. As a one-time activity, check that the Coverage Analyzer is active in your QA system. If not, then activate it.

   You should also check that automatic history recording is active. If need be, then activate it. History recording provides data for the Global Display of the Coverage Analyzer.

2. As a one-time activity, define one or more test groups in your QA System. Register the service users or real users that are used for testing to the groups.

   Test groups allow you to display and manage code coverage results from testing in isolation. The statistics are not contaminated by the activities of other users in the QA system.

   Keep the number of test groups to a minimum, in any case below 10.

   If you use separate users for testing different components, then you can create separate test groups for each component.

   If the same users do testing for all development projects, then create a single test group. You can filter the results within the test group by package or program so that you can get code coverage statistics for the different projects.

3. Decide on a process for collecting and analyzing code coverage statistics from your testing:

   If your test cycle is triggered by imports from the development systems into the QA system, then here is a suggested best practice for monitoring code coverage:

   1. Reset the code coverage for your test groups before the new test cycle begins.

      The Coverage Analyzer will already have reset the current statistics for processing blocks that were changed. You should reset the rest of the statistics for the code under test.

      By resetting the code coverage statistics at the start of each cycle, you can get a clear picture of trends in code coverage. Have developers improved their tests in areas in which you reported inadequate code coverage? Has the arrival of new untested code diluted the code coverage in existing applications?

   2. Start recording a new history version.

      Starting a new history with each new test cycle will make it easier for you to use the global display to analyze and report on trends in code coverage.

   3. After the test cycle has finished, use the Global Display to see the code coverage achieved by the tests.

      You can download selected records from the Global Display to a spreadsheet for reporting and analyzing trends in code coverage.

      Where code coverage is inadequate, you can branch to the Detail Display to look for gaps in code coverage that could be closed with improved tests.

   4. Publish the code coverage results together with your gap analyses so that the developers can respond with improved tests.