ACATS Test Modification List

ACATS Test Modification List

Version 4.2A
June 28, 2024

What is this?

This is the ACATS modification list for ACATS 4.2. This document includes test corrections, test removals, and new tests for ACATS 4.2. ACATS 4.2 is the current test suite for Ada, covering the third edition of Ada, ISO/IEC 8652:2012 (usually known as "Ada 2012").

ACATS Modification List version 3.1DD for ACATS 3.1 is the current list for that test suite. ACATS 3.1 should be used for testing implementations that support Amendment 1 (ISO/IEC 8652:1995/AMD 1:2007, often known as "Ada 2005"), and will continue to be maintained only as necessary for the foreseeable future.

ACATS Modification List version 2.6K for ACATS 2.6 is the current list for that test suite. ACATS 2.6 should be used for testing implementations of the second edition of Ada, ISO/IEC 8652:1995 (usually known as "Ada 95"); it also will continue to be maintained only as necessary for the foreseeable future.

Changes from the last list:

This is the initial list for ACATS 4.2. It contains two newly issued tests as well as those tests whose modifications were not yet required for conformity assessments as of July 1, 2024. We repeat the description of those test changes here.>

Test CD30005 contained C code that does not have a function prototype. Function prototypes were introduced in ANSI C in 1989, and have recommended since C99 (not using prototypes has been deprecated in standards since then). Function prototypes will be required in future versions of C, some popular compilers will start rejecting code without prototypes in their next releases. Virtually any C compiler that one would encounter today would support prototypes, so it is best that the ACATS use them appropriately. Note that implementers are allowed to modify the C code of the ACATS, so in the unlikely event that someone needed to run the ACATS on a compiler that doesn't support prototypes, the extra declaration can be removed. Note that the Ada test file includes a copy of the C code in its comments, so both files of this test have been modified.

Test CXA4001 has been modified to use the results of the function calls @186 and @201, in order to ensure that the functions need to be called.

New test C3A1005 tests that the freezing changes of AI12-0155-1 were made (allowing declaration of non-primitive operations of a incomplete type whose completion is given in a body). It also tests that calls can be made with values of a tagged incomplete type.

New test CC51011 is a reissue of a previously withdrawn test. AI12-0155-1 eliminated the rule that previously made this test illegal. The test tries instantiating a generic unit which has a formal incomplete type with an incomplete type whose full type is given in a body.

New tests posted before June 30, 2024 will be required of conformity assessments on January 1, 2025. No new tests were posted between July 1, 2023 and March 31, 2024. New tests posted before July 1, 2023 are required for conformity assessments (these are all included in ACATS 4.2). The effective dates of tests are marked in the lists.

Questions about this document? Contact the ACAA Technical Agent at agent@ada-auth.org.

ACATS 4.2 includes optional Test Grading tools

When the ACATS was designed (as the ACVC in the early 1980s), the intention always was that running it would give a simple and clear Pass or Fail result. However, grading of tests (particularly of B and L Tests) is somewhat subjective and very time-consuming. (Test grading typically involved poring over compiler listings of the entire ACATS with a large highlighter.) The subjectivity was acceptable in the days of common formal testing, as the formal testers would enforce a common grading standard (even if no one could describe what that was), but in an age of self-testing, it reduces the value of the ACATS.

As such, we're introducing a long-overdue set of automated grading tools. These tools are intended to:

Greatly reduce the effort to grade an ACATS test run (and, hopefully to reduce the cost of formal testing to make it a more practical option);
Ensure that all users of the ACATS grade tests in the same way, thus making it more valid to compare ACATS results between different implementations and different testers;
Give implementers certainty as to whether a particular test result is passing (whether the location of an error is "close enough" to the canonical position);
Be able to detect process errors (especially errors of omission) – that is, tests improperly run (for instance, files compiled in the wrong order where the ACATS requires an order), or not run at all (if one is checking off lines in a listing, it's hard to notice that some test or portion of a test is missing).

The tools also will optionally give feedback to implementers about quality of error message positioning and of error cascades (does one error generate multiple messages?). (Since "goodness of an implementation" is not something that the ACATS tests for, this is not an official goal for the tools.)

This is an ambitious set of goals. It's unlikely that they'll be completely met by the initial versions of the tools. As such, their use will be completely optional (but encouraged!) for ACATS 4.2. The tools will be updated periodically by the normal process for updating the ACATS – an ACATS Modification List like this one.

Details on the operations of the tools can be found in the ACATS 4.2 documentation (that documentation can be found at http://www.ada-auth.org/acats-files/4.2/docs/UG-6.HTM). As always, if you have questions, contact the ACAA Technical Agent at agent@ada-auth.org.

Notice:

The ACAA has created and maintains the Ada Conformity Assessment Test Suite for the purpose of conformity assessments conducted in accordance with the International Standard ISO/IEC 18009 - Ada: Conformity assessment of a language processor. This test suite should not be used to make claims of conformance unless used in accordance with ISO/IEC 18009 and any applicable ACAA procedures.

List explanation:

Test numbers given are for the basic test. Categories are "Withdrawn" (the test should not be processed for Ada conformity assessments); "Allowed Modification" (either the original test or the modified one should be processed for Ada conformity assessments); "Modified" (the modified test must be processed for Ada conformity assessments); "Pending New" (the test does not need to be processed for Ada conformity assessments, but will be added in the future); "New" (the test is processed for Ada conformity assessments).

ACATS version labels are provided with all new and modified tests. Use this label to retrieve the test from the ACATS VCS. See "How to get new and modified tests".

In citations, line numbers are given relative to the start of a test file in the format of '@<line#,line#,...>'; if a test program comprises more than one file, then the particular file will be indicated by "fn", where "n" is the test-program name's file-designator numeral. E.g., "B38103E : @f1-31, ... " denotes test file b38103e1.ada. In cases where there are two or more rationales for different errors in a test program, these different rationales with their respective line citations are listed on separate lines. References to the Ada standards are given in the format:
<section|chapter>.<clause>[.<subclause>]:<paragraph>
These references are to the consolidated Ada standard as available at http://www.ada-auth.org/standards/rm12_w_tc1/html/RM-TOC.html.

Main list:

===== Core Language =====

Name     Category  Version label  (effective date; old version label)
         Short description.

C3A1005 [Pending New] New VCS Label: A4_2A
                 Required after 2025-01-01
        Test to check that calls can be made with values of a tagged
        incomplete type.

CC51011 [Pending New] New VCS Label: A4_2A
                 Required after 2025-01-01
        Previously issued test using incomplete types with a formal
        incomplete type, now legal.

CD30005 [Allowed Modification] New VCS Label: A4_1GG Org VCS Label: A4_1
                 Required after 2024-10-01
        Added int to the C function declaration @f0-46 and @f1-51. C
        function prototypes are required in recent versions of C.

CXA4001 [Allowed Modification] New VCS Label: A4_1GG Org VCS Label: A4_1
                 Required after 2024-10-01
        Added uses of Bad_Map after the function calls @186 and @201 in
        order to ensure that the calls are not omitted.

        ===== Annex C =====

[None.]

===== Annex D =====

[None.]

===== Annex E =====

[None.]

===== Annex F =====

[None.]

===== Annex G =====

[None.]

===== Annex H =====

[None.]

===== Support =====

VERSION.A [Modified]  VCS Label: A4_2
          The ACATS Version string is now found here. It now contains the
          ACATS Modication List identifier, so it is revised for each
          released ACATS Modification List. Thus, this file is different for
          each AML, and is included in the list of modified tests for that
          reason. The purpose of the file is unchanged for each version.

How to get new and modified tests:

The official home of the ACATS test suite is the ACAA web site. The site can be accessed at www.ada-auth.org.

The complete set of official new and modified tests are available in the Mod_4_2A.Zip and Tar files on the ACAA web site. As always, when a multi-file test is modified, we provide all of the files, including the unmodified files

Individual new and modified tests can be retrieved from the ACATS VCS. Simply navigate to the appropriate directory (folder), and select the test you want. Select the revision you want (usually the most recent) and it will be retrieved for you. Tests are stored in directories that correspond to the test's prefix. For instance, the test CD92001.A will be found in the CD directory. You can also browse the differences between versions of a test. [In case of a conflict, the files in the ACATS VCS should be considered the 'official' versions.]

If you cannot get the files from the official site or you cannot get the files in a format appropriate for your processor, please contact your ACAL.

The baseline ACATS 4.2 is available in various formats on the ACAA web site.

Summary lists:

Withdrawn Tests