ACATS Test Modification List

ACATS Test Modification List

Version 4.1B
November 18, 2016

What is this?

This is the ACATS modification list for ACATS 4.1. This document includes test corrections, test removals, and new tests for ACATS 4.1. ACATS 4.1 is the current test suite for Ada, covering the third edition of Ada, ISO/IEC 8652:2012 (usually known as "Ada 2012").

ACATS Modification List version 3.1N for ACATS 3.1 is the current list for that test suite. ACATS 3.1 should be used for testing implementations that support Amendment 1 (ISO/IEC 8652:1995/AMD 1:2007, often known as "Ada 2005"), and will continue to be maintained as necessary for the foreseeable future.

ACATS Modification List version 2.6D for ACATS 2.6 is the current list for that test suite. ACATS 2.6 should be used for testing implementations of the second edition of Ada, ISO/IEC 8652:1995 (usually known as "Ada 95"); it also will continue to be maintained as necessary for the foreseeable future.

Changes from the last list:

A number of tests changed status on July 1st and October 1st, this list reflects those changes.

New test B611014 checks that prefixes of Old attribute references that are potentially unevaluated must statically denote an object. New test B611015 checks that prefixes of Old attributes cannot have a limited type. New test B611016 checks that Pre'Class and Post'Class cannot be given on entries and protected operations even if the type is tagged. New test B611017 checks that Pre'Class cannot be given on an overriding subprogram if none of the original subprogram give Pre'Class. New test B611018 checks 6.1.1(10-16/3). New test B611019 checks 6.1.1(17/3).

New tests posted between October 1, 2016 and December 31, 2016 will become required for conformity assessments on July 1, 2017. New tests posted between July 1, 2016 and September 30, 2016 will become required for conformity assessments on April 1, 2017. New tests posted between April 1, 2016 and June 30, 2016 will become required for conformity assessments on January 1, 2017. New tests posted before April 1, 2016 are required for conformity assessments. The effective dates of tests are marked in the lists.

Questions about this document? Contact the ACAA Technical Agent at agent@ada-auth.org.

ACATS 4.1 includes optional Test Grading tools

ACATS 4.1 introduces new test grading tools.

When the ACATS was designed (as the ACVC in the early 1980s), the intention always was that running it would give a simple and clear Pass or Fail result. However, grading of tests (particularly of B and L Tests) is somewhat subjective and very time-consuming. (Test grading typically involved poring over compiler listings of the entire ACATS with a large highlighter.) The subjectivity was acceptable in the days of common formal testing, as the formal testers would enforce a common grading standard (even if no one could describe what that was), but in an age of self-testing, it reduces the value of the ACATS.

As such, we're introducing a long-overdue set of automated grading tools. These tools are intended to:

Greatly reduce the effort to grade an ACATS test run (and, hopefully to reduce the cost of formal testing to make it a more practical option);
Ensure that all users of the ACATS grade tests in the same way, thus making it more valid to compare ACATS results between different implementations and different testers;
Give implementers certainty as to whether a particular test result is passing (whether the location of an error is "close enough" to the canonical position);
Be able to detect process errors (especially errors of omission) – that is, tests improperly run (for instance, files compiled in the wrong order where the ACATS requires an order), or not run at all (if one is checking off lines in a listing, it's hard to notice that some test or portion of a test is missing).

The tools also will optionally give feedback to implementers about quality of error message positioning and of error cascades (does one error generate multiple messages?). (Since "goodness of an implementation" is not something that the ACATS tests for, this is not an official goal for the tools.)

This is an ambitious set of goals. It's unlikely that they'll be completely met by the initial versions of the tools. As such, their use will be completely optional (but encouraged!) for ACATS 4.1. The tools will be updated periodically by the normal process for updating the ACATS – an ACATS Modification List like this one.

Details on the operations of the tools can be found in the ACATS 4.1 documentation (that documentation can be found at http://www.ada-auth.org/acats-files/4.1/docs/UG-6.HTM). As always, if you have questions, contact the ACAA Technical Agent at agent@ada-auth.org.

Notice:

The ACAA has created and maintains the Ada Conformity Assessment Test Suite for the purpose of conformity assessments conducted in accordance with the International Standard ISO/IEC 18009 - Ada: Conformity assessment of a language processor. This test suite should not be used to make claims of conformance unless used in accordance with ISO/IEC 18009 and any applicable ACAA procedures.

List explanation:

Test numbers given are for the basic test. Categories are "Withdrawn" (the test should not be processed for Ada conformity assessments); "Allowed Modification" (either the original test or the modified one should be processed for Ada conformity assessments); "Modified" (the modified test must be processed for Ada conformity assessments); "Pending New" (the test does not need to be processed for Ada conformity assessments, but will be added in the future); "New" (the test is processed for Ada conformity assessments).

ACATS version labels are provided with all new and modified tests. Use this label to retrieve the test from the ACATS VCS. See "How to get new and modified tests".

In citations, line numbers are given relative to the start of a test file in the format of '@<line#,line#,...>'; if a test program comprises more than one file, then the particular file will be indicated by "fn", where "n" is the test-program name's file-designator numeral. E.g., "B38103E : @f1-31, ... " denotes test file b38103e1.ada. In cases where there are two or more rationales for different errors in a test program, these different rationales with their respective line citations are listed on separate lines. References to the Ada standards are given in the format:
<section|chapter>.<clause>[.<subclause>]:<paragraph>
These references are to the Ada standard as available at http://www.adaic.org/resources/add_content/standards/12rm/html/RM-TOC.html.

Main list:

===== Core Language =====

Name     Category  Version label  (effective date; old version label)
         Short description.

B38003D [Modified] VCS Label: A4_1A
        Corrected error tags @70,76,82,88 so "optional" and "error" are
        on the same line, as required by the test standards.

C3A0030 [New] VCS Label: A4_1A
        Check that an access discriminant is only null excluding if it
        includes a null exclusion.

B3A1001 [Modified] VCS Label: A4_1A
        Split Pack6 so possible error sets don't intersect; fixed
        overlong lines.

B3A1002 [Modified] VCS Label: A4_1A
        Split Pack6 so possible error sets don't intersect; fixed
        overlong lines.

B455002 [Modified] VCS Label: A4_1A
        Corrected casing of error tags and fixed overlong lines.

C480001 [New] VCS Label: A4_1A
        Check that an initialized allocator for a class-wide type does
        an accessibility check on the tag of the initializing object.

C52101A [Modified] VCS Label: A4_1A
        Corrected comment @49 not to have the form of an error tag.

B611001 [New] VCS Label: A4_1A
        Check that Pre and Post are only allowed on
        subprograms and entries (and not on subprogram instances).        

B611002 [New] VCS Label: A4_1A
        Check that Pre and Post are not allowed on subprograms
        acting as completions.

B611003 [New] VCS Label: A4_1A
        Check that Pre'Class and Post'Class are only allowed on primitive
        subprograms of tagged types. (Normal tagged types and interfaces.)

B611004 [New] VCS Label: A4_1A
        Check that Pre'Class and Post'Class are not allowed on
        subprograms acting as completions.

B611005 [New] VCS Label: A4_1A
        Check that Pre and Post are not allowed on abstract
        subprograms and null procedures.

B611006 [New] VCS Label: A4_1A
        Check that the restrictions on Pre'Class and
        Post'Class expressions added by AI12-0113-1 are enforced.

B611007 [New] VCS Label: A4_1A
        Check that Pre'Class and Post'Class are not allowed on entries
        and protected subprograms of untagged task and protected types.

B611008 [New] VCS Label: A4_1A
        Check various rules about the use of the Result attribute.

B611009 [New] VCS Label: A4_1A
        Check that F'Result is not allowed in the body of F.

B611010 [New] VCS Label: A4_1A
        Check that the Old attribute is not allowed in a subprogram or
        entry body, or in an accept statement.

B611011 [New] VCS Label: A4_1A
        Check the Old attribute cannot be used within a precondition
        expression nor in a generic specification outside of a
        postcondition expression.

B611012 [New] VCS Label: A4_1A
        Check that the prefix of an Old attribute cannot contain the
        Result attribute, another Old attribute, nor the index of a
        quantified expression.

B611013 [New] VCS Label: A4_1A
        For a discrete X, check that X'Old has the nominal subtype of X.

B611014 [Pending New] New VCS Label: A4_1B
                 Required after 2017-07-01
        If an Old attribute reference occurs in at a place where it is
        potentially unevaluated, check that the prefix must statically
        denote an object.

B611015 [Pending New] New VCS Label: A4_1B
                 Required after 2017-07-01
        Check that the prefix of an Old attribute cannot have a limited
        type.

B611016 [Pending New] New VCS Label: A4_1B
                 Required after 2017-07-01
        Check that tagged task and tagged protected types do not allow
        Pre'Class and Post'Class, nor are they allow on operations of
        those types.

B611017 [Pending New] New VCS Label: A4_1B
                 Required after 2017-07-01
        Check that Pre'Class cannot be specified for an overriding of a
        subprogram that does not specify Pre'Class.

B611018 [Pending New] New VCS Label: A4_1B
                 Required after 2017-07-01
        Check 6.1.1(10-16/3).

B611019 [Pending New] New VCS Label: A4_1B
                 Required after 2017-07-01
        Check 6.1.1(17/3).

C611A01 [New] VCS Label: A4_1A
        Check that preconditions and postconditions where both specific
        and class-wide contracts apply are checked properly.

C611A02 [New] VCS Label: A4_1A
        Check that the correct expressions are evaluated for specific
        preconditions and postconditions, and that they are evaluated
        at the correct point.

C611A03 [New] VCS Label: A4_1A
        Check that the correct expressions are evaluated for class-wide
        preconditions and postconditions, and that they are evaluated
        at the correct point.

C611B01 [New] VCS Label: A4_1A
        Check that X'Old has the tag of X and not the tag of the nominal
        subtype of X (for Post).

C611B02 [New] VCS Label: A4_1A
        Check that X'Old has the tag of X and not the tag of the nominal
        subtype of X (for Post'Class).

B62001C [Modified] VCS Label: A4_1A
        Changed error @35 to be optional, as the syntax error @33 is
        required (and the comment that only one error need be detected is
        bogus).

B62001D [Modified] VCS Label: A4_1A
        Changed error @41 to be optional, as the syntax error @33 is
        required (and the comment that only one error need be detected is
        bogus).

C650B04 [New] VCS Label: A4_1A
        Check that the result of a function that returns a class-wide
        tagged type does an accessibility check on the tag of the result
        object when the result expression is class-wide.

B740003 [Modified] VCS Label: A4_1A
        Modified criteria @75 so that it doesn't look like an error tag to
        the grading tools; also fixed an overlong line.

B750A07 [New] VCS Label: A4_1A
        Check that only legal expressions are allowed
        in the return expression of an expression function that returns a
        limited type.
        
B83F02C [Modified] VCS Label: A4_1A
        Updated many optional error markers to match ACATS standards.

B95007B [Modified] VCS Label: A4_1A
        Fixed error markers @62,66,70 to match ACATS standards.

BB42001 [New] VCS Label: A4_1A
        Check that the arguments of a pragma Assert have to
        have the correct types.

CB42001 [New] VCS Label: A4_1A
        Check the usual operation of pragma Assert, including the optional
        message.

BB42002 [New] VCS Label: A4_1A
        Check that the arguments of a pragma Assert cannot be reordered
        even if named notation is used.

CB42002 [New] VCS Label: A4_1A
        Check that if the assertion policy is Ignore, none of the
        expressions of a pragma Assert are evaluated and no exception is
        raised by the pragma.

BB42003 [New] VCS Label: A4_1A
        Check that a pragma Assertion_Policy is illegal if the
        policy is not recognized.

CB42003 [New] VCS Label: A4_1A
        Check that procedure Ada.Assertions.Assert evaluates its arguments
        and raises Assertion_Error regardless of the assertion policy.

CD11001 [New] VCS Label: A4_1A
        Check that appropriate entities are visible in an aspect
        specification.

BD4003A [Modified] VCS Label: A4_1A
        Moved error marker from @66 to @67 so it is on the incorrect
        construct.

CXB3023 [Pending New] New VCS Label: A4_1A
                 Required after 2017-01-01
        Check that a value of an unchecked union type (defined with
        pragma Unchecked_Union) can be passed to a C function.

CXB3024 [Pending New] New VCS Label: A4_1A
                 Required after 2017-01-01
        Check that a value of an unchecked union type (defined with
        aspect Unchecked_Union) can be passed to a C function.

===== Annex C =====

[None.]

===== Annex D =====

[None.]

===== Annex E =====

[None.]

===== Annex F =====

[None.]

===== Annex G =====

[None.]

===== Annex H =====

[None.]

===== Support =====

F611A00 [New] VCS Label: A4_1A
        Foundation for tests C611A01, C611A02, and C611A03.

F611B00 [New] VCS Label: A4_1A
        Foundation for tests C611B01 and C611B02.

How to get new and modified tests:

The official home of the ACATS test suite is the ACAA web site. The site can be accessed at www.ada-auth.org.

The complete set of official new and modified tests are available in the Mod_4_1B.Zip and Tar files on the ACAA web site.

Individual new and modified tests can be retrieved from the ACATS VCS. Simply navigate to the appropriate directory (folder), and select the test you want. Select the revision you want (usually the most recent) and it will be retrieved for you. Tests are stored in directories that correspond to the test's prefix. For instance, the test CD92001.A will be found in the CD directory. You can also browse the differences between versions of a test. [In case of a conflict, the files in the ACATS VCS should be considered the 'official' versions.]

If you cannot get the files from the official site or you cannot get the files in a format appropriate for your processor, please contact your ACAL.

The baseline ACATS 4.1 is available in various formats on the ACAA web site.

Summary lists:

Withdrawn Tests