ACATS Test Modification List

Version 4.1A
June 29, 2016

What is this?

This is the ACATS modification list for ACATS 4.1. This document includes test corrections, test removals, and new tests for ACATS 4.1. ACATS 4.1 is the pending test suite for Ada, covering the third edition of Ada, ISO/IEC 8652:2012 (usually known as "Ada 2012"); it replaces ACATS 4.0 on July 1, 2016.

ACATS Modification List version 3.1M for ACATS 3.1 is the current list for that test suite. ACATS 3.1 should be used for testing implementations that support Amendment 1 (ISO/IEC 8652:1995/AMD 1:2007, often known as "Ada 2005"), and will continue to be maintained as necessary for the foreseeable future.

ACATS Modification List version 2.6D for ACATS 2.6 is the current list for that test suite. ACATS 2.6 should be used for testing implementations of the second edition of Ada, ISO/IEC 8652:1995 (usually known as "Ada 95"); it also will continue to be maintained as necessary for the foreseeable future.

Changes from the last list:

This is the first list for ACATS 4.1; it contains tests from the most recent ACATS Modification List 4.0 which have effective dates after July 1st, 2016.

Tests B38003D, B455002, and B83F02C had incorrect error tags repaired.

Tests B3A1001 and B3A1002 had Pack6 split so that the possible error sets do not intersect.

Test C52101A was corrected to not have a comment in the form of an ERROR: tag. These should not appear in C-Tests.

Tests B62001C and B62001D were changed to make the second error optional, as the first error is illegal syntax and thus has to be detected.

The Pass/Fail criteria for test B740003 was updated so that the test tools don't pick up an error marker.

"Note" was replaced by "Optional Error" in test B95007B.

The error marker for the second error in test BD4003A was moved so that it was on the incorrect construct.

Tests CXB3023 and CXB3024 were corrected to have missing N/A => ERROR markers added. This reset their required date, such that they are not included in ACATS 4.1 itself.

New tests posted between July 1, 2016 and September 30, 2016 will become required for conformity assessments on April 1, 2017. New tests posted between April 1, 2016 and June 30, 2016 will become required for conformity assessments on January 1, 2017. New tests posted between January 1, 2016 and March 31, 2016 will become required for conformity assessments on October 1, 2016. New tests posted before January 1, 2016 are part of ACATS 4.1 and thus are required for conformity assessments. The effective dates of tests are marked in the lists.

Questions about this document? Contact the ACAA Technical Agent at agent@ada-auth.org.

ACATS 4.1 includes optional Test Grading tools

ACATS 4.1 introduces new test grading tools.

When the ACATS was designed (as the ACVC in the early 1980s), the intention always was that running it would give a simple and clear Pass or Fail result. However, grading of tests (particularly of B and L Tests) is somewhat subjective and very time-consuming. (Test grading typically involved poring over compiler listings of the entire ACATS with a large highlighter.) The subjectivity was acceptable in the days of common formal testing, as the formal testers would enforce a common grading standard (even if no one could describe what that was), but in an age of self-testing, it reduces the value of the ACATS.

As such, we're introducing a long-overdue set of automated grading tools. These tools are intended to:

The tools also will optionally give feedback to implementers about quality of error message positioning and of error cascades (does one error generate multiple messages?). (Since "goodness of an implementation" is not something that the ACATS tests for, this is not an official goal for the tools.)

This is an ambitious set of goals. It's unlikely that they'll be completely met by the initial versions of the tools. As such, their use will be completely optional (but encouraged!) for ACATS 4.1. The tools will be updated periodically by the normal process for updating the ACATS – an ACATS Modification List like this one.

Details on the operations of the tools can be found in the ACATS 4.1 documentation (that documentation can be found at http://www.ada-auth.org/acats-files/4.1/docs/UG-6.HTM). As always, if you have questions, contact the ACAA Technical Agent at agent@ada-auth.org.

Notice:

The ACAA has created and maintains the Ada Conformity Assessment Test Suite for the purpose of conformity assessments conducted in accordance with the International Standard ISO/IEC 18009 - Ada: Conformity assessment of a language processor. This test suite should not be used to make claims of conformance unless used in accordance with ISO/IEC 18009 and any applicable ACAA procedures.

List explanation:

Test numbers given are for the basic test. Categories are "Withdrawn" (the test should not be processed for Ada conformity assessments); "Allowed Modification" (either the original test or the modified one should be processed for Ada conformity assessments); "Modified" (the modified test must be processed for Ada conformity assessments); "Pending New" (the test does not need to be processed for Ada conformity assessments, but will be added in the future); "New" (the test is processed for Ada conformity assessments).

ACATS version labels are provided with all new and modified tests. Use this label to retrieve the test from the ACATS VCS. See "How to get new and modified tests".

In citations, line numbers are given relative to the start of a test file in the format of '@<line#,line#,...>'; if a test program comprises more than one file, then the particular file will be indicated by "fn", where "n" is the test-program name's file-designator numeral. E.g., "B38103E : @f1-31, ... " denotes test file b38103e1.ada. In cases where there are two or more rationales for different errors in a test program, these different rationales with their respective line citations are listed on separate lines. References to the Ada standards are given in the format:
   <section|chapter>.<clause>[.<subclause>]:<paragraph>
These references are to the Ada standard as available at http://www.adaic.org/resources/add_content/standards/12rm/html/RM-TOC.html.

 

Main list:

===== Core Language =====

Name     Category  Version label  (effective date; old version label)
         Short description.

B38003D [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Corrected error tags @70,76,82,88 so "optional" and "error" are
        on the same line, as required by the test standards.

C3A0030 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that an access discriminant is only null excluding if it
        includes a null exclusion.

B3A1001 [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Split Pack6 so possible error sets don't intersect; fixed
        overlong lines.

B3A1002 [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Split Pack6 so possible error sets don't intersect; fixed
        overlong lines.

B455002 [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Corrected casing of error tags and fixed overlong lines.

C480001 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that an initialized allocator for a class-wide type does
        an accessibility check on the tag of the initializing object.

C52101A [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Corrected comment @49 not to have the form of an error tag.

B611001 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that Pre and Post are only allowed on
        subprograms and entries (and not on subprogram instances).        

B611002 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that Pre and Post are not allowed on subprograms
        acting as completions.

B611003 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that Pre'Class and Post'Class are only allowed on primitive
        subprograms of tagged types. (Normal tagged types and interfaces.)

B611004 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that Pre'Class and Post'Class are not allowed on
        subprograms acting as completions.

B611005 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that Pre and Post are not allowed on abstract
        subprograms and null procedures.

B611006 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the restrictions on Pre'Class and
        Post'Class expressions added by AI12-0113-1 are enforced.

B611007 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that Pre'Class and Post'Class are not allowed on entries
        and protected subprograms of untagged task and protected types.

B611008 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check various rules about the use of the Result attribute.

B611009 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that F'Result is not allowed in the body of F.

B611010 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the Old attribute is not allowed in a subprogram or
        entry body, or in an accept statement.

B611011 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check the Old attribute cannot be used within a precondition
        expression nor in a generic specification outside of a
        postcondition expression.

B611012 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the prefix of an Old attribute cannot contain the
        Result attribute, another Old attribute, nor the index of a
        quantified expression.

B611013 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        For a discrete X, check that X'Old has the nominal subtype of X.

C611A01 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that preconditions and postconditions where both specific
        and class-wide contracts apply are checked properly.

C611A02 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the correct expressions are evaluated for specific
        preconditions and postconditions, and that they are evaluated
        at the correct point.

C611A03 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the correct expressions are evaluated for class-wide
        preconditions and postconditions, and that they are evaluated
        at the correct point.

C611B01 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that X'Old has the tag of X and not the tag of the nominal
        subtype of X (for Post).

C611B02 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that X'Old has the tag of X and not the tag of the nominal
        subtype of X (for Post'Class).

B62001C [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Changed error @35 to be optional, as the syntax error @33 is
        required (and the comment that only one error need be detected is
        bogus).

B62001D [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Changed error @41 to be optional, as the syntax error @33 is
        required (and the comment that only one error need be detected is
        bogus).

C650B04 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the result of a function that returns a class-wide
        tagged type does an accessibility check on the tag of the result
        object when the result expression is class-wide.

B740003 [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Modified criteria @75 so that it doesn't look like an error tag to
        the grading tools; also fixed an overlong line.

B750A07 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that only legal expressions are allowed
        in the return expression of an expression function that returns a
        limited type.
        
B83F02C [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Updated many optional error markers to match ACATS standards.

B95007B [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Fixed error markers @62,66,70 to match ACATS standards.

BB42001 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the arguments of a pragma Assert have to
        have the correct types.

CB42001 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check the usual operation of pragma Assert, including the optional
        message.

BB42002 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that the arguments of a pragma Assert cannot be reordered
        even if named notation is used.

CB42002 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that if the assertion policy is Ignore, none of the
        expressions of a pragma Assert are evaluated and no exception is
        raised by the pragma.

BB42003 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that a pragma Assertion_Policy is illegal if the
        policy is not recognized.

CB42003 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that procedure Ada.Assertions.Assert evaluates its arguments
        and raises Assertion_Error regardless of the assertion policy.

CD11001 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Check that appropriate entities are visible in an aspect
        specification.

BD4003A [Allowed Modification] New VCS Label: A4_1A Org VCS Label: A4_1
                 Required after 2016-10-01
        Moved error marker from @66 to @67 so it is on the incorrect
        construct.

CXB3023 [Pending New] New VCS Label: A4_1A
                 Required after 2017-01-01
        Check that a value of an unchecked union type (defined with
        pragma Unchecked_Union) can be passed to a C function.

CXB3024 [Pending New] New VCS Label: A4_1A
                 Required after 2017-01-01
        Check that a value of an unchecked union type (defined with
        aspect Unchecked_Union) can be passed to a C function.

===== Annex C =====

[None.]

===== Annex D =====

[None.]

===== Annex E =====

[None.]

===== Annex F =====

[None.]

===== Annex G =====

[None.]

===== Annex H =====

[None.]

===== Support =====

F611A00 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Foundation for tests C611A01, C611A02, and C611A03.

F611B00 [Pending New] New VCS Label: A4_1A
                 Required after 2016-10-01
        Foundation for tests C611B01 and C611B02.

 

How to get new and modified tests:

The official home of the ACATS test suite is the ACAA web site. The site can be accessed at www.ada-auth.org.

The complete set of official new and modified tests are available in the Mod_4_1A.Zip and Tar files on the ACAA web site.

Individual new and modified tests can be retrieved from the ACATS VCS. Simply navigate to the appropriate directory (folder), and select the test you want. Select the revision you want (usually the most recent) and it will be retrieved for you. Tests are stored in directories that correspond to the test's prefix. For instance, the test CD92001.A will be found in the CD directory. You can also browse the differences between versions of a test. [In case of a conflict, the files in the ACATS VCS should be considered the 'official' versions.]

If you cannot get the files from the official site or you cannot get the files in a format appropriate for your processor, please contact your ACAL.

The baseline ACATS 4.1 is available in various formats on the ACAA web site.

 

Summary lists:

Withdrawn Tests

(Refer to the main list for details)

None yet.

Allowed Modification Tests

(Refer to the main list for details)

B38003D  2016-10-01
B3A1001  2016-10-01
B3A1002  2016-10-01
B455002  2016-10-01
B62001C  2016-10-01
B62001D  2016-10-01
B740003  2016-10-01
B83F02C  2016-10-01
B95007B  2016-10-01
BD4003A  2016-10-01
C52101A  2016-10-01

Modified Tests

(Refer to the main list for details)

None yet.

New Tests

(Refer to the main list for details)

None yet.

Pending New Tests

(Refer to the main list for details)

B611001  2016-10-01
B611002  2016-10-01
B611003  2016-10-01
B611004  2016-10-01
B611005  2016-10-01
B611006  2016-10-01
B611007  2016-10-01
B611008  2016-10-01
B611009  2016-10-01
B611010  2016-10-01
B611011  2016-10-01
B611012  2016-10-01
B611013  2016-10-01
B750A07  2016-10-01
BB42001  2016-10-01
BB42002  2016-10-01
BB42003  2016-10-01
C3A0030  2016-10-01
C480001  2016-10-01
C611A01  2016-10-01
C611A02  2016-10-01
C611A03  2016-10-01
C611B01  2016-10-01
C611B02  2016-10-01
C650B04  2016-10-01
CB42001  2016-10-01
CB42002  2016-10-01
CB42003  2016-10-01
CD11001  2016-10-01
CXB3023  2017-01-01
CXB3024  2017-01-01
F611A00  2016-10-01
F611B00  2016-10-01