Operating Procedures

for Ada Conformity Assessments

Version 3.1

December 10, 2013

Ada Resource Association
P.O. Box 4072
Oakton, VA 22124

TABLE OF CONTENTS

EXECUTIVE SUMMARY 1
1. INTRODUCTION 5
1.1 Background 5
2. GLOSSARY OF TERMS 7
3. BODIES AND RESPONSIBILITIES 11
3.1 Ada Conformity Assessment Laboratories (ACALs) 11
3.2 Ada Conformity Assessment Authority (ACAA) 11
3.2.1 Sponsor 11
3.2.2 Technical Agent 12
3.2.3 Advisory Board 12
3.3 Conformity Assessment Clients 13
4. THE ADA CONFORMITY ASSESSMENT TEST SUITE 15
4.1 Applicability of ACATS Test Programs 15
4.2 Test Modification 15
4.3 Customization 16
4.4 ACATS Grading 16
4.5 ACATS Availability 17
4.5.1 ACATS Version Control System 18
4.6 ACATS Configuration Management 18
4.6.1 ACATS Modification List 18
4.6.2 ACATS Modification Categories 18
4.6.3 ACATS Baseline Version 19
4.6.4 ACATS Tests used 20
5. CONFORMITY ASSESSMENT 21
5.1 Establishment of Agreement 21
5.2 Self-Test Evaluation 22
5.2.1 Client Testing 22
5.2.2 Submission of Results 22
5.2.3 ACAL Analysis and Test-Issue Resolution 23
5.2.4 Incomplete Self-test Evaluation 24
5.2.5 Successful Self-testing 24
5.3 Witness Testing 24
5.4 Documentation 25
5.4.1 The Ada Conformity Assessment Test Report 25
5.4.2 The Ada Conformity Assessment Certificate 26
5.5 Use of Obsolete ACATS Versions 27
5.6 Retention of Records 27
5.7 Advertising Status 28
6. TEST CHALLENGE AND RESOLUTION PROCESS 29
6.1 Introduction 29
6.2 Resolution Process 29
6.3 Types of Resolutions 30
6.4 Reconsideration of Rejected Petitions 30
6.5 Summary 31
7. EXTENSIBILITY OF CONFORMITY ASSESSMENT 33
7.1 Implementation Classes 33
7.1.1 Base Implementation Class 33
7.1.2 Maintained Implementation Class 33
7.1.3 Rehosted Implementation Class 34
7.2 Equivalence of ACATS Results 34
7.3 The ACATR Supplement 35
7.4 Requirements for Certification by Extension or Derivation 36
7.5 Expiration of Certification by Extension or Derivation 36
7.6 Challenging Certification by Extension or Derivation 37
7.6.1 Information Required to Challenge Certification 37
7.6.2 The Challenge Process 37

APPENDICES

POINTS OF CONTACT A-1
TEST ISSUE FORMAT B-1
DECLARATION OF CONFORMITY C-1
ACATR SUPPLEMENT FORMAT D-1
ACRONYMS E-1
REFERENCES F-1

Executive Summary

The International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), through Working Group 9 (WG9) of Subcommittee 22 (SC22) of their Joint Technical Committee 1 (JTC1), have established an International Standard titled Ada: Conformity Assessment of a Language Processor (ISO/IEC 18009:1999), referenced in this document as [ISO 99]. This standard specifies requirements for five aspects of a conformity assessment program, as follows:
The current document represents the Ada Resource Association (ARA)’s establishment of an ACAA governed by the procedures (ACAP) given herein. The primary goal of this document is to conform to the International Standard [ISO 99] (providing an ACAP as defined in that document).
[ISO 99] requires a periodic review and update of the ACAP. This document represents the results of the first such review and update. The major change is to allow vendors to use newer versions of the test suite for certifications by extension and derivation, and accordingly to extend the expiration dates of these certifications.
The following paragraphs summarize the conformity assessment system by using the language of the International Standard [ISO 99].

Ada Conformity Assessment Test Suite

The ACATS was originally derived from the Ada Compiler Validation Capability (ACVC). The ACVC was a conformity test suite (with supporting documents) developed under contract to the United States Government and made available for public use; it was designed to ensure that Ada processors achieve a high degree of conformity to the Ada language standard. A processor is tested when operating in a specific hardware and software configuration. The ACATS is customized by a testing laboratory (ACAL) for each processor that is subjected to conformity assessment; customization consists of adjusting the ACATS appropriately for various implementation characteristics. The ACATS is maintained by the ACAA; new releases of the ACATS are defined by changes resulting from the discovery of deficiencies in the test programs and by changes in the Ada standard (for instance, the adoption of Technical Corrigendum 1 [TC1], Amendment 1 [Amd1], and Ada 2012 [Ada2012]).
For an Ada processor to successfully complete conformity assessment, it must process each test program (for the “core” language) of a customized ACATS so that the result is graded Passed, Inapplicable, or Unsupported by ACATS grading criteria. A testing laboratory customizes the test suite for a particular processor by appropriately setting test parameters, by removing “withdrawn” tests (tests ruled by the ACAA to be in error) and certain inapplicable tests, by splitting as needed test files with multiple intended errors so to enable complete error detection, by using any other modified tests as directed by the ACAA, and by including each optional set of tests (see below) as requested by the client.
In addition to the specification of a “core” language, [Ada2012] contains several Specialized Needs Annexes (SNAs); these specify language requirements designed to meet the particular needs of various general application domains, such as information-systems programming. A processor for [Ada2012] need not include implementation of any of these annexes, or it may implement only some of the features of these annexes. Whereas all ACATS test programs for the core language must be processed during conformity assessment, those for the SNAs are processed only upon client request. A conformity assessment is judged successful (leading to the issuance of an ACAC) only if all tests for the core are correctly processed; the certificate will additionally give credit for support of a SNA to the extent that the relevant set of tests is correctly processed.

Conformity Assessment

Conformity assessment involves interaction between the ACAL client and both the ACAL and the ACAA. The assessment process consists of well-defined actions which, when completed successfully, result in the award of an ACAC for the tested processor. The key actions in the conformity assessment of an Ada processor are:
  1. The client and an ACAL reach a formal agreement for conformity assessment, including the dates for the submittal of the results of client-administered processing of the ACATS and for ACAL witness testing.
  2. The client petitions for deviation from the requirements of each ACATS test program that is believed to be wrong for the candidate implementation(s).
  3. The ACAA rules on the client petitions.
  4. The client processes the ACATS on the candidate processor(s) and submits the results to the ACAL.
  5. The ACAL analyzes the results of the client’s independent processing of the ACATS. (If the results are not acceptable, the previous action must be repeated, and new results analyzed.)
  6. The ACAL conducts witness testing of the candidate processor(s), documents this testing in an Ada Conformity Assessment Test Report (ACATR), and submits the ACATR to the ACAA for review.
  7. The client signs a Declaration of Conformity for each candidate processor.
  8. The ACAA reviews the ACATR, with comments to the ACAL. The ACAA recommends to the ACAL that a conformity assessment certificate be issued for the tested processor(s), if the testing is successful.
  9. The ACAL issues a conformity assessment certificate for each tested processor upon the successful completion of the preceding actions.
Hence, successful testing of an Ada processor concludes with the ACAL's awarding a conformity assessment certificate for that processor (working in a specific configuration) to the client. This conformity assessment certificate attests that the processor has been subjected to an Ada conformity assessment and that no evidence of non-conformity was found. The processor is said to be "certified as conforming," as described in [ISO 99], clause 2.1. The processor will be listed in the ARA's Certified Processors List (CPL). The client may perform maintenance on the processor and may claim conformity for such derived versions in accordance with the ACAA procedures, so long as the client ensures that they produce the same ACATS results as are documented in the ACATR. This maintenance may even include adaptive maintenance that enables the processor to run on entirely different host computers (i.e., re-hosting) or to target closely related target computers. The ACAA provides a means for listing derived processors in the CPL.

1. INTRODUCTION

This document provides operating procedures of the Ada Conformity Assessment Authority (ACAA). This body is a part of an organization that meets the requirements for assessing conformity of an Ada language processor, as given in [ISO 99]. The other bodies making up this organization are the Ada Conformity Assessment Laboratories (ACAL), which perform the actual conformity assessments using the Ada Conformity Assessment Test Suite (ACATS). The end product of a successful conformity assessment is an Ada Conformity Assessment Certificate (ACAC), indicating that a particular Ada language processor is "certified as conforming," as defined in [ISO 99].
This document forms an Ada Conformity Assessment Procedure (ACAP), as defined in [ISO 99].
Detailed procedures regarding the application of the ACATS are given in the ACATS User's Guide [ACATS UG].

1.1 Background

The United States Department of Defense (DoD) sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. The AJPO established a certification system to realize the benefits of standardization, which include the ability to transfer software and programming expertise between computer systems that use a conforming Ada processor. When the AJPO ceased its operation of the certification system, the Ada Validation Facilities agreed to act as ACALs under the provisions of the emerging International Standard (now standardized as [ISO 99]). The Ada Resource Association, in cooperation with the Ada Joint Program Office, facilitated the identification and U.S. Government funding for a candidate ACAA and produced this document defining its operating procedures. The ACALs then agreed to designate the identified organization as the ACAA and to use these procedures as the ACAP.
It is important to note the scope and intent of conformity assessment. The purpose of conformity assessment is to ensure that Ada processors achieve a high degree of conformity with the Ada standard ([Ada2012]). Characteristics such as performance and suitability for a particular application are not specified by the standard, and thus are outside the scope of Ada conformity assessment. Moreover, the ACATS is a set of test programs intended to check broadly for correct implementation; it is not possible to exhaustively test for conformity. Thus, conformity is checked only to the extent of these tests; processors that are certified as conforming may fail to conform to the standard in ways peculiar to each, under particular circumstances.
Witness testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of Ada conformity assessment is to identify processors that may be procured and used to develop application programs that meet the [Ada2012] goals of portability and interoperability.
The ACATS (test suite) is not designed to replace the client’s quality assurance testing or systematically to detect inconsistencies or “bugs”, but to verify that the tested processor correctly supports all required features. Rather than exhaustive testing of permutations of features, the test suite contains a carefully chosen set of test cases that cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard.
Neither is conformity assessment intended as a means of performance benchmarking. The Ada Conformity Assessment Test Report (ACATR) which documents the witness testing does not contain information about the speed, cost, or efficiency of executing the conformity assessment tests.

2. GLOSSARY OF TERMS

Ada: The programming language defined by [Ada2012].
Ada Conformity Assessment: The process of checking the conformity of a processor to the Ada programming language standard [Ada2012] and of issuing a conformity assessment certificate for the implementation.
Ada Conformity Assessment Authority (ACAA): The organization that provides the technical and administrative oversight of Ada Conformance assessment activities.
Ada Conformity Assessment Certificate (ACAC): A certificate issued by authority of the ACAA and an ACAL for a successfully tested Ada processor (see Section 5.4).
Ada Conformity Assessment Laboratory (ACAL): An independent testing laboratory performing Ada conformity assessments.
Ada Conformity Assessment Test Report (ACATR): A report produced by an ACAL that documents the witness testing of an Ada processor (see Section 5.4).
Ada Conformity Assessment Test Suite (ACATS): The means for testing conformity of Ada processors, consisting of the test suite, the support programs, and the User's Guide [ACATS UG]. The ACATS was based on the Ada Compiler Validation Capability (ACVC).
Ada Conformity Assessment Test Suite Modification List (ACATS Modification List): A listing of modifications to the ACATS correcting flaws found in the suite. This list is periodically issued by the ACAA (see Section 4.6.1).
Ada Conformity Assessment Test Suite Version Control System (ACATS VCS): A publicly accessible program that stores the current version of the ACATS, as well as any modified tests used for conformity assessments (see Section 4.5.1).
Ada processor: A processor for the Ada programming language as defined in [Ada2012].
Ada Rapporteur Group (ARG): A subgroup of ISO/IEC/JTC1/SC22/WG9, the International Organization for Standardization Working Group for Ada. Members of the ARG are appointed by the convener of the ISO working group for the purpose of resolving issues with respect to the interpretation of the Ada programming language.
Adaptive Maintenance: Maintenance performed to adapt a product to a changed environment.
Base implementation class: An implementation class in which the processor has been awarded certified status through testing by an ACAL (see Section 7.1.1).
Certified Processors List (CPL): A publicly available list of processors to which the ACAA has granted certified status. The CPL is maintained by the ACAA.
Certified status: (Also "certified as conforming") The status granted to an Ada processor by the award of an ACAC (see Section 5.4).
Client: An individual or corporate entity that has an agreement with an ACAL that specifies the terms and conditions for ACAL services (of any kind) to be performed. (Also used to refer to an organization that intends to make such an agreement.)
Computer system: A system containing one or more computers and associated software. [ANSI/IEEE 90] In this document, a computer system comprises the hardware and software (operating systems, kernels) that are essential to the operation of the processor or the compiled code; e.g., input/output devices are generally not included in this definition.
Configuration: A specific host and target computer system. “Configuration” is usually used along with “processor” to completely specify a conformity assessment.
Configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specific requirements. [ANSI/IEEE 90]
Conformity: Fulfillment by a product, process or service of all requirements specified. [ISO/IEC 86] See also Subclause 1.1.3 of [Ada2012].
Core language: The Sections 1-13 and Annexes A, B and J of [Ada2012].
Corrective maintenance: Maintenance performed to correct faults.
Customized test suite: The ACATS tests, adjusted as necessary, that must be used for witness testing of a given Ada processor (see Section 4.3).
Declaration of Conformity: A formal statement from a client declaring that conformity is realized on the Ada processor for which conformity assessment status is requested (see Section 5.2).
Equivalence (of ACATS results): A set R of ACATS results is equivalent to a previously evaluated set of results, P, provided that (1) the ACATS used in producing R is either the same as the ACATS used in producing P, with the possible exception of using different implementation-specific substitution values, or is the current ACATS with the same code modifications applied as the ACATS used in producing P; and (2) application of the ACATS grading rules results satisfies all of the following: (a) every test graded as Passed in P is also graded as Passed in R; (b) every test graded as Inapplicable in P is graded as Passed or Inapplicable in R; (c) every test graded as Unsupported in P is graded as Passed, Inapplicable, or Unsupported in R; and (d) every test not present in P is graded as Passed, Inapplicable, or Unsupported in R (see Section 7.2).
Host computer system: The computer system on which a processor is installed and executes.
Implementation: A processor running on a particular configuration.
Implementation class: A collection of implementations that are sufficiently closely related that the certified status of one member of the collection may be extended to the other members, provided that certain conditions are met (see Section 7.1).
Maintained implementation class: An implementation class in which the processor is derived (by applying corrective and perfective maintenance changes) directly from the processor that has been awarded certified status by ACAL testing, and in which the processor has a mode of operation in which it can produce ACATS results equivalent to those of the processor named in the ACAC (see Section 7.1.2).
Operating system: A collection of software, firmware, and hardware elements that controls the execution of computer programs and provides such services as computer resource allocation, job control, input/output control, and file management in a computer system. [ANSI/IEEE 90]
Perfective maintenance: Maintenance performed to improve performance or maintainability. [ANSI/IEEE 90]
Processor: A compiler, translator, or interpreter. The processor includes all tools used in creating programs. For instance, many systems will include a linker in the processor. A processor works in conjunction with, but does not include, a configuration. In this document, processor typically means an Ada processor.
Rehosted implementation class: An implementation class in which the processor is derived directly (by applying corrective, perfective, and adaptive maintenance changes) from the processor that was awarded certified status by ACAL testing; in which the common target computer system is the same as that of the certified processor; and in which the processor has a mode of operation in which it can produce ACATS results that are equivalent to those of the certified processor (see Section 7.1.3).
Self-testing: The process of producing the results of processing an appropriately customized test suite by the client (see Section 5.2).
Software maintenance: Modification of a software product after delivery to correct faults, to improve performance, or to adapt the product to a changed environment. [ANSI/IEEE 90]
Specialized Needs Annexes: Annexes C through H of [Ada2012]. These Annexes define standards for additional functionality required by specific application areas. An Ada processor may support some or none of these annexes.
Target computer system: The computer system on which the executable code generated by a processor is loaded and executes.
Test issue: (Also "dispute") Any problem arising during conformity assessment (see Section 6).
Validated: Equivalent to “certified status”. The status granted to an Ada processor by the award of an ACAC (see Section 5.4). We define this equivalent term to correspond to common usage in the Ada community.
Validation Certificate: Equivalent to “Ada Conformity Assessment Certificate (ACAC)”. A certificate issued by authority of the ACAA and an ACAL for a successfully tested Ada processor (see Section 5.4). We define this equivalent term to correspond to common usage in the Ada community.

3. BODIES AND RESPONSIBILITIES

This section specifies the roles of the bodies that are responsible for Ada conformity assessment of clients who receive service from them.

3.1 Ada Conformity Assessment Laboratories (ACALs)

An ACAL is an independent testing laboratory that performs Ada conformity assessment activities. [ISO 99] includes a list of requirements that a testing laboratory must meet in order to be considered an ACAL. These requirements will not be repeated here. The ACAL operates under an ACAP consisting of its own operating procedures and the procedures defined in this document. An ACAL performs the following principal functions:

3.2 Ada Conformity Assessment Authority (ACAA)

The ACAA ensures worldwide commonality of the Ada Conformity Assessment Process. The technical and administrative functions of the ACAA are carried out by a technical agent. It is established by a sponsor and is advised by an Advisory Board.

3.2.1 Sponsor

The Ada Resource Association, a trade association of Ada product suppliers, sponsors the ACAA. The sponsor is responsible for the following:

3.2.2 Technical Agent

ACAA technical agent supports and coordinates the activities of the ACALs by:

3.2.3 Advisory Board

The ACAA Advisory Board represents the interests of the wider Ada community in the Ada Conformity Assessment process. Issues of policy and procedures are brought to the attention of the Board, which may make recommendations as to their resolution. Board members are appointed by the ACAA sponsor, and include (but are not limited to) the following:

3.3 Conformity Assessment Clients

A client is an individual or organization that contracts with an ACAL for conformity assessment services. Clients are required to provide accurate and complete information as specified in these procedures and the procedures of the ACAL.

4. THE ADA CONFORMITY ASSESSMENT TEST SUITE

The designated ACATS is the suite of conformity tests, support software, and documentation formerly known as the Ada Compiler Validation Capability (ACVC). The ACVC was developed under various contracts with the United States Department of Defense. It is designed to demonstrate the conformity of an Ada processor with the standard [Ada2012]. The use of the ACATS is documented in the ACATS User’s Guide[ACATS UG], which explains the criteria for evaluating the results of the individual tests. While the ACVC was produced under contract to the United States Government, it is available to any individual or organization. The ACAA controls the content of the ACATS as it is used in conformity assessments, including the modification and addition of tests. Questions concerning Ada conformity assessment or comments on ACATS test programs should be submitted to the ACAA (see Appendix A, Points of Contact).

4.1Applicability of ACATS Test Programs

Each ACATS test program has one or more test objectives that are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada processor (for example, “check that the proper exception is raised when Float’Machine_Overflows is True”). These test programs generally contain an explicit indication of their applicability and the expected behavior of processors for which they do not apply. The determination of applicability is made according to the grading criteria in the ACATS User's Guide or in the internal test documentation, or as a ruling by the ACAA. For a processor to be certified as conforming, all applicable test programs for the core language (as defined in [Ada2012]) must be processed and passed according to the specified grading criteria.
Reference [Ada2012] includes certain sections designated as Specialized Needs Annexes (SNA). The set of ACATS test programs for any of the SNAs will be processed only upon client request (to demonstrate full or partial support of the Annex). As permitted by [Ada2012], test programs for the SNAs may be rejected at compile time or may exhibit run-time behavior that indicates a lack of support for requirements that only apply to SNAs. The ACAA may rule that tests producing such behavior are graded as "Unsupported". If the ACAA finds that the behavior is not in accordance with the permission granted by [Ada2012], then the tests are graded as "Failed". Tests graded as Unsupported are reported in the ACATR and the ACAC, but these results do not affect the designation of the processor as being certified as conforming. On the other hand, tests graded as Failed are evidence of non-conformity, precluding the issuance of an ACAC for the candidate processor.

4.2 Test Modification

The various ACALs and the ACAA strive to apply the ACATS as uniformly as is practical to all Ada processors. In order to apply common test objectives that depend on implementation-dependent characteristics (e.g., line lengths and numeric types), some test programs must be adjusted to a given implementation following the procedures in [ACATS UG]. These adjustments consist of inserting implementation-dependent values in prescribed places in certain test programs.
In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (for example, the algorithm for recovering from syntax errors). The allowable changes for each Ada processor are determined by the [ACATS UG] and the ACAA, and may require ACAL assistance — especially in the case of processor error-recovery problems.
In order to meet a test objective, it may be required to modify the code, the processing method, or the grading of a test program. Only the ACAA shall make the decision to use any of these modifications, as described below:

4.3 Customization

The ACAL customizes the ACATS for each processor that is subject to witness testing. This customization always includes making all requiredimplementation-dependent substitutions. It may also include making code modifications that the ACAA directs for that specific conformity assessment as well as removing some inapplicable test programs as allowed by the ACATS User's Guide.

4.4 ACATS Grading

The result of processing an ACATS test program can be given only one of four possible grades: Passed, Inapplicable, Unsupported, and Failed. The first three grades are considered to constitute acceptable results. ACATS test programs that contain illegalities (which an implementation must detect) generate diagnostic output that must be inspected manually or by pattern-matching algorithms, matching system diagnostics to the intended errors. Executable ACATS test programs generate output using report procedures, which can be graded automatically. The ACATS report package, Report, contains specific output procedures for the two grades Failed and Inapplicable. If neither of these is invoked, the Report.Result procedure will report Passed or Tentatively Passed (indicating that the test has passed if manual inspection reveals that specific additional requirements are met). These results are the only ones that are generated by the test code (if no result is reported, that is, if the test completes abnormally, the result is graded Failed). The grade Unsupported is established as a means of grading tests that apply to the Specialized Needs Annexes (SNAs), as explained below.
The ACATS test programs for the SNAs pose two problems for using the three conventional grades of Passed, Inapplicable, and Failed. The broad problem is that full support of any such Annex is not required for conformity to [Ada2012] — there may be no support, or merely partial support. Unfortunately, there is no way to discriminate between full and partial support if only those three grades are used, since the grades Inapplicable and Failed are not appropriate for this (an implementation is not allowed to provide deviant semantics for an unsupported Annex feature — that would be a conformity assessment failure). The second problem is that there are some test programs for Core features that are applicable also to a SNA, in particular, the test programs for representation items. These programs constitute tests for features that are defined in the Core as optional, but are mandatory for full support of the Systems Programming Annex (which itself is mandatory for full support of the Real-Time Systems Annex).
Therefore, the ACAL grades the result of processing such an ACATS test program (i.e., one that uses a feature required by, or defined in, an Annex) as Unsupported, if the prima facie result is failure but the implementation's processing of the test program is an acceptable form of non-support. For example, if a processor does not support a particular form of a representation clause, it must reject any test program that uses it — such rejection of an executable test is usually graded Failed, but is graded Unsupported if the implementation does not claim support of the relevant SNA. A processor that accepts the representation clause and reports Failed, on the other hand, is deemed to have failed the test regardless of any claim (or lack thereof) of support for the SNA.

4.5 ACATS Availability

The current baseline version of the ACATS is available to the general public from an ACAL or from an Internet site supported by the ARA. The current and any previous versions of the ACATS Modification List are available from the same sources. ACALs may assist the client in format conversion when providing the ACATS in a particular distribution medium. If a client has a need for a superseded version of the ACATS, it may be available from the ACAA or from an ACAL. See APPENDIX A for points of contact

4.5.1 ACATS Version Control System

The ACAA maintains an official ACATS web site, accessible via the Internet. The web site includes a web-accessible version control system, which contains the official version of the ACATS. Both old and new versions of tests are accessible given the test name and version label desired.
Instructions for using the ACATS version control system are available on the web page.

4.6 ACATS Configuration Management

Test challenges and ARG interpretations may reveal flaws in the ACATS. The ACAA may remove, repair, and insert tests in response to those needs. These test suite changes are listed in the Ada Conformity Assessment Test Suite Modification List (ACATS Modification List, or AML for short). The ACAA issues this list as needed. The AML contains information about affected tests and instructions for acquiring new and modified tests. The official versions of the tests (including modified and new tests) are available via the official ARA web site and other sources (see Section 4.5 ACATS Availability)

4.6.1 ACATS Modification List

New versions of the ACATS Modification List will be issued when test suite changes are needed. The list contains at least the following information:
The official version of a test is always available from the ACATS VCS. Information in the list is advisory only; in the case of a conflict between the ACATS VCS, and the list, the ACATS VCS is assumed correct.

4.6.2 ACATS Modification Categories

The ACAA may issue various kinds of test suite modifications. Each affected test has one of the following categories:
Withdrawn:The test is seriously flawed. It may have errors that cannot be corrected, or may require extensive corrections. It is removed from the test suite. Conformity assessments do not process such tests.
If a correction of a Withdrawn test is prepared, it will be treated as a new test.
Allowed Modification: The test has minor flaws. A modified version has been posted on the ACATS VCS. The test has an effective date that specifies when the test will be moved to the Modified Category.
The effective date will always be at least three months after the posting date, and will be at the beginning of a quarter (that is, January 1st, April 1st, July 1st, or October 1st). A conformity assessment may choose to process the original test or the new, modified test. Test choices can be made on an individual test basis. That is, a conformity assessment may choose to process some new modified tests while using the original tests for others.
Modified:The test has minor flaws. A modified version has been posted on the ACATS VCS. Conformity assessments must process the modified test.
Pending New: The test is newly created, or is a correction of a test that was previously withdrawn, or has added test cases. It is posted on the ACATS VCS. The test has an effective date that specifies when the test will be moved to the New category. The effective date will always be at least six months after the posting date, and will be at the beginning of a quarter (that is, January 1st, April 1st, July 1st, or October 1st). Pending New tests are not used for conformity assessment until the effective date is reached. Users of the test suite are encouraged to run the tests as soon as possible.
If it becomes necessary to modify a Pending New test, the effective date is adjusted as if the test was newly created.
New: The test is newly created, or is a correction of a test that was previously withdrawn, or has added test cases. It has been posted on the ACATS VCS for at least six months. Conformity assessments must process the test.

4.6.3 ACATS Baseline Version

The ACAA baselines the ACATS periodically. When the ACATS is baselined, the original official set of files is updated with all of the changes specified in the ACATS Modification List. Allowed Modification and Pending New tests are not included (since they have not been available long enough to include). Additionally, the documentation associated with the test suite is updated. Future versions of the ACATS Modification List are then based on the new test version. Any Allowed Modification and Pending New tests will be listed in the initial version of the ACATS Modification List for the new baseline version.
The effective date of a baseline version will be announced at least three months prior to its being effective, and should be at the beginning of a quarter (that is, January 1st, April 1st, July 1st, or October 1st). The documentation will be available not less than 30 days prior to it being effective. Conformity assessments started after the effective date must use the new baseline test suite. (Note that the tests that make up the baseline version are known on the date that the effective date is announced, and are accessible on the web site, even before the final version of the baseline documentation is available.)

4.6.4 ACATS Tests used

The tests used for a particular conformity assessment are defined as the tests from a particular baseline version of the ACATS, modified as follows:
Each ACATR identifies the baseline version of the ACATS, and documents all modifications made to that baseline version of the ACATS.
Conformity assessments must use the most recent version of the ACATS Modification List at the start of witness testing. (Note that the set of required tests is always known at least three months in advance.) Most tests and support files modified for an individual conformity assessment are included in the ACATS Version Control System. Tests modified only by making implementation-dependent substitutions (typically by using a tool) will not be included. B-Test splits allowed by section 4.2 and tests modified only by splitting between compilation units will also not be included.
The files will be posted on the ACATS Version Control System before the ACATR is issued. Each conformity assessment has a unique version label, which can be used on the web site to access all of the files that differ from the baseline versions. The version label is included in the ACATR. It is intended that the information in the ACATR and the files available on the official ACATS Version Control System will allow users to reproduce the conformity assessment on their own.

5. CONFORMITY ASSESSMENT

In order for a client to obtain a conformity assessment certificate and an ACATR, the client, the ACAL, and the ACAA must complete number of steps. The same ACATS version, including the application of the requirements of the ACATS Modification List, must be used to complete the steps described in this section. Anyone intending to obtain a conformity assessment certificate should contact an ACAL without delay for advice on the handling of the ACATS, on interpretation of the test grading criteria, and on the operational procedures of that ACAL.
The required steps follow:
  1. Establishment of Agreement
  2. Self-Test Evaluation
  3. Witness Testing
  4. Documentation

5.1 Establishment of Agreement

In order to obtain conformity assessment services, an interested party must become a client of an ACAL by reaching a formal agreement. This agreement addresses the following topics:
The schedule for events, deliverables, and payments should take into account the fact that certain steps in the conformity assessment process require interaction with the ACAA. The ACAA and ACAL will keep confidential a client’s intent to obtain a conformity assessment certificate and the projected schedule for conformity assessment. If the client requests more restrictive confidentiality conditions for reasons of national security or procurement sensitivity, the client will provide to the ACAL an official, written statement describing the request and the reason(s) for the request; the ACAL will also obtain further guidance from the ACAA.

5.2 Self-Test Evaluation

Self-test evaluation entails a series of actions and is usually where the bulk of the conformity assessment effort is expended. These actions are described in the following subsections.

5.2.1 Client Testing

After entering into a formal agreement, the client obtains a customized test suite from the ACAL. (At the client’s risk, the client may prepare this customized test suite according to instructions in the ACATS User's Guide, rather than obtaining it from an ACAL). The client then processes all the tests in this customized test suite using the candidate processor on the candidate configuration or on another configuration that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of options controlling the production of information output. (For example, options that control the format of listings, the format of error messages, and the generation of listings may vary.) Any other exception constitutes a test issue that must be resolved with the ACAL (see Section 5.2.3). Test issues should be sent to the ACAL for analysis as soon as possible.
Self-test activities include as a minimum the processing of an appropriately customized test suite by the client, preparation of a client supplied Declaration of Conformity, and submission of any test issues.

5.2.2 Submission of Results

Upon completion of self-testing, the client delivers the complete set of results in the agreed format to the ACAL. (See Section 5.2.4 for an alternative to submission of complete results.)
Results are accompanied by the following information:
The Declaration of Conformity states that the organization responsible for the production, maintenance or distribution of the Ada processor is offering a product that is in conformity with [Ada2012]. The client must ensure that the information contained in the Declaration of Conformity does not infringe on the rights of a third party, and may be required to provide a written statement of consent from any third party involved. The Declaration of Conformity becomes part of the ACAL records and is copied into the ACATR. The ACAL will not issue a certificate until the ACAA has reviewed a signed Declaration of Conformity. (See Appendix C for an example of the Declaration of Conformity.)

5.2.3 ACAL Analysis and Test-Issue Resolution

The ACAL analyzes the client's submitted results of self-testing, checking that all test programs have produced acceptable results according to the ACATS evaluation criteria. During this analysis period, the client and the ACAL resolve any test issues.
A test issue is defined to be any of the following:
A client may challenge an ACATS test program's correctness or applicability to a particular implementation. Such challenges should be presented to the ACAL in the petition format given in Appendix B. The ACAL will forward any petitions to the ACAA for resolution; the ACAA will strive to rule on the petition within two weeks of receiving it. The ACAA reports all challenges and rulings to each ACAL. However, an ACAL may not apply an ACAA ruling for one conformity assessment to another conformity assessment without the ACAA so directing. (See Section 6 for a description of the Challenge and Resolution Process.)
In some cases, it may be agreed to leave a test issue until witness testing. For example, it might be impossible to check the processing of control characters by inspecting printed results. The ACAL will note any unresolved issues and describe the results that are expected during witness testing. It is also possible that the client information for the production of the customized test suite (see Section 5.2.2) was insufficient, so that corrections to the customized test suite must be made, requiring additional processing.

5.2.4 Incomplete Self-test Evaluation

The ACAL and the client may agree that, at the client’s risk, parts of the customized test suite need not be processed during self-testing. There are two typical situations, as follows:
The normal practice is to submit complete self-testing results for at least one of the implementations under test. The ACAL may require the submission of complete self-testing results.

5.2.5 Successful Self-testing

Self-testing is successful if the analysis of results and the resolution of test issues show that all results have been provided and are acceptable. Self-testing is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues is deferred until witness testing by agreement between the ACAL and the client.

5.3 Witness Testing

Upon successful completion of self-testing, with or without caveats, the ACAL witnesses testing of the Ada processor in accordance with the formal agreement between the ACAL and client. Witness testing takes place in the presence of qualified ACAL ersonnel. “Presence” means either physical presence or telepresence as agreed between the ACAL and client. Telepresence must be sufficient for the ACAL to carry out the observations and monitoring required below. The ACAL supplies a customized test suite that it has prepared based upon client information and any information collected during the resolution of test issues. The customized test suite will include the set of test programs for the core language and each set, as requested by the client, any (or none) of the Specialized Needs Annexes (SNAs). The ACAL verifies that the processor identification, including identification of the processor and configuration (hardware systems and operating systems), matches that given in the Declaration of Conformity. (If it does not agree, then the client must provide a new Declaration of Conformity.)
The ACAL observes the installation of the customized ACATS on the host computer system, monitors the processing of the customized ACATS on the host and target computer systems, and evaluates the results. The entire customized test suite should be run on a single copy of the Ada processor on a single configuration using a unique set of option settings of the processor. (Differences in options controlling the production of information output and differences from accepted test issues are allowed. See section 5.2.1.) If the ACAL determines that the results agree with those obtained from self-testing and are satisfactory with respect to any caveats, the witness testing has been successful; otherwise, the test is unsuccessful. If any result of testing with a set of test programs for a Specialized Needs Annex is unacceptable, the test report and certificate of the conformity assessment will not recognize that the set was processed.

5.4 Documentation

Each conformity assessment effort is documented by an ACATR, and each successful effort is further documented by an ACAC.

5.4.1 The Ada Conformity Assessment Test Report

An ACATR is produced for each processor and configuration subjected to witness testing. Each ACATR contains, at a minimum, the following information:

5.4.1.1 ACATR Production

The ACATR is prepared by the ACAL but includes material that is produced by the client, such as the documented processor options used during witness testing. A draft version of the ACATR, based on results and circumstances implied by the evaluation of self-testing results, is sent to the ACAA for review. The draft version is also submitted to the client for review during witness testing, and is updated to account for client comments and observations made during witness testing. For a successful conformity assessment, the final version of the ACATR is signed by the ACAL and the ACAA. For an unsuccessful conformity assessment, the final ACATR is provided to the client only.
Final test reports will never be modified. If it becomes necessary to correct a final ACATR, the ACAL will prepare a separate document titled “Supplement to Ada Conformity Assessment Test Report <unique report identifier>”. Such a supplement will meet the applicable requirements of section 5.4.1.

5.4.1.2 ACATR Availability

The final version of the ACATR for a successful conformity assessment is available to the general public from the client, from the ACAL that produced it, and from the ACAA in electronic form. The ACAL may require payment of a fee for ACATR reproduction and delivery. (See Appendix A for points of contact.) By including an appropriate request on the Declaration of Conformity (Appendix C), the client may disallow public availability of the ACATR and the ACAC.

5.4.2 The Ada Conformity Assessment Certificate

With the concurrence of the ACAA, the ACAL issues an Ada Conformity Assessment Certificate (ACAC) for each processor and configuration that was subject to successful witness testing. The information on the certificate is derived from the client's Declaration of Conformity and the ACATR. The ACAC conveys to the processor and configuration the status of certified as conforming, as defined in [ISO 99]. An entry is made in the CPL for each ACAC, unless the client has requested confidentiality on the Declaration of Conformity (see Appendix C).
The ACAC contains the following information:
Note that an ACAC attests that testing was performed on a specific processor using a specific test suite running on a specific configuration, following the Ada Conformity Assessment Procedure, and that no evidence of non-conformity was detected. It does not certify that the processor is free of defects, nor does it certify that the processor is usable for any particular purpose.
ACACs expire two years after issuance. When an ACAC expires, the corresponding entry in the CPL is clearly identified as expired. (Entries for derived processors may also expire at the same time, see section 7.5). Certificates expire in order to encourage periodic retesting of processors, which ensures that they continue to meet the requirements of conformity assessment.

5.5 Use of Obsolete ACATS Versions

For some special procurement requirements, a client might wish to have witness testing done with an obsolete version of the ACATS. The ACAP does not include any procedures for recognizing testing with obsolete test suite versions, but the ACALs may provide such a service outside the system. Ada Conformity Assessment Certificates will not be issued for testing with obsolete test suites, nor will CPL entries be created based on such testing.

5.6 Retention of Records

The ACAA retains a copy of each ACATR (which includes a copy of the Declaration of Conformity and the ACAC), records pertaining to issues and their resolution, and a copy of each registration request. The ACAL retains a copy of each ACATR, a copy of the customized ACATS used in witness testing, and a copy of the witness testing results. The ACAA retains its records until at least five years following expiration of the ACAC. Each ACAL's procedures specify the length of time that its records are retained, but records must be retained at least seven years after the completion of witness testing.

5.7 Advertising Status

The client must agree not to advertise or make public claims that the Ada processor is certified as conforming until after receiving the ACAC or receiving formal notification from the ACAL that it has issued an ACAC. A client who intends to advertise the completion of events that indicates progress toward completion of conformity assessment must sign a waiver of confidentiality. If a waiver of confidentiality has been signed with the ACAL, the ACAL will respond to inquiries about the client’s advertisements or public claims by acknowledging receipt of conformity assessment materials (i.e., a formal agreement, self-testing results, or witness testing results) without judgment concerning the success of the witness testing.

6.TEST CHALLENGE AND RESOLUTION PROCESS

This section presents the process whereby tests may be challenged, possibly resulting in their modification or withdrawal.

6.1 Introduction

A “deviation” is defined by the ACATS User's Guide as any result from processing an ACATS test program that is not a Passed or Inapplicable result according to the established grading criteria. This intentionally broad definition of a “deviation” is intended to ensure that processor implementers bring all deviant test results to the attention of the ACAA or ACAL, without assuming that such results are acceptable. In petitioning for acceptance of a deviation, the petitioner provides a rationale for each challenge made against a test program. Petitions are sent to the ACAA, usually electronically, by the petitioner or by an ACAL on behalf of its client. For each deviation that is accepted (that is, when the ACAA rules in favor of the petition), generally some correction is indicated for the cited tests. The ACAA may withdraw a test program or require that a modified version of the test be processed (see section 6.3). Withdrawal of a test program or the provision of a modified version of a test results in the release of a new version of the ACATS Modification List.

6.2 Resolution Process

The ACAA typically resolves challenges by any of three methods:
  1. a resolution that was made previously is applied to the current petition (e.g., the same petition might be submitted at different times by different petitioners);
  2. the resolution can be determined unequivocally based on the Ada standard or Ada Commentaries; or
  3. the resolution is based on the deliberations of a body of Ada experts.
Although these procedures do not set a time limit for reaching a resolution, the ACAA attempts to rule on petitions within two weeks. Clients should submit challenges well in advance of a scheduled witness testing date (see Section 5.1).
On receipt of a petition, the ACAA checks whether the issue matches any that have been previously resolved. If the challenge is new, it is given an initial ACAA analysis that involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous deliberations. Often the ACAA consults Ada experts in order to resolve a petition. The identity of the petitioner is not disclosed when consulting outside experts. Resolution of a petition is made by the ACAA, and all ACALs are informed of the resolution.

6.3 Types of Resolutions

The resolution of a petition is either an acceptance or rejection of the petitioner’s arguments. Acceptance can result in withdrawal of the test program from the ACATS, or a modification for conformity assessment. A test issue may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. If the challenge shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the ACAA will direct that the test(s) be processed with a test modification.
There are three types of test modification: Code, Processing, and Grading modifications.
All test modifications are documented in the ACATR.

6.4 Reconsideration of Rejected Petitions

A petitioner may resubmit a rejected petition, clearly stating additional information and reasoning as to why the original petition resolution is incorrect. The ACAA will resolve the resubmitted petition based on the deliberations of a body of Ada experts. A resolution of the resubmitted petition will be provided in no more than three weeks after submission.
A petitioner may resubmit a petition twice. A petitioner who has resubmitted a petition at least once may also request an extended resolution. In an extended resolution, the ACAA forwards the challenge to the ARG for resolution. (Extended resolution is not available for issues that have an interpretation approved in the last two years.) It is not anticipated that the ARG will resolve the issue in time for the conformity assessment that gave rise to it. Therefore, the tests involved in an extended resolution will be graded as Unsupported; they will not be graded as failures for the purpose of issuing a certificate of conformity.
However, the expiration date of the certificate shall be marked "pending issue resolution by ISO/WG9". The certificate shall expire on the day on which WG9 approves an interpretation of the Standard contradicting the petition and the processor will be removed from the Certified Processors List, or on its normal expiration date, whichever is sooner. The mark shall be removed from the Certified Processors List if WG9 approves an interpretation of the Standard confirming the petition.

6.5 Summary

There is no limit on the number of test programs that can be challenged by a petitioner. Although there is a risk that a petition will not be decided in a conformity assessment client’s favor, early submission of petitions can reduce the risk that a conformity assessment will not be successfully completed on schedule. Any interested party may challenge an ACATS test program.

7.EXTENSIBILITY OF CONFORMITY ASSESSMENT

As permitted by [ISO 99], the ACAA provides mechanisms for extending the certified status of a tested processor to an implementation class (a set of closely related processors operating on a range of compatible configurations). This section describes these certification extension mechanisms.

7.1 Implementation Classes

An Ada processor is typically designed to be used on any member of a set of host and target computer-system pairs; furthermore, a processor is usually provided with different modes of operation (also known as “options” or “switch settings"). In witness testing, a processor is tested under one mode of operation on a particular configuration (host-target pair). The particular processor that is tested may be viewed as representing an implementation class, consisting of a particular (binary) processor and any configuration (host-target pair) on which it operates and produces equivalent ACATS results. Related implementation classes may include processors that are maintained versions of the test processor, and processors for which the host system is different. The ACAA may extend the "certified conforming" status to entire implementation classes.
The tested processor may be viewed as representative of several related implementation classes. These classes are categorized and defined in the following subsections.

7.1.1 Base Implementation Class

A base implementation class includes a single (binary) processor that has achieved certified status through a complete conformity assessment (including witness testing). The processor may operate on multiple (closely related) configurations. The target instruction set architecture and target operating system of the additional configurations must be the same as or a superset of those of the witness tested processor. The host system must be able to execute the witness tested processor. The processor must have a mode in which it can produce ACATS results that are equivalent (see section 7.2) to those of the tested processor for each configuration in the class.

7.1.2 Maintained Implementation Class

A maintained implementation class is a class that includes a single (binary) processor that satisfies the following conditions:
The restriction of maintenance changes to corrective and perfective maintenance implies that the processor must have the same configuration(s) as the base processor class for the processor named in the ACAC.

7.1.3 Rehosted Implementation Class

A rehosted implementation class is a class that includes a single (binary) processor that satisfies the following conditions:
Adaptive maintenance may include limited changes to enable the processor to operate on a different host system from that of the processor named in the ACAC.
A rehosted implementation class may include closely related target systems. As with a base implementation class, the target instruction set architecture and target operating system of the additional target systems must be the same as or a superset of those of the processor named in the ACAC.

7.2 Equivalence of ACATS Results

The conditions for extending the certified status of a processor require that the candidate processor be capable of producing ACATS results that are equivalent to those produced by the certified processor and configuration. In this context, equivalent ACATS results are those satisfying the following conditions:
The ACAA must approve any deviation from the above requirements.

7.3 The ACATR Supplement

The purpose of the ACATR Supplement is to document the extension of certified status to an implementation class. See Appendix D for a sample ACATR. The Supplement contains the following information:

7.4 Requirements for Certification by Extension or Derivation

A client may request certification by extension for a base implementation class by submitting an ACATR Supplement (see section 7.3) to an ACAL. Similarly, a client may request certification by derivation for one or more implementation classes by submitting an ACATR Supplement to an ACAL.
The Ada Conformity Assessment Certificate (ACAC) referenced in an ACATR Supplement must have been issued within the five years previous to the date of submission of the supplement.
When submitting an ACATR Supplement (except as noted below), the client must certify that a representative processor and configuration was tested using a customized ACATS as described above, and that the results were equivalent as defined by section 7.2. The client should be prepared to substantiate this claim as requested by the ACAA or ACAL.
The receiving ACAL will check all test result differences indicated in the ACATR Supplement, checking that all such test programs have produced acceptable results according to the ACATS evaluation criteria. Any test issues it identifies shall be resolved as described for Self-Testing, see section 5.2.3.
Once any test issues identified have been resolved (possibly by modification of the supplement), the ACAL will append a summary of the test results differences to the supplement and then will submit the supplement to the ACAA for approval. On approval, the ACAA will create CPL entries identifying the implementation class as certified by extension or derivation. The ACAA will automatically reject a supplement with any unresolved test issues.
The ACATR Supplement for a certification by extension can be submitted at the same time as the Declaration of Conformity for a conformity assessment. In this case, the supplement does not need to include a certification of equivalent results since this is tested by the ACAL for the conformity assessment.
The ACAA will reject obviously unreasonable claims of compatible configurations, but will not do any in-depth analysis of such claims. Users should regard the claims as vendor claims of compatibility.

7.5 Expiration of Certification by Extension or Derivation

Certification by extension or derivation using the same test suite and modifications as the original certificate (ACAC) expires at the same time as the original ACAC. In contrast, certification by extension or derivation using the current test suite and modification expires two years after issuance. As with an ACAC, when certification by extension or derivation expires, the corresponding entry in the CPL is clearly identified as expired. Certificates expire in order to encourage periodic retesting of processors, which ensures that they continue to meet the requirements of conformity assessment.

7.6 Challenging Certification by Extension or Derivation

Any interested party may challenge any approved certification by extension or derivation. Such a challenge must include non-conforming output on a member configuration of the implementation class. If, after analysis by the ACAA and rebuttal by the client, the processor is found to violate the requirements of certification by extension or derivation, the certification will be removed or corrected.

7.6.1 Information Required to Challenge Certification

Anyone wishing to challenge an approved certification by extension or derivation, must provide the following information to the ACAA:
The ACAA will acknowledge receipt of the challenge. Note that deviations from the options or customized test suite used for the representative testing on which the certificate is based will greatly increase the chances of the challenge being rejected.

7.6.2 The Challenge Process

The ACAA will analyze all received challenges, drawing on the test reports for the original conformity assessments, ACATR Supplements, and other relevant materials. If the analysis shows that the challenge has merit, it will be forwarded to the original testing ACAL and to the client for rebuttal. The client will be allowed thirty (30) days to prepare a rebuttal to the challenge. Such a rebuttal should show why the behavior of the processor is conforming, or demonstrate that the processor does in fact conform when the tests are processed. The ACAA will rule on the challenge after either receiving the rebuttal or the expiration of the designated time. The ruling will be distributed to the ACALs, the client, and the challenger.
If the final ruling is that a challenge is upheld, the certificate by extension or derivation will be removed from the CPL or, by agreement between the client and the ACAA, modified to remove the offending configuration. If an upheld challenge demonstrates that the client fraudulently certified the testing of the representative processor and configuration in the ACATR Supplement, the client’s right to submit ACATR Supplements will be suspended for a period not less than six months.

APPENDIX A

POINTS OF CONTACT

Ada Resource Association

Ben Brosgol, President

Ada Resource Association
P.O. Box 4072
Oakton VA 22124
Email: brosgol@adacore.com

Ada Conformity Assessment Laboratories

Jean-Pierre Rosen
AdaLog
2 rue du Docteur Lombard
92441 Issy-les-Moulineaux Cedex
FRANCE
Tel: +33 1 45 29 21 52
FAX: 33 1 45 29 25 00
Email: info@adalog.fr
URL: http://www.adalog.fr

Ada Conformity Assessment Authority

Randall Brukardt
ACAA
P.O. Box 1512
Madison, WI 53701
Tel: 608-245-0375
FAX: 608-245-0379
Email: Agent@ada-auth.org
URL: http://www.ada-auth.org

Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG)

Jeffrey Cousins
BAE Systems Integrated System Technologies Limited
Warwick House, PO Box 87, Farnborough Aerospace Centre,
Farnborough, Hants, GU14 6YU, UK
Tel: +44 (0)20 8329 5430
Fax: +44 (0)20 8329 5001
Email: Jeff.Cousins@baesystems.com

Ada Conformity Assessment Test Suite (ACATS)

The ACATS is available to the general public from an ACAL; it is also available from the ACAA Internet site.
URL: http://www.ada-auth.org
The site includes downloadable versions of complete ACATS, the ACATS VCS for access to individual files and modifications, recent versions of the ACATS Modification List, and packaged versions of new and modified tests.
Questions concerning Ada conformity assessment or comments on ACATS test programs should be sent to the ACAA (see above).

APPENDIX B

TEST ISSUE FORMAT

[Part A]
Petitioner: <client name>
Configuration: <host / target hardware and operating systems>
ACATS Version: <ACATS version number>
Self-Test Submittal Date: <due date for self-testing results>

Part A will be completed once by each client; part B will be completed for each test issue. It is not necessary for a self-testing submittal date to have been established. Part A information is treated as confidential.


[Part B]
Reference: <test name (,test name)>
Summary: <brief description of the test issue>
Discussion: <detailed description of the test issue>

In this Discussion, arguments should be specified using test line numbers and references to pertinent sections of the Ada standard, Technical Corrigendum, or Commentaries (AI-xxxx). The petitioner must describe the behavior of the implementation for the test or tests that are challenged, stating the particular test messages produced. The detailed description can be limited to the particular segment of test code that is challenged. Relevant source code with processor messages should be included. (For a group of tests that cause essentially the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.)
If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified in the Discussion. It is especially important that implementations that fail to pass some test due to capacity limitations be described in enough detail for the ACAA to assess the reasonableness of these limitations.
Failure to fully specify the points pertinent to a test issue might result in an adverse decision, with the petitioner having to argue the case further with a second submittal to the ACAA. It is also possible that the Summary will suffice to adequately present a test issue.

APPENDIX C

DECLARATION OF CONFORMITY

Declaration of Conformity

Identification

Client: <client organization name>
Certificate Awardee: <if different from client>
ACAL: <name of Ada Conformity Assessment Laboratory>
ACATS Version: <version number of ACATS>

Ada Processor and Configuration

Processor: <name and version number of Ada processor>
Host Computer System: <host hardware and operating system>
Target Computer System: <target hardware and operating system>


Declaration:
I, the undersigned, representing the Client, declare that the Client knows of no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:2012) in the Ada processor above. <The next sentence should normally be deleted.> The Conformity Assessment Test Report and Ada Conformity Assessment Certificate associated with this effort are not to be made public without the Client's permission.



________________________________________ ________________
  <Name>       Date
  <Title>
  <Client Organization>

<Omit the remainder if the certificate awardee is the client>

Declaration:
I, the undersigned, representing the Certificate Awardee, declare that the Certificate Awardee knows of no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:2012) in the Ada processor above.



_______________________________________ ________________
  <Name>       Date
  <Title>
  <Organization>

APPENDIX D

ACATR SUPPLEMENT FORMAT

Supplement to ACATR <Reference ACATR number>
<Submittal Date>
name of client organization> (the client) hereby requests that the Ada Conformity Assessment Authority (ACAA) extend the certified status documented in the above-referenced Ada Conformity Assessment Test Report (ACATR) and in Ada Conformity Assessment Certificate (ACAC) <Reference ACAC number> to the implementation class(es) described in the following pages.

Technical Contact:
<Name>
<Address>
<E-Mail>



Approval: ________________________________ ________________
        {name}     Date
Ada Conformity Assessment Authority

Implementation Class Information

<Complete the remaining pages for each processor class>
Implementation Class Category: <Base, Maintenance, or Rehosted>
Processor Identification: <Name, Version, and Release identification>
Host Systems:
<Description of host computer models and operating systems. Ranges may be used.>
Target Systems:
<Describe as above or use "Same as host" or "Any host">
Representative Processor and Configuration Tested:
Host System:
<Identification of specific host system (hardware and operating system) of tested processor>
Target System:
<Identification of specific target system (hardware and operating system) of tested processor. "Same as host" is acceptable.>
ACATS Version Used for Testing Representative Processor:
<Version and ACATS VCS label. This version and label must either be the version and label used for witness testing the processor tested in the original ACATR, or the current version and label as of the date of submission. >
Client Certification of Testing and Processor Derivation:
I, the undersigned, representing the Client, certify that the above identified representative processor was tested on the described configuration with the customized ACATS version described above, including the code modifications and implementation-defined substitution values that were used in the conformity assessment leading to the certificate named in this Supplement, with modifications described in this Supplement, and that the testing results were the same as those obtained in that conformity assessment, with exceptions as described in this Supplement. I further declare that the Client knows of no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:2012) in the identified representative processor above. I further certify that the above-identified representative processor and configuration meets the definition of base, maintained, or rehosted implementation (as described in the Operating Procedures for Ada Conformity Assessment).



________________________________ ________________
<Name> - <Client>           Date


<For maintenance and rehosted implementation classes>
Maintenance Changes:
<Include a brief description of the significant changes in the compiler in deriving it from the compiler named in the ACAC. Include the type of maintenance (Corrective, Perfective, or Adaptive) for each change noted.>


ACATS Modifications:
<Describe each change in the ACATS used in testing the reference processor, as compared to the customized ACATS used in testing the processor named in the certificate. Justification is required for any change other than the use of different implementation-dependent substitution values and the use of the current ACATS version. For tests different solely because of the use of the current ACATS version, a list of the test names is sufficient.>
Test Results Differences:
<Describe and justify each difference between test results produced in testing the reference processor as compared to those produced in testing the processor named in the certificate. For tests whose results differ solely because of the use of the current ACATS version, a list of the test names is sufficient. The actual results for those tests must be submitted to the ACAL along with this supplement in a format acceptable to the ACAL. The ACAL will attach a summary report of those results here.>

APPENDIX E

ACRONYMS

This document contains a number of acronyms, whose meanings are given in the following list:
ACAA
Ada Conformity Assessment Authority
ACAC
Ada Conformity Assessment Certificate
ACAL
Ada Conformity Assessment Laboratory
ACATR
Ada Conformity Assessment Test Report
ACATS
Ada Conformity Assessment Test Suite
AJPO
Ada Joint Program Office
ANSI
American National Standards Institute
ARA
Ada Resource Association
ARG
Ada Rapporteur Group
CPL
Certified Processor List
DoC
Declaration of Conformity
DoD
Department of Defense
IEEE
Institute of Electrical and Electronic Engineers
ISO
International Organization for Standardization
SNA
Specialized Needs Annex
WG9
Working Group 9 (of ISO/IEC JTC1/SC22)

APPENDIX F

REFERENCES

This document references the following publications:
[ACATS UG]
ACATS User's Guide, version 3.0, December 2007. This document is updated periodically; references in this document refer to the most recent version unless otherwise noted.
[Ada2012]
ANSI/ISO/IEC 8652:2012 Ada 2012 Reference Manual, December 2012 (supersedes [Ada95]).
[Ada95]
ANSI/ISO/IEC 8652:1995 Ada 95 Reference Manual, January 1995 (supersedes [Ada83]).
[Ada83]
American National Standards Institute and United States Department of Defense: ANSI/MIL-STD-1815A Reference Manual for The Ada Programming Language, 1983 Note: This standard is identical with ISO/8652:1987.
[Amd1]
ISO/IEC 8652:1995/AMD 1:2007 Programming languages -- Ada Amendment 1 (usually called Ada 2005)
[ANSI/IEEE 90]
American National Standards Institute / Institute of Electrical and Electronic Engineers, Inc., Standard 610.12-1990; “ANSI/IEEE Standard Glossary of Software Engineering Terminology”.
[ISO 74]
International Standards Organization: ISO 2382/I-1974 Data Processing - Vocabulary - Section 01: Fundamental Terms.
[ISO/IEC 91]
International Standards Organization: ISO/IEC, Guide 2, 6th edition 1991 - General Terms and Their Definitions Concerning Standardization and Related Activities.
[ISO 99]
ISO/IEC 18009:1999, Information Technology -- Programming Languages -- Ada: Conformity Assessment of a Language Processor
[TC1]
ISO/IEC 8652:1995/COR.1:2001 Programming languages -- Ada Technical Corrigendum 1