Ada Resource Association
Table of Contents
The International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), through Working Group 9 (WG9) of Subcommittee 22 (SC22) of their Joint Technical Committee 1 (JTC1), are establishing an International Standard entitled Ada: Conformity Assessment a Language Processor (FCD 18009), referenced in this document as [ISO 99]. This Final Committee Draft specifies requirements for five aspects of a conformity assessment program, as follows:
The current document represents the Ada Resource Association (ARA)s establishment of an ACAA governed by the procedures (ACAP) given herein. The primary goals of this procedures document are as follows:
The following paragraphs summarize the conformity assessment system by using the language of the International Standard [ISO 99] to describe the AJPO validation system.
Ada Conformity Assessment Test Suite
The ACAA designates the Ada Compiler Validation Capability (ACVC) as the ACATS. The ACVC is a conformity test suite (and supporting documents) developed under contract to the United States Government and made available for public use; it is designed to ensure that Ada processors achieve a high degree of conformity to the Ada language standard. A processor is tested when operating in a specific hardware and software configuration. The ACATS is customized by a testing laboratory (ACAL) for each processor that is subjected to conformity assessment; customization consists of adjusting the ACATS appropriately for various implementation characteristics. The ACATS is maintained by the ACAA; new releases of the ACATS are defined by changes resulting from the discovery of deficiencies in the test programs.
For an Ada processor to successfully complete conformity assessment, it must process each test program (for the "core" language) of a customized ACATS so that the result is graded passed, inapplicable, or unsupported by ACATS grading criteria. A testing laboratory customizes the test suite for a particular processor by appropriately setting test parameters, by removing "withdrawn" tests (tests ruled by the ACAA to be in error) and certain inapplicable tests, by splitting as needed test files with multiple intended errors so to enable complete error detection, by using any other modified tests as directed by the ACAA, and by including each optional set of tests (see below) as requested by the client.
In addition to the specification of a "core" language, [Ada95] contains several Specialized Needs Annexes (SNAs); these specify language requirements designed to meet the particular needs of various general application domains, such as information-systems programming. A processor for [Ada95] need not include implementation of any of these annexes, or it may implement only some of the features of these annexes. Whereas all ACATS test programs for the core language must be processed during conformity assessment, those for the SNAs are processed only upon client request. A conformity assessment is judged successful (leading to the issuance of an ACAC) only if all tests for the core are correctly processed; the certificate will additionally give credit for support of a SNA to the extent that the relevant set of tests is correctly processed.
Conformity assessment involves interaction between the ACAL client and both the ACAL and the ACAA. The assessment process consists of well defined actions which, when completed successfully, result in the award of an ACAC for the tested processor. The key actions in the conformity assessment of an Ada processor are:
1. The client and an ACAL reach a formal agreement for conformity assessment, including the dates for the submittal of the results of client-administered processing of the ACATS and for ACAL witness testing.
2. The client petitions for deviation from the requirements of each ACATS test program that is believed to be wrong for the candidate implementation(s).
3. The ACAA rules on the client petitions.
4. The client processes the ACATS on the candidate processor(s) and submits the results to the ACAL.
5. The ACAL analyzes the results of the clients independent processing of the ACATS. (If the results are not acceptable, the previous action must be repeated, and new results analyzed.)
6. The ACAL conducts witness testing of the candidate processor(s), documenting this testing in an Ada Conformity Assessment Test Report (ACATR) which the ACAL submits to the ACAA for review.
7. The client signs a Declaration of Conformity for each candidate processor.
8. The ACAA reviews the ACATR, with comments to the ACAL. The ACAA recommends to the ACAL that a conformity assessment certificate be issued for the tested processor(s), if the testing is successful.
9. The ACAL issues a conformity assessment certificate for each tested processor upon the successful completion of the preceding actions.
Hence, successful testing of an Ada processor concludes with the ACAL's awarding a conformity assessment certificate for that processor (working in a specific configuration) to the client. This conformity assessment certificate attests that the processor has been subjected to an Ada conformity assessment and that no evidence of non-conformity was found. The processor is said to be "certified as conforming," as described in [ISO 99], clause 2.1. The processor will be listed in the ARA's Certified Processors List (CPL). The client may perform maintenance on the processor and may claim conformity for such derived versions in accordance with the ACAA procedures, so long as the client ensures that they produce the same ACATS results as are documented in the ACATR. This maintenance may even include adaptive maintenance that enables the processor to run on entirely different host computers (i.e., re-hosting) or to target closely related target computers. The ACAA provides a means for listing derived processors in the CPL.
This document provides operating procedures of the Ada Conformity Assessment Authority (ACAA). This body is a part of an organization that meets the requirements for assessing conformity of an Ada language processor, as given in [ISO 99]. The other bodies making up this organization are the Ada Conformity Assessment Laboratories (ACAL), which perform the actual conformity assessments using the Ada Conformity Assessment Test Suite (ACATS). The end product of a successful conformity assessment is an Ada Conformity Assessment Certificate (ACAC), indicating that a particular Ada language processor is "certified as conforming," as defined in [ISO 99]. This document forms an Ada Conformity Assessment Procedure (ACAP), as defined in [ISO 99]. Detailed procedures regarding the application of the ACATS are given in the User's Guide.
The United States Department of Defense (DoD)sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. The AJPO established a certification system to realize the benefits of standardization which include the ability to transfer software and programming expertise between computer systems that use a conforming Ada processor. When the AJPO ceased its operation of the certification system, the Ada Validation Facilities agreed to act as ACALs under the provisions of the emerging International Standard (now [ISO 99]). The Ada Resource Association, in cooperation with the Ada Joint Program Office, facilitated the identification and U.S. Government funding for a candidate ACAA and produced this document defining its operating procedures. The ACALs then agreed to designate the identified organization as the ACAA and to incorporate these procedures into the ACAP.
It is important to note the scope and intent of conformity assessment. The purpose of conformity assessment is to ensure that Ada processors achieve a high degree of conformity with the standard [Ada95]. Characteristics such as performance and suitability for a particular application are not specified by the standard, and thus are outside the scope of Ada conformity assessment. Moreover, it is important to note that the ACATS is a set of test programs intended to check broadly for correct implementation; it is not possible to exhaustively test for conformity. Thus, conformity is checked only to the extent of these tests; processors that are certified as conforming may fail to conform to the standard in ways peculiar to each, under particular circumstances.
Witness testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of Ada conformity assessment is to identify processors that may be procured and used to develop application programs that meet the [Ada95] goals of portability and interoperability. The ACATS (test suite) is not designed to replace the clients quality assurance testing or systematically to detect inconsistencies or "bugs", but to verify that the tested processor correctly supports all required features. Rather than exhaustive testing of permutations of features, the test suite contains a carefully-chosen set of test cases that cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard. Neither is conformity assessment intended as a means of performance benchmarking. The Ada Conformity Assessment Test Report (ACATR) which documents the witness testing does not contain information about the speed, cost, or efficiency of executing the conformity assessment tests.
Ada: The programming language defined by [Ada95].
Ada Conformity Assessment: Ada Conformity Assessment Authority (ACAA): The organization which provides the technical and administrative oversight of Ada Conformance assessment activities.
Ada Conformity Assessment Certificate (ACAC): A certificate issued by authority of the ACAA and an ACAL for a successfully tested Ada processor (see Section 5.4).
Ada Conformity Assessment Laboratory (ACAL): An independent testing laboratory performing Ada conformity assessments
. Ada Conformity Assessment Test Report (ACATR): A report produced by an ACAL that documents the witness testing of an Ada processor (see Section 5.4).
Ada Conformity Assessment Test Suite (ACATS): The means for testing conformity of Ada processors, consisting of the test suite, the support programs, and the User's Guide. The ACATS is also known as the Ada Compiler Validation Capability (ACVC).
Ada Conformity Assessment Test Suite Modification List (ACATS Modification List): A listing of modifications to the ACATS necessitated by flaws found in the suite. This list is periodically issued by the ACAA (see Section 4.6.1).
Ada Conformity Assessment Test Suite Version Control System (ACATS VCS): A publicly accessible program which stores the current version of the ACATS, as well as any modified tests used for conformity assessments (see Section 4.5.1).
Ada processor: A processor for the Ada programming language as defined in [Ada95].
Ada Rapporteur Group (ARG): A subgroup of ISO/IEC/JTC1/SC22/WG9, the International Organization for Standardization Working Group for Ada. Members of the ARG are appointed by the convener of the ISO working group for the purpose of resolving issues with respect to the interpretation of the Ada programming language.
Adaptive Maintenance: Maintenance performed to adapt a product to a changed environment. Base implementation class: An implementation class in which the processor has been awarded certified status through testing by an ACAL (see Section 7.3.1).
Certified Processors List (CPL): A publicly available list of processors to which the ACAA has granted certified status. The CPL is maintained by the ACAA.
Certified status: (Also "certified as conforming") The status granted to an Ada processor by the award of an ACAC (see Section 5.4).
Client: An individual or corporate entity who has an agreement with an ACAL that specifies the terms and conditions for ACAL services (of any kind) to be performed. (Also used to refer to an organization that intends to make such an agreement.)
Computer system: A system containing one or more computers and associated software. [ANSI/IEEE 90] In this document, a computer system comprises the hardware and software (operating systems, kernels) that are essential to the operation of the processor or the compiled code (e.g., I/O devices are generally not included in this definition).
Configuration: A specific host and target computer system. "Configuration" is usually used along with "processor" to completely specify a conformity assessment.
Configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specific requirements. [ANSI/IEEE 90]
Conformity: Fulfillment by a product, process or service of all requirements specified. [ISO/IEC 86] See also Subclause 1.1.3 of [Ada95].
Core language: The Sections 1-13 and Annexes A, B and J of [Ada95].
Corrective maintenance: Maintenance performed to correct faults.
Customized test suite: The ACATS tests, adjusted as necessary, that must be used for witness testing of a given Ada processor (see Section 4.3).
Declaration of Conformity: A formal statement from a client assuring that conformity is realized on the Ada processor for which conformity assessment status is requested (see Section 5.2).
Equivalence (of ACATS results): A set R of ACATS results is equivalent to a previously evaluated set of results, P, provided that (1) the ACATS used in producing R is the same as the ACATS used in producing P, with the possible exception of using different implementation-specific substitution values; and (2) application of the ACATS grading rules results satisfies all the following: (a) every test graded as passed in P is also graded as passed in R; (b) every test graded as inapplicable in P is graded as passed or inapplicable in R; and (c) every test graded as unsupported in P is graded as passed, inapplicable, or unsupported in P (see Section 7.2).
Host computer system: The computer system on which a processor is installed and executes.
Implementation: A processor running on a particular configuration.
Implementation class: A collection of implementations that are sufficiently closely related that the certified status of one member of the collection may be extended to the other members, provided that certain conditions are met (see Section 7.1).
Maintained implementation class: An implementation class in which the processor is derived (by applying corrective and perfective maintenance changes) directly from the processor that has been awarded certified status by ACAL testing, and in which the processor has a mode of operation in which it can produce ACATS results that are equivalent to those of the processor named in the ACAC (see Section 7.3.2).
Operating system: A collection of software, firmware, and hardware elements that controls the execution of computer programs and provides such services as computer resource allocation, job control, input/output control, and file management in a computer system. [ANSI/IEEE 90]
Perfective maintenance: Maintenance performed to improve performance or maintainability. [ANSI/IEEE 90]
Processor: A compiler, translator, or interpreter. The processor includes all tools used in creating programs. For instance, many systems will include a linker in the processor. A processor works in conjunction with, but does not include, a configuration. In this document, processor typically means Ada processor.
Rehosted implementation class: An implementation class in which the processor is derived directly (by applying corrective, perfective and adaptive maintenance changes) from the processor that was awarded certified status by ACAL testing; in which the common target computer system is the same as that of the certified processor; and in which the processor has a mode of operation in which it can produce ACATS results that are equivalent to those of the certified processor (see Section 7.3.3).
Self-testing: The process of producing the results of processing an appropriately customized test suite by the client (see Section 5.2).
Software maintenance: Modification of a software product after delivery to correct faults, to improve performance, or to adapt the product to a changed environment. [ANSI/IEEE 90]
Specialized Needs Annexes: Annexes C through H of [Ada95]. These Annexes define standards for additional functionality required by specific application areas. An Ada processor may support some or none of these annexes.
Target computer system: The computer system on which the executable code generated by a processor is loaded and executes.
Test issue: (Also "dispute") Any problem arising during conformity assessment (see Section 6).
Validated: Equivalent to "certified status". The status granted to an Ada processor by the award of an ACAC (see Section 5.4). We define this equivalent term to correspond to common usage in the Ada community.
This section specifies the roles of the bodies that are responsible for Ada conformity assessment of clients who receive service from them.
An ACAL is an independent testing laboratory that performs Ada conformity assessment activities. [ISO 99] includes a list of requirements that a testing laboratory must meet in order to be considered an ACAL. These requirements will not be repeated here. The ACAL operates under an ACAP consisting of its own operating procedures and the procedures defined in this document. An ACAL performs the following principal functions:
The ACAA ensures world-wide commonality of the Ada Conformity Assessment Process. The technical and administrative functions of the ACAA are carried out by a technical agent. It is established by a sponsor and is advised by an Advisory Board.
The ACAA is sponsored by the Ada Resource Association, a trade association of Ada product suppliers. The sponsor is responsible for the following:
ACAA technical agent supports and coordinates the activities of the ACALs by:
The ACAA Advisory Board represents the interests of the wider Ada community in the Ada Conformity Assessment process. Issues of policy and procedures are brought to the attention of the Board, which may make recommendations as to their resolution. Board members are appointed by the ACAA sponsor, and include (but are not limited to) the following:
A client is an individual or organization that contracts with an ACAL for conformity assessment services. Clients are required to provide accurate and complete information as specified in these procedures and the procedures of the ACAL.
The designated ACATS is the suite of conformity tests, support software, and documentation known as the Ada Compiler Validation Capability (ACVC). The ACVC was developed under various contracts with the United States Department of Defense. It is designed to demonstrate the conformity of an Ada processor with the standard [Ada95]. The use of the ACATS is documented in the ACVC Users Guide, which explains the criteria for evaluating the results of the individual tests. While the ACVC was produced under contract to the United States Government, it is available to any individual or organization. The ACAA controls the content of the ACATS as it is used in conformity assessments. Questions concerning Ada conformity assessment or comments on ACATS test programs should be submitted to the ACAA (see Appendix A, Points of Contact).
Each ACATS test program has one or more test objectives which are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada processor (for example, "check that the proper exception is raised when FloatMachine_Overflows is True"). These test programs generally contain an explicit indication of their applicability and the expected behavior of processors for which they do not apply. The determination of applicability is made according to the grading criteria in the User's Guide or in the internal test documentation, or as a ruling by the ACAA. For a processor to be certified as conforming, all applicable test programs for the core language (as defined in [Ada95]) must be processed and passed according to the specified grading criteria.
Reference [Ada95] includes certain sections designated as Specialized Needs Annexes (SNA). The set of ACATS test programs for any of the SNAs will be processed only upon client request (to demonstrate full or partial support of the Annex). As permitted by [Ada95], test programs for the SNAs may be rejected at compile time or may exhibit run-time behavior that indicates a lack of support for requirements that only apply to SNAs. The ACAA may rule that tests producing such behavior are graded as "Unsupported". If the ACAA finds that the behavior is not in accordance with the permission granted by [Ada95], then the tests are graded as "Failed". Tests graded as Unsupported are reported in the ACATR and the ACAC, but these results do not affect the designation of the processor as being certified as conforming. On the other hand, tests graded as Failed are evidence of non-conformity, precluding the issuance of an ACAC for the candidate processor.
The various ACALs and the ACAA strive to apply the ACATS as uniformly as practical to all Ada processors. In order to apply common test objectives that depend on implementation-dependent characteristics (e.g., line lengths and numeric types), some test programs must be adjusted to a given implementation following the procedures given in the User's Guide. These adjustments consist of the insertion of implementation-dependent values in prescribed places in certain test programs.
In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (for example, the algorithm for recovering from syntax errors). The allowable changes for each Ada processor are determined by the User's Guide and the ACAA, and may require ACAL assistance especially in the case of processor error-recovery problems.
In order to meet a test objective, it may be required to modify the code, the processing method, or the grading of a test program. Only the ACAA shall make the decision to use any of these modifications, as described below:
The ACATS is customized by the ACAL for each processor that is subject to witness testing. This customization always includes making the implementation-dependent substitutions. It may also include making code modifications that the ACAA directs for that specific conformity assessment as well as removal of some inapplicable test programs as allowed by the User's Guide.
The result of processing an ACATS test program can be given only one of four possible grades: Passed, Inapplicable, Unsupported, and Failed. The first three grades are considered to constitute acceptable results. ACATS test programs that contain illegalities (which an implementation must detect) generate diagnostic output which must be inspected manually or by pattern-matching algorithms, matching system diagnostics to the intended errors. Executable ACATS test programs generate output using report procedures, which can be graded automatically. The ACATS report package, Report, contains specific output procedures for the two grades Failed and Inapplicable. If neither of these is invoked, the Report.Result procedure will report Passed or Tentatively Passed (indicating that the test has passed if manual inspection reveals that specific additional requirements are met). These results are the only ones that are generated by the test code (if no result is reported, that is, if the test completes abnormally, the result is graded Failed). The grade Unsupported is established as a means of grading tests that apply to the Specialized Needs Annexes (SNAs), as explained below.
The ACATS test programs for the SNAs pose two problems for using the three conventional grades of Passed, Inapplicable, and Failed. The broad problem is that full support of any such Annex is not required for conformity to [Ada95] there may be no support, or merely partial support. Unfortunately, there is no way to discriminate between full and partial support if only those three grades are used, since the grades Inapplicable and Failed are not appropriate for this (an implementation is not allowed to provide deviant semantics for an unsupported Annex feature that would be a conformity assessment failure). The second problem is that there are some test programs for Core features that are applicable also to a SNA, in particular, the test programs for representation items. These programs constitute tests for features that are defined in the Core as optional, but are mandatory for full support of the Systems Programming Annex (which itself is mandatory for full support of the Real-Time Systems Annex).
Therefore, the ACAL grades the result of processing such an ACATS test program (i.e., one that uses a feature required by, or defined in, an Annex) as Unsupported, if the prima facie result is failure but the implementation's processing of the test program is an acceptable form of non-support. For example, if a processor does not support a particular form of a representation clause, it must reject any test program that uses it such rejection of an executable test is usually graded Failed, but is graded Unsupported if the implementation does not claim support of the relevant SNA. A processor that accepts the representation clause and reports Failed, on the other hand, is deemed to have failed the test regardless of any claim (or lack thereof) of support for the SNA.
The current baseline version of the ACATS is available to the general public from an ACAL or from an Internet site supported by the ARA. The current and any previous versions of the ACATS Modification List are available from the same sources. ACALs may assist the client in format conversion when providing the ACATS in a particular distribution medium. If a client has a need for a superseded version of the ACATS, it may be available from the ACAA or from an ACAL. See APPENDIX A for points of contact.
The ACAA maintains an official ACATS web site, accessible via the Internet. The web site includes a web-accessible version control system, which contains the official version of the ACATS. Both old and new versions of tests are accessible given the test name and version label desired.
Instructions for using the ACATS version control system are available on the web page.
Test challenges and ARG interpretations may reveal flaws in the ACATS. The ACAA may remove, repair, and insert tests in response to these needs. These test suite changes are listed on the Ada Conformity Assessment Test Suite Modification List (ACATS Modification List, or AML for short). This list is issued as needed by the ACAA, and contains information about affected tests, and how to acquire new and modified tests. The official versions of the tests (including modified and new tests) are available via the official ARA web site and other sources (see Section 4.5)
New versions of the ACATS Modification List will be issued when test suite changes are needed. The list contains at least the following information:
The official version of a test always is available from the ACATS VCS. Information in the list is advisory only; in the case of a conflict between the ACATS VCS, and the list, the ACATS VCS is assumed correct.
The ACAA may issue various kinds of test suite modifications. Each affected test has one of the following categories:
The ACAA baselines the ACATS approximately once per year. When the ACATS is baselined, the original official set of files are updated with all of the changes specified in the ACATS Modification List. Allowed modification and Pending New tests are not included (these have not been available long enough to include). Additionally, the documentation associated with the test suite is updated. Future versions of the ACATS Modification List are then based on the new test version. Any Allowed Modification and Pending New tests will be listed in the initial version of the ACATS Modification List for the new baseline version.
The effective date of a baseline version will be announced at least 3 months prior to it being effective, and should be at the beginning of a quarter (that is, January 1st, April 1st, July 1st, or October 1st). The documentation will be available not less than 30 days prior to it being effective. Conformity assessments started after the effective date must use the new baseline test suite. (Note that the tests that make up the baseline version are known on the date that the effective date is announced, and are accessible on the web site, even before the final version of the baseline documentation is available.)
The tests used for a particular conformity assessment are defined as the tests from a particular baseline version of the ACATS, modified as follows:
Each ACATR identifies the baseline version of the ACATS, and documents all modifications made to that baseline version of the ACATS.
Conformity assessments must use the most recent version of the ACATS Modification List at the start of witness testing. (Note that the set of required tests is always known at least 3 months in advance.)Most tests and support files modified for an individual conformity assessment are included in the ACATS Version Control System. Tests modified only by making implementation-dependent substitutions (typically by using a tool) will not be included. B-Test splits allowed by section 4.2 and tests modified only by splitting between compilation units will also not be included.
The files will be posted on the ACATS Version Control System before the ACATR is issued. Each conformity assessment has a unique version label, which can be used on the web site to access all of the files that differ from the baseline versions. The version label is included in the ACATR. It is intended that the information in the ACATR and the files available on the official ACATS Version Control System will allow users to reproduce the conformity assessment on their own.
There are a number of steps that must be completed by a client, the ACAL, and the ACAA so that the client obtains a conformity assessment certificate and an ACATR. The same ACATS version, including the application of the requirements of the ACATS Modification List, must be used to complete the steps described in this section. Step 3, Witness Testing, must begin before the current ACATS version expires or else no certificate will be issued. Anyone intending to obtain a conformity assessment certificate should contact an ACAL without delay for advice on the handling of the ACATS, on interpretation of the test grading criteria, and on the operational procedures of that ACAL.
The required steps follow:
1. Establishment of Agreement
In order to obtain conformity assessment services, an interested party must become a client of an ACAL by reaching a formal agreement. This agreement addresses the following topics:
The schedule for events, deliverables, and payments should take into account the fact that certain steps in the conformity assessment process require interaction with the ACAA. The ACAA and ACAL will keep confidential a clients intent to obtain a conformity assessment certificate and the projected schedule for conformity assessment. If the client requests more restrictive confidentiality conditions for reasons of national security or procurement sensitivity, the client will provide to the ACAL an official, written statement describing the request and the reason(s) for the request; the ACAL will also obtain further guidance from the ACAA.
Self-test evaluation entails a series of actions and is usually where the bulk of the conformity assessment effort is expended. These actions are described in the following subsections.
After entering into a formal agreement, the client obtains a customized test suite from the ACAL. (At the clients risk, the client may prepare this customized test suite according to instructions in the User's Guide, rather than obtaining it from an ACAL). The client then processes all the tests in this customized test suite using the candidate processor on the candidate configuration or on another configuration that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of options controlling the production of information output. (For example, options which control the format of listings, the format of error messages, and the generation of listings may be varied.) Any other exception constitutes a test issue that must be resolved with the ACAL (see Section 5.2.3). Test issues should be sent to the ACAL for analysis as soon as possible.
Self-test activities include as a minimum the processing of an appropriately customized test suite by the client, preparation of a client supplied Declaration of Conformity, and submission of any test issues.
Upon completion of self-testing, the client delivers the complete set of results in the agreed format to the ACAL. (See Section 5.2.4 for an alternative to submission of complete results.
Results are accompanied by the following information:
The Declaration of Conformity states that the organization responsible for the production, maintenance or distribution of the Ada processor is offering a product that is in conformity with [Ada95]. The client must ensure that the information contained in the Declaration of Conformity does not infringe on the rights of a third party, and may be required to provide a written statement of consent from any third party involved. The Declaration of Conformity becomes part of the ACAL records and is copied into the ACATR. The ACAL will not issue a certificate until the ACAA has reviewed a signed Declaration of Conformity. (See Appendix C for an example of the Declaration of Conformity.)
The ACAL analyzes the client's submitted results of self-testing, checking that all test programs have produced acceptable results according to the ACATS evaluation criteria. During this analysis period, any test issues are resolved.
A test issue is defined to be any of the following:
A client may challenge an ACATS test program's correctness or applicability to a particular implementation. Such challenges should be presented to the ACAL in the petition format given in Appendix B. The ACAL will forward any petitions to the ACAA for resolution; the ACAA will strive to rule on the petition within two weeks of receiving it. The ACAA reports all challenges and rulings to each ACAL. However, an ACAL may not apply an ACAA ruling for one conformity assessment to another conformity assessment without the ACAA so directing. (See Section 6 for a description of the Challenge and Resolution Process.)
In some cases, it may be agreed to leave a test issue until witness testing. For example, it might be impossible to check the processing of control characters by inspecting printed results. The ACAL will note any unresolved issues and describe the results that are expected during witness testing. It is also possible that the client information for the production of the customized test suite (see Section 5.2.2) was insufficient, so that corrections to the customized test suite must be made, requiring additional processing.
The ACAL and the client may agree that, at the clients risk, parts of the customized test suite need not be processed during self-testing. There are two typical situations, as follows:
The normal practice is to submit complete self-testing results for at least one of the implementations under test. The ACAL may require the submission of complete self-testing results.
Self-testing is successful if the analysis of results and the resolution of test issues show that all results are provided and are acceptable. Self-testing is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues is deferred until witness testing by agreement between the ACAL and the client.
Upon successful completion of self-testing, with or without caveats, the ACAL witnesses testing of the Ada processor in accordance with the formal agreement between the ACAL and client. Witness testing takes place in the physical presence of qualified ACAL personnel. The ACAL supplies a customized test suite that it has prepared based upon client information and any information collected during the resolution of test issues. The customized test suite will include the set of test programs for the core language and each set, as requested by the client, any (or none) of the Specialized Needs Annexes (SNAs). The ACAL verifies that the processor identification, including identification of the processor and configuration (hardware systems and operating systems), matches that given in the Declaration of Conformity. (If it does not agree, then the client must provide a new Declaration of Conformity.)
The ACAL observes the installation of the customized ACATS on the host computer system, monitors the processing of the customized ACATS on the host and target computer systems, and evaluates the results. The entire customized test suite should be run on a single copy of the Ada processor on a single configuration using a unique set of option settings of the processor. (Differences in options controlling the production of information output, and those from accepted test issues are allowed. See section 5.2.1) If the ACAL determines that the results agree with those obtained from self-testing and are satisfactory with respect to the caveats, the witness testing has been successful; otherwise, the test is unsuccessful. If any result of testing with a set of test programs for a Specialized Needs Annex is unacceptable, the test report and certificate of the conformity assessment will not recognize that the set was processed.
Each conformity assessment effort is documented by an ACATR, and each successful effort is further documented by an ACAC.
An ACATR is produced for each processor and configuration subjected to witness testing. Each ACATR contains, at a minimum, the following information:
The ACATR is prepared by the ACAL but includes material that is produced by the client, such as the documented processor options used during witness testing. A draft version of the ACATR, based on results and circumstances implied by the evaluation of self-testing results, is sent to the ACAA for review. The draft version is also submitted to the client for review during witness testing, and is updated to account for client comments and observations made during witness testing. For a successful conformity assessment, the final version of the ACATR is signed by the ACAL and the ACAA. For an unsuccessful conformity assessment, the final ACATR is provided to the client only. Final test reports will never be modified. If it becomes necessary to correct a final ACATR, a separate document entitled "Supplement to Ada Conformity Assessment Test Report <unique report identifier>" will be prepared. Such a supplement will meet the applicable requirements of section 5.4.1.
The final version of the ACATR for a successful conformity assessment is available to the general public from the client, from the ACAL that produced it, and from the ACAA in electronic form. The ACAL may require payment of a fee for ACATR reproduction and delivery. (See Appendix A for points of contact.) By including an appropriate request on the Declaration of Conformity (Appendix C), the client may disallow public availability of the ACATR and the ACAC.
With the concurrence of the ACAA, the ACAL issues an Ada Conformity Assessment Certificate (ACAC) for each processor and configuration that was subject to successful witness testing. The information on the certificate is derived from the client's Declaration of Conformity and the ACATR. The ACAC conveys to the processor and configuration the status of certified as conforming, as defined in [ISO 99]. An entry is made in the CPL for each ACAC, unless the client has requested confidentiality on the Declaration of Conformity (see Appendix C).
The ACAC contains the following information:
Note that an ACAC attests that testing was performed on a specific processor using a specific test suite running on a specific configuration, following the Ada Conformity Assessment Procedure, and that no evidence of non-conformity was detected. It does not certify that the processor is free of defects, nor does it certify that the processor is usable for any particular purpose.
ACACs expire two years after issuance. When a ACAC expires, the corresponding entry in the CPL is removed, along with any entries for derived processors (see section 7.4). Certificates expire in order to encourage periodic retesting of processors, which ensures that they continue to meet the requirements of conformity assessment.
Versions For some special procurement requirements, a client might wish to have witness testing done with an obsolete version of the ACATS. The ACAP does not include any procedures for recognizing testing with obsolete test suite versions, but the ACALs may provide such a service outside the system. Ada Conformity Assessment Certificates will not be issued for testing with obsolete test suites, nor will CPL entries be created based on such testing.
The ACAA retains a copy of each ACATR (which includes a copy of the Declaration of Conformity and the ACAC), records pertaining to issues and their resolution, and a copy of each registration request. The ACAL retains a copy of each ACATR, a copy of the customized ACATS used in witness testing, and a copy of the witness testing results. The ACAA retains its records until at least three years following expiration of the ACATS version used in testing. Each ACAL's procedures specify the length of time that its records are retained, but records must be retained at least three years after the completion of witness testing.
The client must agree not to advertise or make public claims that the Ada processor is certified as conforming until after receiving the ACAC or receiving formal notification from the ACAL that it has issued an ACAC. A waiver of confidentiality must be signed by a client who intends to advertise the completion of events that indicate progress toward completion of conformity assessment. If a waiver of confidentiality has been signed with the ACAL, the ACAL will respond to inquiries about the clients advertisements or public claims by acknowledging receipt of conformity assessment materials (i.e., a formal agreement, self-testing results, or witness testing results) without judgment concerning the success of the witness testing.
This section presents the process whereby tests may be challenged, possibly resulting in their modification or withdrawal.
A "deviation" is defined by the User's Guide as any result from processing an ACATS test program that is not a passed or inapplicable result according to the established grading criteria. This intentionally broad definition of a "deviation" is intended to ensure that processor implementers bring all deviant test results to the attention of the ACAA or ACAL, without assuming that such results are acceptable. In petitioning for acceptance of a deviation, the petitioner provides a rationale for each challenge made against a test program. Petitions are sent to the ACAA, usually electronically, by the petitioner or by an ACAL on behalf of its client. For each deviation that is accepted (i.e., when the ACAA rules in favor of the petition), generally some correction is indicated for the cited tests. The ACAA may withdraw a test program or require that a modified version of the test be processed (see section 6.4). Withdrawal of a test program or the provision of a modified version of a test results in the release of a new version of the ACATS Modification List.
The ACAA typically resolves challenges by any of three methods:
1. a resolution that was made previously is applied to the current petition (e.g., the same petition might be submitted at different times by different petitioners);
2. the resolution can be determined unequivocally based on the Ada standard or Ada Commentaries; or 3. the resolution is based on the deliberations of a body of Ada experts. Although these procedures do not set a limit on the length of time for reaching a resolution, the ACAA attempts to rule on petitions within two weeks. Clients should submit challenges well in advance of a scheduled witness testing date (see Section 5.1).
On receipt of a petition, the ACAA checks whether the issue matches any that had been previously resolved. If the challenge is new, it is given an initial ACAA analysis which involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous deliberations. Often the ACAA consults Ada experts in order to resolve a petition. The identity of the petitioner is not disclosed when consulting outside experts. Resolution of a petition is made by the ACAA, and all ACALs are informed of the resolution.
The resolution of a petition is either an acceptance or rejection of the petitioners arguments. Acceptance can result in withdrawal of the test program from the ACATS, or a modification for conformity assessment. A test issue may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. If the challenge shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the ACAA will direct that the test(s) be processed with a test modification.
There are three types of test modification: Code, Processing, and Grading modifications.
All test modifications are documented in the ACATR.
A petitioner may resubmit a rejected petition, clearly stating additional information and reasoning as to why the original petition resolution is incorrect. The ACAA will resolve the resubmitted petition based on the deliberations of a body of Ada experts. A resolution of the resubmitted petition will be provided in no more than three weeks after submission.
A petitioner may resubmit a petition twice. A petitioner who has resubmitted a petition at least once can also request an extended resolution. In an extended resolution, the ACAA forwards the challenge to the ARG for resolution. (Extended resolution is not available for issues which have an interpretation approved in the last two years.) It is not anticipated that the ARG will resolve the issue in time for the conformity assessment giving rise to it. Therefore, the tests involved in an extended resolution will be graded as Unsupported, they will not be graded as failures for the purpose of issuing a certificate of conformity.
However, the expiration date of the certificate shall have a mark "pending issue resolution by ISO/WG9". The certificate shall expire on the day on which WG9 approves an interpretation of the Standard contradicting the petition and the processor will be removed from the Certified Processors List, or on its normal expiration date, whichever is sooner. The mark shall be removed from the Certified Processors List if WG9 approves an interpretation of the Standard confirming the petition.
There is no limit on the number of test programs that can be challenged by a petitioner. Although there is a risk that a petition will not be decided in a conformity assessment clients favor, early submission of petitions can reduce the risk that a conformity assessment will not be successfully completed on schedule. Any interested party may challenge an ACATS test program.
As permitted by [ISO 99], the ACAA provides mechanisms for extending the certified status of a tested processor to an implementation class (a set of closely related processors operating on a range of compatible configurations). This section describes these certification extension mechanisms.
An Ada processor is typically designed to be used on any member of a set of host and target computer-system pairs; furthermore, a processor is usually provided with different modes of operation (also known as "options" or "switch settings"). In witness testing, a processor is tested under one mode of operation on a particular configuration (host-target pair). The particular processor that is tested may be viewed as representing an implementation class, consisting of a particular (binary) processor and any configuration (host-target pair) on which it operates and produces equivalent ACATS results. Related implementation classes may include processors which are maintained versions of the test processor, and processors for which the host system is different. The ACAA may extend the "certified conforming" status to entire implementation classes.
The tested processor may be viewed as representative of several related implementation classes. These classes are categorized and defined in the following subsections.
A base implementation class includes a single (binary) processor that has achieved certified status through a complete conformity assessment (including witness testing). The processor may operate on multiple (closely related) configurations. The target instruction set architecture and target operating system of the additional configurations be the same as or a superset of those of the witness tested processor. The host system must be able to execute the witness tested processor. The processor must have a mode in which it can produce ACATS results that are equivalent (see section 7.2) to those of the tested processor for each configuration in the class.
A maintained implementation class is a class that includes a single (binary) processor that satisfies the following conditions:
The restriction of maintenance changes to corrective and perfective maintenance implies that the processor must have the same configuration(s) as the base processor class for the processor named in the ACAC.
A rehosted implementation class is a class that includes a single (binary) processor that satisfies the following conditions:
Adaptive maintenance may include limited changes to enable the processor to operate on a different host system from that of the processor named in the ACAC.
A rehosted implementation class may include closely related target systems. As with a base implementation class, the target instruction set architecture and target operating system of the additional target systems must be the same as or a superset of those of the processor named in the ACAC.
The conditions for extending the certified status of a processor require that the candidate processor be capable of producing ACATS results that are equivalent to those produced by the certified processor and configuration. In this context, equivalent ACATS results are results satisfying the following conditions:
Any deviation from the above requirements must be approved by the ACAA.
The purpose of the ACATR Supplement is to document the extension of certified status to an implementation class. See Appendix D for a sample ACATR. The Supplement contains the following information:
A client may request certification by extension for a base implementation class by submitting an ACATR Supplement (see section 7.3) to an ACAL. The receiving ACAL will submit this supplement to the ACAA for approval. On approval, the ACAA will create CPL entries identifying the implementation class as certified by extension.
The ACATR supplement for a base implementation class can be submitted at the same time as the Declaration of Conformity for a conformity assessment. In this case, the supplement can be simplified to eliminate duplicated information. This includes the certification of equivalent results (since this is tested by the ACAL for the conformity assessment).
The ACAA will reject obviously unreasonable claims of compatible configurations, but will not do any in-depth analysis of such claims. Users should regard the claims as vendor claims of compatibility.
A client may request certification by derivation for one or more implementation classes by submitting an ACATR Supplement (see Appendix D) to an ACAL. The receiving ACAL will submit this supplement to the ACAA for approval. On approval, the ACAA will create CPL entries identifying the implementation as certified by derivation. Certification by derivation is used for maintained and rehosted implementation classes.
For certification by derivation, the client must certify that a representative processor and configuration was tested using the customized ACATS (perhaps modified) that was used in the original conformity assessment. The client should be prepared to substantiate this claim as requested by the ACAA and/or ACAL.
The ACAA will reject obviously unreasonable claims of compatible configurations, but will not do any in-depth analysis of such claims. Users should regard the claims as vendor claims of compatibility.
Any interested party may challenge any approved certification by extension or derivation. Such a challenge must include non-conforming output on a member configuration of the implementation class. If, after analysis by the ACAA and rebuttal by the client, the processor is found to violate the requirements of certification by extension or derivation, the certification will be removed or corrected.
Anyone wishing to challenge an approved certification by extension or derivation, must provide the following information to the ACAA:
The ACAA will acknowledge receipt of this challenge. Note that deviations from the options or customized test suite used for the witness testing on which the original certificate is based will greatly increase the chances of the challenge being rejected.
The ACAA will analyze all received challenges, drawing on the test reports for the original conformity assessments, ACATR Supplements, and other relevant materials. If the analysis shows that the challenge has merit, it will be forwarded to the original testing ACAL and to the client for rebuttal. The client will be allowed thirty (30) days to prepare a rebuttal to the challenge. Such a rebuttal should show why the behavior of the processor is conforming, or demonstrate that the processor does in fact conform when the tests are processed. After receiving the rebuttal (or not having received a rebuttal within the designated time), the ACAA will rule on the challenge. The ruling will be distributed to the ACALs, the client, and the challenger.
If the final ruling is that a challenge is upheld, the certificate by extension or derivation will be removed from the CPL, or, by agreement between the client and the ACAA, modified to remove the offending configuration. If an upheld challenge demonstrates that the client fraudulently certified the testing of the representative processor and configuration in the ACATR Supplement, the clients right to submit ACATR Supplements will be suspended for a period not less than six months.
POINTS OF CONTACT
Ada Resource Association
Oliver Cole, Secretary
Ada Conformity Assessment Laboratories
Ada Conformity Assessment Authority
Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG)
Dr. Erhard Ploedereder
In either case, look for "compilers" and "ACATS."
Questions concerning Ada conformity assessment or comments on ACATS test programs should be sent to the ACAA (see above).<
TEST ISSUE FORMAT
Petitioner:<client name> Configuration:<host / target hardware and operating systems> ACATS Version:<ACATS version number> Self-Test Submittal Date:<due date for self-testing results> Part A will be completed once by each client; part B will be completed for each test issue. It is not necessary for a self-testing submittal date to have been established. Part A information is treated as confidential.
Reference:<test name (,test name)> Summary:<brief description of the test issue> Discussion:<detailed description of the test issue> In this Discussion, arguments should be specified using test line numbers and references to pertinent sections of the Ada standard or Commentaries (AI-xxxx). The petitioner must describe the behavior of the implementation for the test or tests that are challenged, stating the particular test messages that are produced. It is sufficient for the detailed description to be limited to the particular segment of test code that is challenged. Relevant source code with processor messages should be included. (For a group of tests that cause essentially the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.) If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified in the Discussion. It is especially important that implementations that fail to pass some test due to capacity limitations be described in enough detail for the ACAA to assess the reasonableness of these limitations. Failure to fully specify the points pertinent to a test issue might result in an adverse decision being made, with the petitioner having to further argue the case with a second submittal to the ACAA. It is also possible that the Summary will suffice to adequately present a test issue.
DECLARATION OF CONFORMITY
Declaration of ConformityIdentification Client:<client organization name> Certificate Awardee:<if different from client> ACAL:<name of Ada Conformity Assessment Laboratory > ACATS Version:<version number of ACATS>
Ada Processor and Configuration Processor:<name and version number of Ada processor> Host Computer System:<host hardware and operating system> Target Computer System:<target hardware and operating system>
Declaration: I, the undersigned, representing the Client, declare that the Client knows of no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:1995) in the Ada processor above. <The next sentence should normally be deleted.> The Conformity Assessment Test Report and Ada Conformity Assessment Certificate associated with this effort is not to be made public without Client permission. ________________________________________ ________________ Name Date <Title> <Client Organization> Omit the remainder if the certificate awardee is the client> Declaration: I, the undersigned, representing the Certificate Awardee, declare that the Certificate Awardee knows of no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:1995) in the Ada processor above. ________________________________________ ________________ Name Date <Title> <Organization>
ACATR SUPPLEMENT FORMAT
Supplement to ACATR <Reference ACATR number>
<name of client organization> (the client) hereby requests that the Ada Conformity Assessment Authority (ACAA) extend the certified status documented in the above-referenced Ada Conformity Assessment Test Report (ACATR) and in Ada Conformity Assessment Certificate (ACAC) <Reference ACAC number> to the implementation class(es) described in the following pages.
Implementation Class Information
<Complete the remaining pages for each processor class>
Implementation Class Category:<Base, Maintenance, or Rehosted> Processor Identification:<Name, Version, and Release identification> Host Systems:<Description of host computer models and operating systems. Ranges may be used.> Target Systems:<Describe as above or use "Same as host" or "Any host"> Representative Processor and Configuration Tested:
Host System:<Identification of specific host system (hardware and operating system) of tested processor> Target System:Identification of specific target system (hardware and operating system) of tested processor. "Same as host" is acceptable.> Client Certification of Testing and Processor Derivation: I, the undersigned, representing the client, certify that the above identified representative processor was tested on the described configuration with the customized ACATS that was used in the conformity assessment leading to the certificate named in this Supplement, with modifications described in this Supplement, and that the testing results were the same as those obtained in that conformity assessment, with exceptions as described in this Supplement. I further declare that the Client knows of no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:1995) in the identified representative processor above. I further certify that the above identified representative processor and configuration meets the definition of base, maintained, rehosted implementation (as described in the Operating Procedures for Ada Conformity Assessment). ________________________________________________
<For maintenance and rehosted implementation classes>
Include a brief description of the significant changes in the compiler in deriving it from the compiler named in the ACAC. Include the type of maintenance (Corrective, Perfective, or Adaptive)for each change noted.
Describe each change in the ACATS used in testing the reference processor, as compared to the customized ACATS used in testing the processor named in the certificate. Justification is required for any change other than the use of different implementation-dependent substitution values.>
Test Results Differences:
Describe and justify each difference between test results produced in testing the reference processor as compared to those produced in testing the processor named in the certificate.
This document contains a number of acronyms, whose meanings are given in the following list:
The following publications are referenced in this document:
[Ada95] ANSI/ISO/IEC 8652:1995 Ada 95 Reference Manual, January 1995 (supersedes [Ada83]).
[Ada83] American National Standards Institute and United States Department of Defense: ANSI/MIL-STD-1815A Reference Manual for The Ada Programming Language, 1983 Note: This standard is identical with ISO/8652:1987.
[ANSI/IEEE 90] American National Standards Institute / Institute of Electrical and Electronic Engineers, Inc., Standard 610.12-1990; "ANSI/IEEE Standard Glossary of Software Engineering Terminology".
[ISO 74] International Standards Organization: ISO 2382/I-1974 Data Processing - Vocabulary - Section 01: Fundamental Terms.
[ISO/IEC 91] International Standards Organization: ISO/IEC, Guide 2, 6th edition 1991 - General Terms and Their Definitions Concerning Standardization and Related Activities.
[ISO 99] FCD 18009, Information Technology -- Programming Languages -- Ada: Conformity Assessment of a Language Processor, ISO/IEC JTC1/SC22/WG9 N355, January 18, 1999