for Ada Conformity Assessments
April 23, 2001
Ada Resource Association
P.O. Box 1540
Fairfax, VA 22038
This document is also available in MS Word and Acrobat (.pdf) formats.
TABLE OF CONTENTS
- Executive Summary
- 1. Introduction
- 1.1 Background
- 2. Glossary of Terms
- 3. Bodies and Responsibilities
- 3.1 Ada Conformity Assessment Laboratories (ACALs)
- 3.2 Ada Conformity Assessment Authority (ACAA)
- 3.2.1 Sponsor
- 3.2.2 Technical Agent
- 3.2.3 Advisory Board
- 3.3 Conformity Assessment Clients
- 4. The Ada Conformity Assessment Test Suite
- 4.1 Applicability of ACATS Test Programs
- 4.2 Test Modification
- 4.3 Customization
- 4.4 ACATS Grading
- 4.5 ACATS Availability
- 4.5.1 ACATS Version Control System
- 4.6 ACATS Configuration Management
- 4.6.1 ACATS Modification List
- 4.6.2 ACATS Modification Categories
- 4.6.3 ACATS Baseline Version
- 4.6.4 ACATS Tests used
- 5. Conformity Assessment
- 5.1 Establishment of Agreement
- 5.2 Self-Test Evaluation
- 5.2.1 Client Testing
- 5.2.2 Submission of Results
- 5.2.3 ACAL Analysis and Test-Issue Resolution
- 5.2.4 Incomplete Self-test Evaluation
- 5.2.5 Successful Self-testing
- 5.3 Witness Testing
- 5.4 Documentation
- 5.4.1 The Ada Conformity Assessment Test Report
- 5.4.2 The Ada Conformity Assessment Certificate
- 5.5 Use of Obsolete ACATS Versions
- 5.6 Retention of Records
- 5.7 Advertising Status
- 6. Test Challenge and Resolution Process
- 6.1 Introduction
- 6.2 Resolution Process
- 6.3 Types of Resolutions
- 6.4 Reconsideration of Rejected Petitions
- 6.5 Summary
- 7. Extensibility of Conformity Assessment
- 7.1 Implementation Classes
- 7.1.1 Base Implementation Class
- 7.1.2 Maintained Implementation Class
- 7.1.3 Rehosted Implementation Class
- 7.2 Equivalence of ACATS Results
- 7.3 The ACATR Supplement
- 7.4 Requirements for Certification by Extension or Derviation
- 7.5 Expiration of Certification by Extension or Derivation
- 7.6 Challenging Certification by Extension or Derivation
- 7.6.1 Information Required to Challenge Certification
- 7.6.2 The Challenge Process
- Appendix A - Points of Contact
- Appendix B - Test Issue Format
- Appendix C - Declaration of Conformity
- Appendix D - ACATR Supplement Format
- Appendix E - Acronyms
- Appendix F - References
The International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), through Working Group 9 (WG9) of Subcommittee 22 (SC22) of their Joint Technical Committee 1 (JTC1), have established an International Standard titled Ada: Conformity Assessment of a Language Processor (ISO/IEC 18009:1999), referenced in this document as [ISO 99]. This standard specifies requirements for five aspects of a conformity assessment program, as follows:
The current document represents the Ada Resource Association (ARA)'s establishment of an ACAA governed by the procedures (ACAP) given herein. The primary goal of this document is to conform to the International Standard [ISO 99] (providing an ACAP as defined in that document).
[ISO 99] requires a periodic review and update of the ACAP. This document represents the results of the first such review and update. The major change is to allow vendors to use newer versions of the test suite for certifications by extension and derivation, and accordingly to extend the expiration dates of these certifications.
The following paragraphs summarize the conformity assessment system by using the language of the International Standard [ISO 99].
Ada Conformity Assessment Test Suite
The ACATS was originally derived from the Ada Compiler Validation Capability (ACVC). The ACVC was a conformity test suite (with supporting documents) developed under contract to the United States Government and made available for public use; it was designed to ensure that Ada processors achieve a high degree of conformity to the Ada language standard. A processor is tested when operating in a specific hardware and software configuration. The ACATS is customized by a testing laboratory (ACAL) for each processor that is subjected to conformity assessment; customization consists of adjusting the ACATS appropriately for various implementation characteristics. The ACATS is maintained by the ACAA; new releases of the ACATS are defined by changes resulting from the discovery of deficiencies in the test programs and by changes in the Ada standard (for instance, the adoption of Technical Corrigendum 1 [TC1]).
For an Ada processor to successfully complete conformity assessment, it must process each test program (for the "core" language) of a customized ACATS so that the result is graded Passed, Inapplicable, or Unsupported by ACATS grading criteria. A testing laboratory customizes the test suite for a particular processor by appropriately setting test parameters, by removing "withdrawn" tests (tests ruled by the ACAA to be in error) and certain inapplicable tests, by splitting as needed test files with multiple intended errors so to enable complete error detection, by using any other modified tests as directed by the ACAA, and by including each optional set of tests (see below) as requested by the client.
In addition to the specification of a "core" language, [Ada95] contains several Specialized Needs Annexes (SNAs); these specify language requirements designed to meet the particular needs of various general application domains, such as information-systems programming. A processor for [Ada95] need not include implementation of any of these annexes, or it may implement only some of the features of these annexes. Whereas all ACATS test programs for the core language must be processed during conformity assessment, those for the SNAs are processed only upon client request. A conformity assessment is judged successful (leading to the issuance of an ACAC) only if all tests for the core are correctly processed; the certificate will additionally give credit for support of a SNA to the extent that the relevant set of tests is correctly processed.
Conformity assessment involves interaction between the ACAL client and both the ACAL and the ACAA. The assessment process consists of well-defined actions which, when completed successfully, result in the award of an ACAC for the tested processor. The key actions in the conformity assessment of an Ada processor are:
- The client and an ACAL reach a formal agreement for conformity assessment, including the dates for the submittal of the results of client-administered processing of the ACATS and for ACAL witness testing.
- The client petitions for deviation from the requirements of each ACATS test program that is believed to be wrong for the candidate implementation(s).
- The ACAA rules on the client petitions.
- The client processes the ACATS on the candidate processor(s) and submits the results to the ACAL.
- The ACAL analyzes the results of the client’s independent processing of the ACATS. (If the results are not acceptable, the previous action must be repeated, and new results analyzed.)
- The ACAL conducts witness testing of the candidate processor(s), documents this testing in an Ada Conformity Assessment Test Report (ACATR), and submits the ACATR to the ACAA for review.
- The client signs a Declaration of Conformity for each candidate processor.
- The ACAA reviews the ACATR, with comments to the ACAL. The ACAA recommends to the ACAL that a conformity assessment certificate be issued for the tested processor(s), if the testing is successful.
- The ACAL issues a conformity assessment certificate for each tested processor upon the successful completion of the preceding actions.
Hence, successful testing of an Ada processor concludes with the ACAL's awarding a conformity assessment certificate for that processor (working in a specific configuration) to the client. This conformity assessment certificate attests that the processor has been subjected to an Ada conformity assessment and that no evidence of non-conformity was found. The processor is said to be "certified as conforming," as described in [ISO 99], clause 2.1. The processor will be listed in the ARA's Certified Processors List (CPL). The client may perform maintenance on the processor and may claim conformity for such derived versions in accordance with the ACAA procedures, so long as the client ensures that they produce the same ACATS results as are documented in the ACATR. This maintenance may even include adaptive maintenance that enables the processor to run on entirely different host computers (i.e., re-hosting) or to target closely related target computers. The ACAA provides a means for listing derived processors in the CPL.
This document provides operating procedures of the Ada Conformity Assessment Authority (ACAA). This body is a part of an organization that meets the requirements for assessing conformity of an Ada language processor, as given in [ISO 99]. The other bodies making up this organization are the Ada Conformity Assessment Laboratories (ACAL), which perform the actual conformity assessments using the Ada Conformity Assessment Test Suite (ACATS). The end product of a successful conformity assessment is an Ada Conformity Assessment Certificate (ACAC), indicating that a particular Ada language processor is "certified as conforming," as defined in [ISO 99].
This document forms an Ada Conformity Assessment Procedure (ACAP), as defined in [ISO 99].
Detailed procedures regarding the application of the ACATS are given in the ACATS User's Guide [ACATS UG].
The United States Department of Defense (DoD) sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. The AJPO established a certification system to realize the benefits of standardization, which include the ability to transfer software and programming expertise between computer systems that use a conforming Ada processor. When the AJPO ceased its operation of the certification system, the Ada Validation Facilities agreed to act as ACALs under the provisions of the emerging International Standard (now standardized as [ISO 99]). The Ada Resource Association, in cooperation with the Ada Joint Program Office, facilitated the identification and U.S. Government funding for a candidate ACAA and produced this document defining its operating procedures. The ACALs then agreed to designate the identified organization as the ACAA and to use these procedures as the ACAP.
It is important to note the scope and intent of conformity assessment. The purpose of conformity assessment is to ensure that Ada processors achieve a high degree of conformity with the Ada standard ([Ada95] as corrected by [TC1]). Characteristics such as performance and suitability for a particular application are not specified by the standard, and thus are outside the scope of Ada conformity assessment. Moreover, the ACATS is a set of test programs intended to check broadly for correct implementation; it is not possible to exhaustively test for conformity. Thus, conformity is checked only to the extent of these tests; processors that are certified as conforming may fail to conform to the standard in ways peculiar to each, under particular circumstances.
Witness testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of Ada conformity assessment is to identify processors that may be procured and used to develop application programs that meet the [Ada95] goals of portability and interoperability.
The ACATS (test suite) is not designed to replace the client’s quality assurance testing or systematically to detect inconsistencies or "bugs", but to verify that the tested processor correctly supports all required features. Rather than exhaustive testing of permutations of features, the test suite contains a carefully chosen set of test cases that cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard.
Neither is conformity assessment intended as a means of performance benchmarking. The Ada Conformity Assessment Test Report (ACATR) which documents the witness testing does not contain information about the speed, cost, or efficiency of executing the conformity assessment tests.
2. GLOSSARY OF TERMS
3. BODIES AND RESPONSIBILITIES
This section specifies the roles of the bodies that are responsible for Ada conformity assessment of clients who receive service from them.
3.1 Ada Conformity Assessment Laboratories (ACALs)
An ACAL is an independent testing laboratory that performs Ada conformity assessment activities. [ISO 99] includes a list of requirements that a testing laboratory must meet in order to be considered an ACAL. These requirements will not be repeated here. The ACAL operates under an ACAP consisting of its own operating procedures and the procedures defined in this document. An ACAL performs the following principal functions:
3.2 Ada Conformity Assessment Authority (ACAA)
The ACAA ensures worldwide commonality of the Ada Conformity Assessment Process. The technical and administrative functions of the ACAA are carried out by a technical agent. It is established by a sponsor and is advised by an Advisory Board.
The Ada Resource Association, a trade association of Ada product suppliers, sponsors the ACAA. The sponsor is responsible for the following:
3.2.2 Technical Agent
ACAA technical agent supports and coordinates the activities of the ACALs by:
3.2.3 Advisory Board
The ACAA Advisory Board represents the interests of the wider Ada community in the Ada Conformity Assessment process. Issues of policy and procedures are brought to the attention of the Board, which may make recommendations as to their resolution. Board members are appointed by the ACAA sponsor, and include (but are not limited to) the following:
3.3 Conformity Assessment Clients
A client is an individual or organization that contracts with an ACAL for conformity assessment services. Clients are required to provide accurate and complete information as specified in these procedures and the procedures of the ACAL.
4. THE ADA CONFORMITY ASSESSMENT TEST SUITE
The designated ACATS is the suite of conformity tests, support software, and documentation formerly known as the Ada Compiler Validation Capability (ACVC). The ACVC was developed under various contracts with the United States Department of Defense. It is designed to demonstrate the conformity of an Ada processor with the standard [Ada95] as corrected by [TC1]. The use of the ACATS is documented in the ACATS User’s Guide[ACATS UG], which explains the criteria for evaluating the results of the individual tests. While the ACVC was produced under contract to the United States Government, it is available to any individual or organization. The ACAA controls the content of the ACATS as it is used in conformity assessments, including the modification and addition of tests. Questions concerning Ada conformity assessment or comments on ACATS test programs should be submitted to the ACAA (see Appendix A, Points of Contact).
4.1 Applicability of ACATS Test Programs
Each ACATS test program has one or more test objectives that are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada processor (for example, "check that the proper exception is raised when Float’Machine_Overflows is True"). These test programs generally contain an explicit indication of their applicability and the expected behavior of processors for which they do not apply. The determination of applicability is made according to the grading criteria in the ACATS User's Guide or in the internal test documentation, or as a ruling by the ACAA. For a processor to be certified as conforming, all applicable test programs for the core language (as defined in [Ada95]) must be processed and passed according to the specified grading criteria.
Reference [Ada95] includes certain sections designated as Specialized Needs Annexes (SNA). The set of ACATS test programs for any of the SNAs will be processed only upon client request (to demonstrate full or partial support of the Annex). As permitted by [Ada95], test programs for the SNAs may be rejected at compile time or may exhibit run-time behavior that indicates a lack of support for requirements that only apply to SNAs. The ACAA may rule that tests producing such behavior are graded as "Unsupported". If the ACAA finds that the behavior is not in accordance with the permission granted by [Ada95], then the tests are graded as "Failed". Tests graded as Unsupported are reported in the ACATR and the ACAC, but these results do not affect the designation of the processor as being certified as conforming. On the other hand, tests graded as Failed are evidence of non-conformity, precluding the issuance of an ACAC for the candidate processor.
4.2 Test Modification
The various ACALs and the ACAA strive to apply the ACATS as uniformly as is practical to all Ada processors. In order to apply common test objectives that depend on implementation-dependent characteristics (e.g., line lengths and numeric types), some test programs must be adjusted to a given implementation following the procedures in [ACATS UG]. These adjustments consist of inserting implementation-dependent values in prescribed places in certain test programs.
In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (for example, the algorithm for recovering from syntax errors). The allowable changes for each Ada processor are determined by the [ACATS UG] and the ACAA, and may require ACAL assistance — especially in the case of processor error-recovery problems.
In order to meet a test objective, it may be required to modify the code, the processing method, or the grading of a test program. Only the ACAA shall make the decision to use any of these modifications, as described below:
The ACAL customizes the ACATS for each processor that is subject to witness testing. This customization always includes making all requiredimplementation-dependent substitutions. It may also include making code modifications that the ACAA directs for that specific conformity assessment as well as removing some inapplicable test programs as allowed by the ACATS User's Guide.
4.4 ACATS Grading
The result of processing an ACATS test program can be given only one of four possible grades: Passed, Inapplicable, Unsupported, and Failed. The first three grades are considered to constitute acceptable results. ACATS test programs that contain illegalities (which an implementation must detect) generate diagnostic output that must be inspected manually or by pattern-matching algorithms, matching system diagnostics to the intended errors. Executable ACATS test programs generate output using report procedures, which can be graded automatically. The ACATS report package, Report, contains specific output procedures for the two grades Failed and Inapplicable. If neither of these is invoked, the Report.Result procedure will report Passed or Tentatively Passed (indicating that the test has passed if manual inspection reveals that specific additional requirements are met). These results are the only ones that are generated by the test code (if no result is reported, that is, if the test completes abnormally, the result is graded Failed). The grade Unsupported is established as a means of grading tests that apply to the Specialized Needs Annexes (SNAs), as explained below.
The ACATS test programs for the SNAs pose two problems for using the three conventional grades of Passed, Inapplicable, and Failed. The broad problem is that full support of any such Annex is not required for conformity to [Ada95] — there may be no support, or merely partial support. Unfortunately, there is no way to discriminate between full and partial support if only those three grades are used, since the grades Inapplicable and Failed are not appropriate for this (an implementation is not allowed to provide deviant semantics for an unsupported Annex feature — that would be a conformity assessment failure). The second problem is that there are some test programs for Core features that are applicable also to a SNA, in particular, the test programs for representation items. These programs constitute tests for features that are defined in the Core as optional, but are mandatory for full support of the Systems Programming Annex (which itself is mandatory for full support of the Real-Time Systems Annex).
Therefore, the ACAL grades the result of processing such an ACATS test program (i.e., one that uses a feature required by, or defined in, an Annex) as Unsupported, if the prima facie result is failure but the implementation's processing of the test program is an acceptable form of non-support. For example, if a processor does not support a particular form of a representation clause, it must reject any test program that uses it — such rejection of an executable test is usually graded Failed, but is graded Unsupported if the implementation does not claim support of the relevant SNA. A processor that accepts the representation clause and reports Failed, on the other hand, is deemed to have failed the test regardless of any claim (or lack thereof) of support for the SNA.
4.5 ACATS Availability
The current baseline version of the ACATS is available to the general public from an ACAL or from an Internet site supported by the ARA. The current and any previous versions of the ACATS Modification List are available from the same sources. ACALs may assist the client in format conversion when providing the ACATS in a particular distribution medium. If a client has a need for a superseded version of the ACATS, it may be available from the ACAA or from an ACAL. See APPENDIX A for points of contact
4.5.1 ACATS Version Control System
The ACAA maintains an official ACATS web site, accessible via the Internet. The web site includes a web-accessible version control system, which contains the official version of the ACATS. Both old and new versions of tests are accessible given the test name and version label desired.
Instructions for using the ACATS version control system are available on the web page.
4.6 ACATS Configuration Management
Test challenges and ARG interpretations may reveal flaws in the ACATS. The ACAA may remove, repair, and insert tests in response to those needs. These test suite changes are listed in the Ada Conformity Assessment Test Suite Modification List (ACATS Modification List, or AML for short). The ACAA issues this list as needed. The AML contains information about affected tests and instructions for acquiring new and modified tests. The official versions of the tests (including modified and new tests) are available via the official ARA web site and other sources (see Section 4.5)
4.6.1 ACATS Modification List
New versions of the ACATS Modification List will be issued when test suite changes are needed. The list contains at least the following information:
The official version of a test is always available from the ACATS VCS. Information in the list is advisory only; in the case of a conflict between the ACATS VCS, and the list, the ACATS VCS is assumed correct.
4.6.2 ACATS Modification Categories
The ACAA may issue various kinds of test suite modifications. Each affected test has one of the following categories:
Withdrawn: The test is seriously flawed. It may have errors that cannot be corrected, or may require extensive corrections. It is removed from the test suite. Conformity assessments do not process such tests.
If a correction of a Withdrawn test is prepared, it will be treated as a new test.
Allowed Modification: The test has minor flaws. A modified version has been posted on the ACATS VCS. The test has an effective date that specifies when the test will be moved to the Modified Category.
The effective date will always be at least three months after the posting date, and will be at the beginning of a quarter (that is, January 1st, April 1st, July 1st, or October 1st). A conformity assessment may choose to process the original test or the new, modified test. Test choices can be made on an individual test basis. That is, a conformity assessment may choose to process some new modified tests while using the original tests for others.
Modified: The test has minor flaws. A modified version has been posted on the ACATS VCS. Conformity assessments must process the modified test.
Pending New: The test is newly created, or is a correction of a test that was previously withdrawn, or has added test cases. It is posted on the ACATS VCS. The test has an effective date that specifies when the test will be moved to the New category. The effective date will always be at least six months after the posting date, and will be at the beginning of a quarter (that is, January 1st, April 1st, July 1st, or October 1st). Pending New tests are not used for conformity assessment until the effective date is reached. Users of the test suite are encouraged to run the tests as soon as possible.
If it becomes necessary to modify a Pending New test, the effective date is adjusted as if the test was newly created.
New: The test is newly created, or is a correction of a test that was previously withdrawn, or has added test cases. It has been posted on the ACATS VCS for at least six months. Conformity assessments must process the test.
4.6.3 ACATS Baseline Version
The ACAA baselines the ACATS approximately once per year. When the ACATS is baselined, the original official set of files is updated with all of the changes specified in the ACATS Modification List. Allowed Modification and Pending New tests are not included (since they have not been available long enough to include). Additionally, the documentation associated with the test suite is updated. Future versions of the ACATS Modification List are then based on the new test version. Any Allowed Modification and Pending New tests will be listed in the initial version of the ACATS Modification List for the new baseline version.
The effective date of a baseline version will be announced at least three months prior to its being effective, and should be at the beginning of a quarter (that is, January 1st, April 1st, July 1st, or October 1st). The documentation will be available not less than 30 days prior to it being effective. Conformity assessments started after the effective date must use the new baseline test suite. (Note that the tests that make up the baseline version are known on the date that the effective date is announced, and are accessible on the web site, even before the final version of the baseline documentation is available.)
4.6.4 ACATS Tests used
The tests used for a particular conformity assessment are defined as the tests from a particular baseline version of the ACATS, modified as follows:
Each ACATR identifies the baseline version of the ACATS, and documents all modifications made to that baseline version of the ACATS.
Conformity assessments must use the most recent version of the ACATS Modification List at the start of witness testing. (Note that the set of required tests is always known at least three months in advance.) Most tests and support files modified for an individual conformity assessment are included in the ACATS Version Control System. Tests modified only by making implementation-dependent substitutions (typically by using a tool) will not be included. B-Test splits allowed by section 4.2 and tests modified only by splitting between compilation units will also not be included.
The files will be posted on the ACATS Version Control System before the ACATR is issued. Each conformity assessment has a unique version label, which can be used on the web site to access all of the files that differ from the baseline versions. The version label is included in the ACATR. It is intended that the information in the ACATR and the files available on the official ACATS Version Control System will allow users to reproduce the conformity assessment on their own.
5. CONFORMITY ASSESSMENT
In order for a client to obtain a conformity assessment certificate and an ACATR, the client, the ACAL, and the ACAA must complete number of steps. The same ACATS version, including the application of the requirements of the ACATS Modification List, must be used to complete the steps described in this section. Anyone intending to obtain a conformity assessment certificate should contact an ACAL without delay for advice on the handling of the ACATS, on interpretation of the test grading criteria, and on the operational procedures of that ACAL.
The required steps follow:
- Establishment of Agreement
- Self-Test Evaluation
- Witness Testing
5.1 Establishment of Agreement
In order to obtain conformity assessment services, an interested party must become a client of an ACAL by reaching a formal agreement. This agreement addresses the following topics:
The schedule for events, deliverables, and payments should take into account the fact that certain steps in the conformity assessment process require interaction with the ACAA. The ACAA and ACAL will keep confidential a client’s intent to obtain a conformity assessment certificate and the projected schedule for conformity assessment. If the client requests more restrictive confidentiality conditions for reasons of national security or procurement sensitivity, the client will provide to the ACAL an official, written statement describing the request and the reason(s) for the request; the ACAL will also obtain further guidance from the ACAA.
5.2 Self-Test Evaluation
Self-test evaluation entails a series of actions and is usually where the bulk of the conformity assessment effort is expended. These actions are described in the following subsections.
5.2.1 Client Testing
After entering into a formal agreement, the client obtains a customized test suite from the ACAL. (At the client’s risk, the client may prepare this customized test suite according to instructions in the ACATS User's Guide, rather than obtaining it from an ACAL). The client then processes all the tests in this customized test suite using the candidate processor on the candidate configuration or on another configuration that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of options controlling the production of information output. (For example, options that control the format of listings, the format of error messages, and the generation of listings may vary.) Any other exception constitutes a test issue that must be resolved with the ACAL (see Section 5.2.3). Test issues should be sent to the ACAL for analysis as soon as possible.
Self-test activities include as a minimum the processing of an appropriately customized test suite by the client, preparation of a client supplied Declaration of Conformity, and submission of any test issues.
5.2.2 Submission of Results
Upon completion of self-testing, the client delivers the complete set of results in the agreed format to the ACAL. (See Section 5.2.4 for an alternative to submission of complete results.)
Results are accompanied by the following information:
The Declaration of Conformity states that the organization responsible for the production, maintenance or distribution of the Ada processor is offering a product that is in conformity with [Ada95] as corrected by [TC1]. The client must ensure that the information contained in the Declaration of Conformity does not infringe on the rights of a third party, and may be required to provide a written statement of consent from any third party involved. The Declaration of Conformity becomes part of the ACAL records and is copied into the ACATR. The ACAL will not issue a certificate until the ACAA has reviewed a signed Declaration of Conformity. (See Appendix C for an example of the Declaration of Conformity.)
5.2.3 ACAL Analysis and Test-Issue Resolution
The ACAL analyzes the client's submitted results of self-testing, checking that all test programs have produced acceptable results according to the ACATS evaluation criteria. During this analysis period, the client and the ACAL resolve any test issues.
A test issue is defined to be any of the following:
A client may challenge an ACATS test program's correctness or applicability to a particular implementation. Such challenges should be presented to the ACAL in the petition format given in Appendix B. The ACAL will forward any petitions to the ACAA for resolution; the ACAA will strive to rule on the petition within two weeks of receiving it. The ACAA reports all challenges and rulings to each ACAL. However, an ACAL may not apply an ACAA ruling for one conformity assessment to another conformity assessment without the ACAA so directing. (See Section 6 for a description of the Challenge and Resolution Process.)
In some cases, it may be agreed to leave a test issue until witness testing. For example, it might be impossible to check the processing of control characters by inspecting printed results. The ACAL will note any unresolved issues and describe the results that are expected during witness testing. It is also possible that the client information for the production of the customized test suite (see Section 5.2.2) was insufficient, so that corrections to the customized test suite must be made, requiring additional processing.
5.2.4 Incomplete Self-test Evaluation
The ACAL and the client may agree that, at the client’s risk, parts of the customized test suite need not be processed during self-testing. There are two typical situations, as follows:
The normal practice is to submit complete self-testing results for at least one of the implementations under test. The ACAL may require the submission of complete self-testing results.
5.2.5 Successful Self-testing
Self-testing is successful if the analysis of results and the resolution of test issues show that all results have been provided and are acceptable. Self-testing is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues is deferred until witness testing by agreement between the ACAL and the client.
5.3 Witness Testing
Upon successful completion of self-testing, with or without caveats, the ACAL witnesses testing of the Ada processor in accordance with the formal agreement between the ACAL and client. Witness testing takes place in the physical presence of qualified ACAL personnel. The ACAL supplies a customized test suite that it has prepared based upon client information and any information collected during the resolution of test issues. The customized test suite will include the set of test programs for the core language and each set, as requested by the client, any (or none) of the Specialized Needs Annexes (SNAs). The ACAL verifies that the processor identification, including identification of the processor and configuration (hardware systems and operating systems), matches that given in the Declaration of Conformity. (If it does not agree, then the client must provide a new Declaration of Conformity.)
The ACAL observes the installation of the customized ACATS on the host computer system, monitors the processing of the customized ACATS on the host and target computer systems, and evaluates the results. The entire customized test suite should be run on a single copy of the Ada processor on a single configuration using a unique set of option settings of the processor. (Differences in options controlling the production of information output and differences from accepted test issues are allowed. See section 5.2.1.) If the ACAL determines that the results agree with those obtained from self-testing and are satisfactory with respect to any caveats, the witness testing has been successful; otherwise, the test is unsuccessful. If any result of testing with a set of test programs for a Specialized Needs Annex is unacceptable, the test report and certificate of the conformity assessment will not recognize that the set was processed.
Each conformity assessment effort is documented by an ACATR, and each successful effort is further documented by an ACAC.
5.4.1 The Ada Conformity Assessment Test Report
An ACATR is produced for each processor and configuration subjected to witness testing. Each ACATR contains, at a minimum, the following information:
18.104.22.168 ACATR Production
The ACATR is prepared by the ACAL but includes material that is produced by the client, such as the documented processor options used during witness testing. A draft version of the ACATR, based on results and circumstances implied by the evaluation of self-testing results, is sent to the ACAA for review. The draft version is also submitted to the client for review during witness testing, and is updated to account for client comments and observations made during witness testing. For a successful conformity assessment, the final version of the ACATR is signed by the ACAL and the ACAA. For an unsuccessful conformity assessment, the final ACATR is provided to the client only.
Final test reports will never be modified. If it becomes necessary to correct a final ACATR, the ACAL will prepare a separate document titled "Supplement to Ada Conformity Assessment Test Report <unique report identifier>". Such a supplement will meet the applicable requirements of section 5.4.1.
22.214.171.124 ACATR Availability
The final version of the ACATR for a successful conformity assessment is available to the general public from the client, from the ACAL that produced it, and from the ACAA in electronic form. The ACAL may require payment of a fee for ACATR reproduction and delivery. (See Appendix A for points of contact.) By including an appropriate request on the Declaration of Conformity (Appendix C), the client may disallow public availability of the ACATR and the ACAC.
5.4.2 The Ada Conformity Assessment Certificate
With the concurrence of the ACAA, the ACAL issues an Ada Conformity Assessment Certificate (ACAC) for each processor and configuration that was subject to successful witness testing. The information on the certificate is derived from the client's Declaration of Conformity and the ACATR. The ACAC conveys to the processor and configuration the status of certified as conforming, as defined in [ISO 99]. An entry is made in the CPL for each ACAC, unless the client has requested confidentiality on the Declaration of Conformity (see Appendix C).
The ACAC contains the following information:
Note that an ACAC attests that testing was performed on a specific processor using a specific test suite running on a specific configuration, following the Ada Conformity Assessment Procedure, and that no evidence of non-conformity was detected. It does not certify that the processor is free of defects, nor does it certify that the processor is usable for any particular purpose.
ACACs expire two years after issuance. When an ACAC expires, the corresponding entry in the CPL is clearly identified as expired. (Entries for derived processors may also expire at the same time, see section 7.6). Certificates expire in order to encourage periodic retesting of processors, which ensures that they continue to meet the requirements of conformity assessment.
5.5 Use of Obsolete ACATS Versions
For some special procurement requirements, a client might wish to have witness testing done with an obsolete version of the ACATS. The ACAP does not include any procedures for recognizing testing with obsolete test suite versions, but the ACALs may provide such a service outside the system. Ada Conformity Assessment Certificates will not be issued for testing with obsolete test suites, nor will CPL entries be created based on such testing.
5.6 Retention of Records
The ACAA retains a copy of each ACATR (which includes a copy of the Declaration of Conformity and the ACAC), records pertaining to issues and their resolution, and a copy of each registration request. The ACAL retains a copy of each ACATR, a copy of the customized ACATS used in witness testing, and a copy of the witness testing results. The ACAA retains its records until at least five years following expiration of the ACAC. Each ACAL's procedures specify the length of time that its records are retained, but records must be retained at least seven years after the completion of witness testing.
5.7 Advertising Status
The client must agree not to advertise or make public claims that the Ada processor is certified as conforming until after receiving the ACAC or receiving formal notification from the ACAL that it has issued an ACAC. A client who intends to advertise the completion of events that indicates progress toward completion of conformity assessment must sign a waiver of confidentiality. If a waiver of confidentiality has been signed with the ACAL, the ACAL will respond to inquiries about the client’s advertisements or public claims by acknowledging receipt of conformity assessment materials (i.e., a formal agreement, self-testing results, or witness testing results) without judgment concerning the success of the witness testing.
6. TEST CHALLENGE AND RESOLUTION PROCESS
This section presents the process whereby tests may be challenged, possibly resulting in their modification or withdrawal.
A "deviation" is defined by the ACATS User's Guide as any result from processing an ACATS test program that is not a Passed or Inapplicable result according to the established grading criteria. This intentionally broad definition of a "deviation" is intended to ensure that processor implementers bring all deviant test results to the attention of the ACAA or ACAL, without assuming that such results are acceptable. In petitioning for acceptance of a deviation, the petitioner provides a rationale for each challenge made against a test program. Petitions are sent to the ACAA, usually electronically, by the petitioner or by an ACAL on behalf of its client. For each deviation that is accepted (that is, when the ACAA rules in favor of the petition), generally some correction is indicated for the cited tests. The ACAA may withdraw a test program or require that a modified version of the test be processed (see section 6.4). Withdrawal of a test program or the provision of a modified version of a test results in the release of a new version of the ACATS Modification List.
6.2 Resolution Process
The ACAA typically resolves challenges by any of three methods:
Although these procedures do not set a time limit for reaching a resolution, the ACAA attempts to rule on petitions within two weeks. Clients should submit challenges well in advance of a scheduled witness testing date (see Section 5.1).
On receipt of a petition, the ACAA checks whether the issue matches any that have been previously resolved. If the challenge is new, it is given an initial ACAA analysis that involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous deliberations. Often the ACAA consults Ada experts in order to resolve a petition. The identity of the petitioner is not disclosed when consulting outside experts. Resolution of a petition is made by the ACAA, and all ACALs are informed of the resolution.
6.3 Types of Resolutions
The resolution of a petition is either an acceptance or rejection of the petitioner’s arguments. Acceptance can result in withdrawal of the test program from the ACATS, or a modification for conformity assessment. A test issue may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. If the challenge shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the ACAA will direct that the test(s) be processed with a test modification.
There are three types of test modification: Code, Processing, and Grading modifications.
All test modifications are documented in the ACATR.
6.4 Reconsideration of Rejected Petitions
A petitioner may resubmit a rejected petition, clearly stating additional information and reasoning as to why the original petition resolution is incorrect. The ACAA will resolve the resubmitted petition based on the deliberations of a body of Ada experts. A resolution of the resubmitted petition will be provided in no more than three weeks after submission.
A petitioner may resubmit a petition twice. A petitioner who has resubmitted a petition at least once may also request an extended resolution. In an extended resolution, the ACAA forwards the challenge to the ARG for resolution. (Extended resolution is not available for issues that have an interpretation approved in the last two years.) It is not anticipated that the ARG will resolve the issue in time for the conformity assessment that gave rise to it. Therefore, the tests involved in an extended resolution will be graded as Unsupported; they will not be graded as failures for the purpose of issuing a certificate of conformity.
However, the expiration date of the certificate shall be marked "pending issue resolution by ISO/WG9". The certificate shall expire on the day on which WG9 approves an interpretation of the Standard contradicting the petition and the processor will be removed from the Certified Processors List, or on its normal expiration date, whichever is sooner. The mark shall be removed from the Certified Processors List if WG9 approves an interpretation of the Standard confirming the petition.
There is no limit on the number of test programs that can be challenged by a petitioner. Although there is a risk that a petition will not be decided in a conformity assessment client’s favor, early submission of petitions can reduce the risk that a conformity assessment will not be successfully completed on schedule. Any interested party may challenge an ACATS test program.
7. EXTENSIBILITY OF CONFORMITY ASSESSMENT
As permitted by [ISO 99], the ACAA provides mechanisms for extending the certified status of a tested processor to an implementation class (a set of closely related processors operating on a range of compatible configurations). This section describes these certification extension mechanisms.
7.1 Implementation Classes
An Ada processor is typically designed to be used on any member of a set of host and target computer-system pairs; furthermore, a processor is usually provided with different modes of operation (also known as "options" or "switch settings"). In witness testing, a processor is tested under one mode of operation on a particular configuration (host-target pair). The particular processor that is tested may be viewed as representing an implementation class, consisting of a particular (binary) processor and any configuration (host-target pair) on which it operates and produces equivalent ACATS results. Related implementation classes may include processors that are maintained versions of the test processor, and processors for which the host system is different. The ACAA may extend the "certified conforming" status to entire implementation classes.
The tested processor may be viewed as representative of several related implementation classes. These classes are categorized and defined in the following subsections.
7.1.1 Base Implementation Class
A base implementation class includes a single (binary) processor that has achieved certified status through a complete conformity assessment (including witness testing). The processor may operate on multiple (closely related) configurations. The target instruction set architecture and target operating system of the additional configurations must be the same as or a superset of those of the witness tested processor. The host system must be able to execute the witness tested processor. The processor must have a mode in which it can produce ACATS results that are equivalent (see section 7.2) to those of the tested processor for each configuration in the class.
7.1.2 Maintained Implementation Class
A maintained implementation class is a class that includes a single (binary) processor that satisfies the following conditions:
The restriction of maintenance changes to corrective and perfective maintenance implies that the processor must have the same configuration(s) as the base processor class for the processor named in the ACAC.
7.1.3 Rehosted Implementation Class
A rehosted implementation class is a class that includes a single (binary) processor that satisfies the following conditions:
Adaptive maintenance may include limited changes to enable the processor to operate on a different host system from that of the processor named in the ACAC.
A rehosted implementation class may include closely related target systems. As with a base implementation class, the target instruction set architecture and target operating system of the additional target systems must be the same as or a superset of those of the processor named in the ACAC.
7.2 Equivalence of ACATS Results
The conditions for extending the certified status of a processor require that the candidate processor be capable of producing ACATS results that are equivalent to those produced by the certified processor and configuration. In this context, equivalent ACATS results are those satisfying the following conditions:
The ACAA must approve any deviation from the above requirements.
7.3 The ACATR Supplement
The purpose of the ACATR Supplement is to document the extension of certified status to an implementation class. See Appendix D for a sample ACATR. The Supplement contains the following information:
7.4 Requirements for Certification by Extension or Derivation
A client may request certification by extension for a base implementation class by submitting an ACATR Supplement (see section 7.3) to an ACAL. Similarly, a client may request certification by derivation for one or more implementation classes by submitting an ACATR Supplement to an ACAL.
The Ada Conformity Assessment Certificate (ACAC) referenced in an ACATR Supplement must have been issued within the five years previous to the date of submission of the supplement.
When submitting an ACATR Supplement (except as noted below), the client must certify that a representative processor and configuration was tested using a customized ACATS as described above, and that the results were equivalent as defined by section 7.2. The client should be prepared to substantiate this claim as requested by the ACAA or ACAL.
The receiving ACAL will check all test result differences indicated in the ACATR Supplement, checking that all such test programs have produced acceptable results according to the ACATS evaluation criteria. Any test issues it identifies shall be resolved as described for Self-Testing, see section 5.2.3.
Once any test issues identified have been resolved (possibly by modification of the supplement), the ACAL will append a summary of the test results differences to the supplement and then will submit the supplement to the ACAA for approval. On approval, the ACAA will create CPL entries identifying the implementation class as certified by extension or derivation. The ACAA will automatically reject a supplement with any unresolved test issues.
The ACATR Supplement for a certification by extension can be submitted at the same time as the Declaration of Conformity for a conformity assessment. In this case, the supplement does not need to include a certification of equivalent results since this is tested by the ACAL for the conformity assessment.
The ACAA will reject obviously unreasonable claims of compatible configurations, but will not do any in-depth analysis of such claims. Users should regard the claims as vendor claims of compatibility.
7.5 Expiration of Certification by Extension or Derivation
Certification by extension or derivation using the same test suite and modifications as the original certificate (ACAC) expires at the same time as the original ACAC. In contrast, certification by extension or derivation using the current test suite and modification expires two years after issuance. As with an ACAC, when certification by extension or derivation expires, the corresponding entry in the CPL is clearly identified as expired. Certificates expire in order to encourage periodic retesting of processors, which ensures that they continue to meet the requirements of conformity assessment.
7.6 Challenging Certification by Extension or Derivation
Any interested party may challenge any approved certification by extension or derivation. Such a challenge must include non-conforming output on a member configuration of the implementation class. If, after analysis by the ACAA and rebuttal by the client, the processor is found to violate the requirements of certification by extension or derivation, the certification will be removed or corrected.
7.6.1 Information Required to Challenge Certification
Anyone wishing to challenge an approved certification by extension or derivation, must provide the following information to the ACAA:
The ACAA will acknowledge receipt of the challenge. Note that deviations from the options or customized test suite used for the representative testing on which the certificate is based will greatly increase the chances of the challenge being rejected.
7.6.2 The Challenge Process
The ACAA will analyze all received challenges, drawing on the test reports for the original conformity assessments, ACATR Supplements, and other relevant materials. If the analysis shows that the challenge has merit, it will be forwarded to the original testing ACAL and to the client for rebuttal. The client will be allowed thirty (30) days to prepare a rebuttal to the challenge. Such a rebuttal should show why the behavior of the processor is conforming, or demonstrate that the processor does in fact conform when the tests are processed. The ACAA will rule on the challenge after either receiving the rebuttal or the expiration of the designated time. The ruling will be distributed to the ACALs, the client, and the challenger.
If the final ruling is that a challenge is upheld, the certificate by extension or derivation will be removed from the CPL or, by agreement between the client and the ACAA, modified to remove the offending configuration. If an upheld challenge demonstrates that the client fraudulently certified the testing of the representative processor and configuration in the ACATR Supplement, the client’s right to submit ACATR Supplements will be suspended for a period not less than six months.
POINTS OF CONTACT
Ada Resource Association
Oliver Cole, SecretaryURL: http://www.adaresource.com
Ada Resource Association
P.O. Box 1540
Fairfax, VA 22038
Ada Conformity Assessment Laboratories
EDS Conformance Testing Center
4646 Needmore Road, Bin #46
P.O. Box 24593
Dayton, OH 45424-0593
Tel: +49 89 5908 6576
Fax: +49 89 5908 6580
Ada Conformity Assessment Authority
P.O. Box 1512
Madison, WI 53701
Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG)
Dr. Erhard Ploedereder
University of Stuttgart
Institute for Computer Science
Breitwiesenstr. 20-22D-70565 Stuttgart
Tel: +49 +711 7816-322
Fax: +49 +711 7816-380
Ada Conformity Assessment Test Suite (ACATS)
The ACATS is available to the general public from an ACAL; it is also available from the ACAA Internet site.
The site includes downloadable versions of complete ACATS, the ACATS VCS for access to individual files and modifications, recent versions of the ACATS Modification List, and packaged versions of new and modified tests.
Questions concerning Ada conformity assessment or comments on ACATS test programs should be sent to the ACAA (see above).
TEST ISSUE FORMAT
Petitioner: <client name>
Configuration: <host / target hardware and operating systems>
ACATS Version: <ACATS version number>
Self-Test Submittal Date: <due date for self-testing results>
Part A will be completed once by each client; part B will be completed for each test issue. It is not necessary for a self-testing submittal date to have been established. Part A information is treated as confidential.
Reference: <test name (,test name)>
Summary: <brief description of the test issue>
Discussion: <detailed description of the test issue>
In this Discussion, arguments should be specified using test line numbers and references to pertinent sections of the Ada standard, Technical Corrigendum, or Commentaries (AI-xxxx). The petitioner must describe the behavior of the implementation for the test or tests that are challenged, stating the particular test messages produced. The detailed description can be limited to the particular segment of test code that is challenged. Relevant source code with processor messages should be included. (For a group of tests that cause essentially the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.)
If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified in the Discussion. It is esp