Measuring processes - About the (protocol) conformance tests

• Definition of conformance test
• ISO 9646 standard
• PICS and PIXIT documents
• Abstract (ATS) and executable (ETS) test suites
• Shortly about the TTCN
• Abstract testing methods
• Process of the execution of conformance test
• Test reports (SCTR, PCTR)

Conformance test serves for, whether an equipment, or a certain part of that equipment ( e.g. an interface) (IUT - Implementation Under Test) conforms for the related standards, or not. Conformance test follows developer tests in the testing hierarchy, namely it is executed, when the prototype of the product is ready.

In order to build operative networks from the product of firms producting different network elements, it is necessary to have standard protocols (communication rule systems). That is why most of the conformance tests are protocol tests. Immediately after that a claim occurs for that we could control, whether a product is able to realize a given protocol perfectly. Unfortunately the fact, that two equipments successfully passed the conformance test (namely it is a proper realization) does not necessarily mean, that they are able to cooperate perfectly, although the chance for this will be definitely better. In most of the cases the problem is given from that standards may contain details, that are not clear, because most of them is written in a natural language (usually in English). This case becomes more difficult, because sometimes solutions of several manufacturers are raised to be a standard, that cannot agree with each other, and these staandard are not compatible with each other. Interoperabilitiy test examines the cooperation of two or more equipments.

Conformance test is an instrumental (dynamic) test, which can be executed by the developer of the product in his own laboratory - according to the operative European conformance evaluation rules - or in an (accredited) test laboratory, prepared for these tests (e.g TVL).

The most important characteristics are the following: on the one hand,it builds up to the OSI layer model, and on the other hand, it considers the indvidual realizations to "black boxes", namely does not deal with their inner build-up, but just with their behaviour towards the outer world.

Conformance test process of equipments built according to the ISO OSI model serving for connecting the open systems with each other, and their interfaces is fixed in the ISO 9646 standard, namely the test according to the standard is standardized, too.

The ISO 9646 standard:

• Specificates documents (PICS, PIXIT), with that the related instructions can be formulated for the test.
• For the unified testing of the numerous implementation it specificates abstract test suites (ATS).
• Specificates a descripiton language for the description of the abstract test suites, it is the Tree and Tabular Combined Notation (TTCN).
• Specificates abstract testing methods.
• Specifies the procession of execution of the conformance test.
• Specificates requirements maintained for test laboratories and their clients for the execution of tests.
• Specificates test results and reporting methods.

PICS and PIXIT documents

The PICS proforma is an integrated part of the up-to-date equipment and interface standards. Example: ATM: ATM UNI 3.1 PICS

ISO 9646 classifies requirements into the following classes:

• mandatory
• conditional
• optional

PICS informs the tester about that maker of the prototype which requirements realizes from the standard, so this document is a kind of list of the realized options and capabilities. Data of PICS are used in numerous phases of the test.

• At the beginning of the test the document is subject of an examination, and the tester compares it to the standard (Example: ATM: ATM UNI 3.1), and checks, whether the maker of the prototype realized all the requirements, or not. This examination is called static test.

• If the prototype was found to be proper at the static test, next step will be the instrumental check of requirements, the dynamic test. Here the switches of test cases are operated on the basis of PICS.

• Finally, an efficient element of the test report (PCTR) is actually a PICS filled in by the tester, with that it is justified, that the tested requirements are appropriate to the related standard, or not.

A PIXIT (Protocol Implementation Extra Information for Testing)
document contains data and test controls, that are necessary for the execution of a dynamic test.

The PIXIT proforma is an integrated part of the up-to-date equipment and interface standards. Example: ATM: ATM UNI 3.1 PIXIT (Annex A)

Abstract (ATS) and executable (ETS) test suites

Word "abstract" means in this case, that realization of this set is independent, namely it contains only abstract instructions, e.g. "sending of a data packet to a given port".

A concrete example from the ATM UNI 3.1 ATS. The row describing test step PCO_A!CELL_NR in TTCN format describes sending of a CELL_NR ATM cell to the PCO_A port of the tester (! is the sign of the sending). Here we will not tell, what type is this cell, and where the port is located.

Test suite was built up hierachically, as it can be seen on the following figure:

• Test group
  • Test Case
    • Test Step
  • Test Case
    • Test Step
• Test group
  • Test Case
    • Test Step
  • Test Case
    • Test Step

Every test suite has an independent aim: to check the correct operation of a determined protocol-function. For sake of a more simple traceability, they are structured into test groups with an optional stratification level number. We can distinguish test cases within a test suite. These are the basic elements of the test operation, involving the sending of one protocol element, or its reception. Test cases can be modularized in the format of test steps to any kind of depth.

Beyond the above mentioned issues, ATS contains further information in many cases, primarily to make implementation easier. These are, for example, relation of statements in PICS and PIXIT with the selection mechanisms appropriate for the aims, and we can also classify the individual timing viewpoints to here.

Standard description language of ATS is TTCN. ATS itself cannot be executed, so it is necessary to convert ATS to a format, which can be run by the tester. This format is called executable test suite (ETS). Steps of the ATS->ETS conversion:

• Selection of test cases related to IUT from the test suite.
• Giving the parameters related to IUT on the basis of PIXIT (parametering)
• Translation of the selected and parametered testing cases into the format, that can be run by the tester.

If we have a TTCN translater to the given tester, then, if we have enough time, we will not really want to test, or, if we want to turn our hairs gray, we can do the translation by ourselves. We will reach our aim much quicker, if we buy the test suite translated by the maker, to that the producer gives even menus for the selection of tests and parametering. This kind of integrated test program allows besides issues mentioned above monitoring and it makes even the test records at the end of the test.

The following results could be generated in case of running of a test case:

• Pass: outcome of test shows, that IUT fulfilled the aim of the test, and during this action it showed nothing else, but a standard operation.
• Fail: IUT violated one of the conformance requirements, not necessarily the one, which was assigned by the test aim in question.
• Inconclusive: There will be neither a "pass", nor a "fail" judgement, because the test aim was not fulfilled, but there were not any protocol errors at all.

Shorthly about TTCN

They are presented in details at the Appendix.

This example [1] is from the fourth, "dynamic" part of the TTCN descripiton. It is worth to begin studying of the language here, because it bears the most of the information. Let us see, what is seen from the above part:

• Name of the test case and the test group
• Purpose of the test suite
• Comments
• and, what the most essential is: description of the dynamic behaviour.

This latter must be interpreted on the following way:

1. Sending out data element described in part "Constraints" to the PCO marked with "A".
2. Starting of a timer named T_Test. (Note: tabulating means a sequence in time!)
3. Several things could happen here:

1. A CELL_NR has arrived to the PCO marked with "B". In this case the verdict is "Pass" (P)
2. A CELL_UNASSIGN has arrived to the PCO marked with "B" In this case we go to the row number 5, and jump back to the label marked with LB1 (it is in Point 3. in this description).
3. T_Test timer is expired. Then the verdict is "Fail" (F)
4. Another data element, that wasn't mentioned yet, has arrived to the PCO marked with "B" The verdict is also "Fail" in this case.

Appendix A: TTCN in details

Abstract testing methods

Implementations of different protocol standard could be embedded into the whole system differently, so they require different testing methods. That is why the ISO 9646-2 standard defines a set of the testing methods, which involves separatedly a test configuration, PCO-types (Point of Control and Observation), and test coordination possibilities.

This set covers the possible protocol implementation/tested system (SUT System Under Test) pairs - the task is to select the righ method from them. Abstract Test Method (ATM) is determined by the following aspects:

• Configuration of IUT (Implementation Under Test) for the whole SUT (System Under Test)
• The whole testing configuration, including the system to be tested,
• All the points, through that the tester can control and test the IUT,
• Correctness of the coordination between different parts of the tester.

Conformance test conceptions standardized by the OSI are based on the well-known seven-layered reference model. Different abstract testing methods are distinguished from each other that at what point of IUT can they be controlled, or tested, namely where can PCO(s) be located. .According to the standard, PCOs can be considered as two holding rows. One of them is the row of events to be sent, another one is the row of newly received events. This controller, and combination of testing events will determine the operation of IUT perceived by the tester. Standard draws basically the scheme of two testing environments:

• Scheme of communication with one real open system (single-party testing context),
• Scheme of competitor communication with several real open systems (multi-party testing context).

As a consequence of characteristic of measurement (usage of one switch at once) we use the previous scheme. We will learn about the three possible methods of this shortly, and a fourth one, used during field-work, in details henceforward. For this, first we should define the necessary concepts shortly:

PDU : Protocol Data Unit

ASP: Abstract Service Primitiv, used on PCOs.

UT: Upper Tester, which approaches services provided by the IUT from the upper protocol layer boundary

LT: Lower Tester, which controls the the behaviour of IUT towards the lower layers.

TCP: Test Coordination Procedure, which provides the aligned testing.

TMP: Test Management Protocol, which is actually a standardized TCP.

So we will talk about firsst three testing methods shortly, that are not in use presently:

Local testing methods: here is the best possibility for the direct observation of IUT, but laboratory conditions are necessary for this, and their realization is quite difficult in practice. In this case we can observe PCOs both at upper and lower service boundaries. UT is realized in the tested system, TCPs are realized as a whole in the tester.

Distributed testing method: this method is very similar to the previous one, the essential differences are the following: UT is placed to the SUT, which requires also application of coordination processes (TCP). Realization of UT can be executed by human cooperation, or from a software. Disadvantage of this method, that a direct interference to the tested system (SUT) is also necessary.

Coordinated testing method: this method provides a solution in case, if we cannot access the upper service layer of IUT. Coordination methods are realized in a standardized form (TMP), and UT is a concrete implementation of the TMP then, within the given SUT. That is why ATS must be completed with test suites, that control, whether the UT is an appropriate implementation of TMP specification, or not.

Model of this method is demonstrated on the following figure:

In case of the remote testing method there is a lack of UT, and its tasks can be taken over by the system under testing (SUT). In the abstract test suite describing conformance requirements only the required effect of coordination methods of test is recorded, dotted lines around UT and TCP are relate to it. Lower tester (LT) has to try prefereably to execute TCP processes according to the information written in PIXIT, but the possibilities for this are quite limited because of the lack of UT. In the test suite there must be recorded ASPs perceiveable by the lower tester, and in form of PDUs the required operation of the system under testing (SUT).

This method has a smaller error recognition capability, but in this case it is the only one, that can be realized, because there is no possibility for a direct interference to the system. It is true in most of the cases, thus it is the most spread abstract testing method.

Process of the execution of conformance test

• Preparation and filling out of PICS and PIXIT documents
• Static conformance test
• Preparation of System Under Test (SUT, prototype) Means of Testing (MOT)
  • Connection and starting of SUT and the tester.
  • ATS->ETS conversion.
• Dynamic conformance test. (Running of ETS)
• Preparation of test report

Test reports (SCTR, PCTR)