Conformance and Interoperability Testing

 

Verification of the conformance of systems (implementations/artefacts) against a specification/standard is needed to aid the development of interoperable systems by making sure that they target that specification/standard in the same syntactic and semantic manner. Subsequently, interoperability testing ensures that systems that implement the same specification/standard do not behave differently in the parts of the architecture that are not covered by the specification.

Therefore, the e-SENS Conformance and Interoperability task force has worked towards developing a reusable, scalable and extensible framework to:

  • Support the sustainability of the e-SENS network by ensuring the conformance and interoperability of the participant systems with the developed e-SENS interoperability specifications, and
  • Facilitate and accelerate adoption of the e-SENS solutions by external organisations by providing an environment for an “extensible interoperability and conformance testing infrastructure” where they are able to test their applications and obtain detailed feedback from conformance and interoperability testing.

 

To achieve these aims, the task force has developed an open source, fully generic control flow management and manipulation framework called ‘Minder’.

 

The Minder test bed

Minder is an open source, generic testing environment for implementations of e-SENS standards/profiles. e-SENS partner organisations can check the technical conformance and interoperability with respect to the e-SENS building blocks. It is built on Play, which is a Java and Scala web application framework. It provides the ability to write test scenarios using the Minder Test Definition Language (MTDL, an extensible SCALA-based scripting language) including the use of external Java library dependencies, which are automatically resolved. It provides a programming environment that enables test experts to perform any kind of behavioural and semantic manipulation on the information flowing between the systems that are interconnected by Minder. A nice example might be the task of decrypting and modifying a message being sent from one system to another (in a test environment) and signing and encrypting the message to run a specific test task on the system that receives the message. Minder provides a web-based GUI for writing MTDL scripts and running them to test external systems. Because of its fully generic nature, Minder is capable of simulating and testing practically any system/artefact against a specification or with respect to arbitrary use cases. Therefore, Minder and its capabilities can be used for any building blocks in order to achieve interoperability to a large extent. Another very powerful feature of Minder is its ability to perform business document (XML) content verification, using its built-in Schema and Schematron verification capabilities. Using Minder, a test expert can perform verification via an XSD schema that resides at a remote URL, in an archive file (JAR, ZIP), and possibly refers to other XSD files (including) in the same archive or remote URL. The same extended capability is also available for Schematron verifications.

Minder is an open source project, and all of its source code and documentation are publicly available. Access Minder here.

 

Testing with Minder

The e-SENS Conformance and Interoperability Testing task force has been using the Minder test bed for e-SENS e-Delivery SAT conformance testing (mostly on the e-SENS AS4 and SMP specifications, as well as the e-SENS Evidence Emitter SBB).

An e-SENS/AS4 Conformance Test Event was carried out. From 1 December to 31 January, gateway vendors that already support the e-SENS/AS4 profile were invited to conformance-test their e-Delivery solution based on a current set of 48 test assertions that cover the full e-SENS/AS4 profile functionality. The test was run using the integrated test bed Minder, which has also been chosen by the e-Delivery DSI. This open source technology is a ‘one-stop shop’ test environment for e-SENS implementations. It is expected that participants will integrate their backend communication to the system with respect to the detailed documentation and PModes provided by the Minder AS4 Conformance Bridge called Kerkovi.

 

GITB compliance

Minder is compliant with the Global e-Business Interoperability Test Bed methodologies (GITB). GITB is a global initiative hosted by the European Committee for Standardization (CEN) and supported by the European Commission. It  focuses on methodologies and architectures that support e-business standards assessment and testing activities from early stages of business standards development to implementation and deployment of large-scale solutions. GITB promotes the reusability of testing resources and capabilities among different domains and different standards.