Cloud Application Management for Platforms Test Assertions Version 1.1

Cloud Application Management for PlatformsTest Assertions Version 1.1

Working Draft 07

26August 2013

Technical Committee:

OASIS Cloud Application Management for Platforms (CAMP) TC)

Chair:

Martin Chapman (), Oracle

Editors:

Jacques Durand (), Fujitsu Limited

Anish Karmarkar (), Oracle

Adrian Otto (), Rackspace Hosting, Inc.

Additional artifacts:

This prose specification is one component of a Work Product which also includes:

??
Other parts (list titles and/or file names)

Related work:

This specification replaces or supersedes:

Specifications replaced by this specification (hyperlink, if available)

This specification is related to:

Related specifications (hyperlink, if available)

Abstract:

This document defines the Test Assertions for V1.1 of the CAMP (…) specification for Platform as a Service (PaaS). Test Assertionsare testable or measurable expressionsrelated to normative statements in a specification for evaluating if an implementation (or part of it) conforms to this specification. These Test Assertions are intended to be used for defining precisely the conditions required by a CAMP implementation in order to conform. They support the testing activity by representing a bridge between the normative statements in the specification and the executable test cases that are parts of a conformance test suite.

Status:

This Working Draft (WD) has been produced by one or more TC Members; it has not yet been voted on by the TC or approved as a Committee Draft (Committee Specification Draft or a Committee Note Draft). The OASIS document Approval Process begins officially with a TC vote to approve a WD as a Committee Draft. A TC may approve a Working Draft, revise it, and re-approve it any number of times as a Committee Draft.

All capitalized terms in the following text have the meanings assigned to them in the OASIS Intellectual Property Rights Policy (the "OASIS IPR Policy"). The full Policy may be found at the OASIS website.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published, and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this section are included on all such copies and derivative works. However, this document itself may not be modified in any way, including by removing the copyright notice or references to OASIS, except as needed for the purpose of developing any document or deliverable produced by an OASIS Technical Committee (in which case the rules applicable to copyrights, as set forth in the OASIS IPR Policy, must be followed) or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by OASIS or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and OASIS DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY OWNERSHIP RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Table of Contents

1Introduction

1.1 Terminology

1.2 Normative References

1.3 Non-Normative References

2Test Assertions Design and Objectives

2.1 Testing Objectives

2.2 Design Principles

2.3 Anatomy of a Test Assertion

2.4 Coverage of the Specification

2.5 Categories of Test Assertions

3Test Assertions

3.1 Test Assertions for Model Serialization and Integrity

3.1.1 Basic Schema Compliance for Platform Resources, Provider Side

3.1.1.1 Platform Serialization (TA-MOD-xxx)

3.1.1.2 PlatformComponentTemplate Serialization (TA-MOD-xxx)

3.1.1.3 PlatformComponentCapability Serialization (TA-MOD- xxx)

3.1.2 Basic Schema Compliance for Deployed Package Resources

3.1.2.1 AssemblyTemplate Serialization (TA-MOD- xxx)

3.1.2.2 AssemblyComponentTemplate Serialization (TA-MOD- xxx)

3.1.2.3 ApplicationComponentTemplate, ComponentRequirement Serializations (TA-MOD-xxx)

3.1.3 Basic Schema Compliance for Application Resources

3.1.3.1 Assembly Serializations (TA-MOD-xxx)

3.1.3.2 ApplicationComponent Serialization (TA-MOD-xxx)

3.1.3.3 ApplicationComponent , PlatformComponent Serializations (TA-MOD-xxx)

3.1.3.4 Parameter Use (TA-MOD-4-n)

3.1.4 Consistency of Serialization with Metadata

3.1.4.1 Required Attribute Enforcement (Provider) (TA-MOD-xxx)

3.1.4.2 Non-Mutability Enforcement (Provider) (TA-MOD-xxx)

3.1.4.3 Non-Mutability Enforcement (Consumer) (TA-MOD-xxx)

3.1.5 Metadata Integrity and Serialization

3.1.5.1 ParametersDefinitions Serialization and Integrity (TA-MOD-xxx)

3.1.5.2 ParametersDefinition Serialization and Integrity (TA-MOD-xxx)

3.1.5.3 PlatformEndpoints

3.1.5.4 PlatformEndpoint (TA-MOD-xxx)

3.1.5.5 Specification Version value (TA-MOD-xxx)

3.1.5.6 Specification Version consistency (TA-MOD-xxx)

3.1.5.7 Implementation Version value (TA-MOD-xxx)

3.1.5.8 Specification Version consistency (TA-MOD-xxx)

3.1.5.9 Formats Serialization

3.1.5.10 Format Serialization

3.1.5.11 Presence of JSON Format

3.1.5.1 Position of JSON Format

3.1.5.2 Format Serialization and Referential Integrity

3.1.5.3 TypeDefinitions Serialization

3.1.5.4 TypeDefinition Serialization

3.1.5.5 TypeDefinition Serialization and Referential Integrity

3.1.5.6 Attribute Definition Serialization

3.1.5.7 Attribute Definition Serialization and Referential Integrity

3.1.6 Various Model Referential and Semantic Constraints

3.1.6.1 Platform references (TA-MOD-xxx)

3.1.6.2 RepresentationSkew Content (TA-MOD-xxx)

3.1.6.3 ACT Not Shared (TA-MOD-xxx)

3.2 Protocol Test Assertions

3.2.1 Support for JSON

3.2.1.1 JSON media-type (1) (TA-PRO-xxx)

3.2.1.2 Duplicate Keys in JSON, Consumer side (TA-PRO-xxx)

3.2.1.3 Duplicate Keys in JSON, Provider side (TA-PRO-xxx)

3.2.2 Resource Queries and HTTP

3.2.2.1 SelectAttr Positive Provider side (TA-PRO-xxx)

3.2.2.2 SelectAttr Positive Consumer side (TA-PRO-xxx)

3.2.2.3 SelectAttr Negative Case (TA-PRO-xxx)

3.2.3 Media Types Formats and Headers

3.2.4 Query Parameter Mechanism

3.2.5 Resource Updates

3.2.5.1 If-Match HTTP Header in PUT Requests (TA-PRO-xxx)

3.2.5.2 ETag HTTP Header in Responses

3.2.5.3 If-Match semantics

3.2.5.4 PUT semantics, regular, Provider (1)

3.2.5.5 PUT semantics, regular, Provider (2)

3.2.5.6 PUT semantics, regular, Consumer

3.2.5.7 PATCH syntax, Consumer

3.2.5.8 PATCH support, Provider

3.2.5.9 PATCH semantics, Provider

3.3 Resource State Changes and Lifecycle Test Assertions

3.3.1 Resource Creation and Consistency of Representations

3.3.1.1 Creation of AssemblyTemplate from a PDP (TA-STA-xxx)

3.3.1.2 Consistency of PDP content and Created templates (TA-STA-xxx)

3.3.1.3 Consistency of Platform Links after Deployment of a PDP by Reference (TA-STA-xxx)

3.3.1.4 Consistency of Platform Links after Deployment of a PDP by Value (TA-STA-xxx)

3.3.1.1 Effective Creation of an Assembly (1) (TA-STA-xxx)

3.3.1.2 Effective Creation of an Assembly (2) (TA-STA-xxx)

3.3.1.3 Effective Creation of an Assembly and Platform Links (TA-STA-xxx)

3.3.1.4 Effective Creation of an Assembly Component (TA-STA-xxx) (1)

3.3.1.5 Effective Creation of an Assembly Component (TA-STA-xxx) (2)

3.3.2 Resource Integrity

3.3.2.1 Dependencies Resolution (TA-STA-xxx)

3.3.3 Resource Deletion

3.3.3.1 Deletion of an Application Instance (TA-STA-xxx)

3.3.3.2 Deletion of an Application Template (TA-STA-xxx)

3.3.3.3 Deletion of an Application Component Instance (TA-STA-xxx)

3.3.3.4 Deletion of an Application Component Template (TA-STA-xxx)

3.3.4 Component Dependencies Resolution

3.3.5 Resource Export

3.3.6 State Changes of an Application Instance

3.3.6.1 Running State of an Application Instance (TA-STA-xxx)

3.3.6.2 Running state to Suspended State of an Application Instance (TA-STA-xxx)

3.3.6.3 Suspended state to Running State of an Application Instance (TA-STA-xxx)

3.4 PDP Test Assertions

3.4.1 PDP Contents

3.4.1.1 PDP Content, by reference, Consumer side (1) (TA-MOD-xxx)

3.4.1.1 PDP Content, by reference, Consumer side (2) (TA-MOD-xxx)

3.4.1.2 PDP Content, by value, Consumer side (1) (TA-MOD-xxx)

3.4.1.3 PDP Content, by value, Consumer side (2) (TA-MOD-xxx)

3.4.1.1 DP Content, Consumer side (1) (TA-MOD-xxx)

3.4.1.1 DP Content, Consumer side (2) (TA-MOD-xxx)

3.4.2 PDP and DP Registration

3.4.2.1 PDP Registration by reference, Consumer side (TA-MOD-xxx)

3.4.2.1 PDP Registration by value, Consumer side (TA-MOD-xxx)

3.4.2.2 PDP Registration by reference: Provider side (TA-MOD-xxx)

3.4.2.1 PDP Registration by value: Provider side (TA-MOD-xxx)

3.4.2.2 Error Handling: bad PDP Registration (1) (TA-ERR-xxx)

3.4.2.1 Setting DP URI on Registration by Reference (TA-MOD-xxx)

3.4.2.1 Setting DP URI on Registration by Value (TA-MOD-xxx)

3.4.2.1 Setting PDP URI on Registration by Reference (TA-MOD-xxx)

3.4.2.1 Setting PDP URI on Registration by Value (TA-MOD-xxx)

3.4.2.2 Registering PDP by Value with .zip format (TA-STA-xxx)

3.4.2.1 Registering a PDP by Value with .tar format (TA-STA-xxx)

3.4.2.1 Registering a PDP by Value with .tar.gz format (TA-STA-xxx)

3.5 General Error Handling Test Assertions

3.5.1 Serialization Consumer Side

3.5.2 Serialization Provider Side

3.5.2.1 Error Handling: Duplicate Keys in JSON (TA-ERR-nnn)

3.5.3 Unauthorized Operations Provider Side

3.5.3.1 Unauthorized Deletion of PlatformComponentTemplate (TA-ERR-nnn)

4Mapping CAMP Conformance Profiles to Test Assertions

4.1.1 section title

4.1.1.1 section title is usually deepest for Table of Contents

5# Conformance

Appendix A.Acknowledgments

Appendix B.Non-Normative Text

B.1 Subsidiary section

B.1.1 Sub-subsidiary section

Appendix C.Revision History

camp-ta-v1.1-wd01Working Draft 0103 December2012

1Introduction

[All text is normative unless otherwise labeled]

1.1Terminology

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119].

1.2Normative References

[RFC2119]Bradner, S.,“Key words for use in RFCs to Indicate Requirement Levels”, BCP 14, RFC 2119, March 1997.

[Reference] [Full reference citation]

1.3Non-Normative References

[Reference] [Full reference citation]

NOTE: The proper format for citation of technical work produced by an OASIS TC (whether Standards Track or Non-Standards Track) is:

[Citation Label]

Work Product title (italicized). Approval date (DD Month YYYY). OASIS Stage Identifier and Revision Number (e.g., OASIS Committee Specification Draft 01). Principal URI (version-specific URI, e.g., with filename component: somespec-v1.0-csd01.html).

For example:

[OpenDoc-1.2]Open Document Format for Office Applications (OpenDocument) Version 1.2. 19 January 2011. OASIS Committee Specification Draft 07.

[CAP-1.2]Common Alerting Protocol Version 1.2. 01 July 2010. OASIS Standard.

2Test Assertions Design and Objectives

2.1Testing Objectives

A Test Assertions is a testable or measurable expression for evaluating the adherence of an implementation (or part of it) to a normative statement(s) in a specification.

The Test Assertions presented here are intended to help a precise understanding of what is expected from implementations of the CAMP specification.

They are also intended as a starting point for actual test suites. Testing for conformance to the CAMP specification occurs typically in two major contexts:

(1)Product Conformance assessment. In this set-up the primary goal is to assess if a single implementation behaves in conformance to CAMP. A test driver is simulating a Consumer, in case the implementation under test is of a Provider. Conversely, a Consumer may also be tested for conformance, in which case there is no test driver but merely some scenarios that the Consumer is expected to execute along with a Provider. the objective of testing is not always to establish conformance or interoperability. Under conformance testing, is also includedthe assessment of which options are or are not supported by an end-point, even if all mandatory requirements are met.

(2)Product Interoperability test rounds.In this set-up the ability of several implementations to interoperate with each-other is evaluated. A set of implementations is tested pair-wise based on test scenarios between a Consumer and Provider. Even if the exchanges are successful, they must be validated as conforming to CAMP (successful interoperability does not imply successful conformance to CAMP, and vice-versa.)

In both cases, some Consumer-Provider exchanges are executed according to some pre-defined scenarios. In both cases, these exchanges must be verified for conformance to CAMP.

There is also a 3rd context where testing for CAMPconformance is likely to occur: routine regression testing on systems in production. In this context, real business exchanges between a Consumer and a Provider are being monitored and tested for conformance. The rationale is that Cloud end-points may often be upgraded (patches, revisions) or their operational context be modified (configuration change, recovery, etc.), which might cause a deterioration of their conformance to CAMP. While it is not convenient to stop operations and undergo a full round of testing each time some end-point is upgrading, it is often sufficient to detect as early as possible a regression, whenever it occurs.

2.2Design Principles

CAMP test assertions are intended to be usable by test suites designed for the various testing contexts and objectives described in the previous section. To achieve this versatility, the following design principles are used:

The test assertions are expressed with the intent to analyze messages or artifacts exchanged between parties, and will not assume any other test input than these messages or artifacts. This means that if a test requires some additional knowledge about a Provider’s set of resources (such as metadata options, extensions, supported capabilities) then the test assertion is expecting to get this knowledge via some preliminary message exchange – e.g. a query from Consumer about the Provider’s metadata.

Each test assertion is designed to apply to a particular pattern of messages (except for those targeting the PDP artifact itself). The test assertion does not make any assumption about how to produce this pattern of message: it only describes it. A concrete test suite written from such test assertions could enforce (script) the test scenarios that will produce the expected message pattern, or could just discover and analyze such patterns in a message log.

Each test assertion is referring to the normative statement(s) in the original specification that it is addressing, yet is may often re-state the normative requirement in a testable way – i.e. translating these normative requirements in terms of what is a “correct message exchange pattern” to be observed between a Consumer and a Producer. Therefore the actual statement addressed by a test assertion (the “NormativeSource” element) is often a testable interpretation of various normative material from the original specification. Clearly an important aspect of designing test assertions is to derive correct and testable interpretations of the original specification requirements.

2.3Anatomy of a Test Assertion

The structure of test assertions follows the recommendation from OASIS TAG (

The general structure used for Test Assertions is as follows:

The core elements of a Test Assertion are:

Test Assertion ID (or Identifier, or TA_Id)

A unique identifier of the test assertion. It facilitates the mapping of assertions to specification statements. It is recommended that the identifier be made universally unique.

Normative Source(s)

These refer to the precise specification requirement(s) or normative statement(s) that the test assertion addresses.

Target

The target – or test target - categorizes an implementation or a part of an implementation of the referred specification that is the main object of the test assertion and of its Normative Sources. In CAMP, the target will typically be a message exchanged between Consumer and Provider (e.g. a response to an HTTP GET), or the PDP artifact itself.

NOTE: the test target should not be confused with the “conformance target”, often different. The test target is the artifact or component actually subject to the particular testing described in the test assertion – e.g. a message - while the conformance target is the entity ultimately evaluated for conformance, e.g. a CAMP Provider implementation or a Consumer endpoint. Of course, the test target can be traced to a conformance target.

Predicate

A predicate asserts, in the form of an expression that evaluates to either “true” or “false”, the feature (a behavior or a property) described in the specification statement(s) referred by the Normative Sources, concerning an instance of Target. If the predicate evaluates to “true” over the test assertion Target, this means that the Target exhibits this feature. “False” means the Target does not exhibit this feature.

The following parts of a test assertion are optional:

Description

An informal definition of the role of the test assertion, with some optional details on some of its parts. This description must not alter the general meaning of the test assertion and its parts. This description may be used to annotate the test assertion with any information useful to its understanding. It does not need to be an exhaustive description of the test assertion.

Prescription Level

A keyword that indicates how imperative it is that the Normative Statement referred to in the Normative Source, be satisfied. Three prescription levels are used:

Mandatory: this value applies for test assertions which address normative statements that include strongly imperative keywords such as MUST / REQUIRED / shall, but also their negative counterparts (MUST NOT, shall not...) as the Predicate is where the negative aspect is expressed.
Preferred: applies for test assertions which address normative statements that can be interpreted as recommendations, using such keywords as (SHOULD / RECOMMENDED) as well as their negative counterparts.
Permitted: applies for test assertions about normative statements that concern optional features about which the specification remains neutral (no recommendation), e.g. using such keywords ase.g MAY , OPTIONAL, may, need not) or (note: be aware that the negative counterparts “NEED NOT or CAN NOT often have a mandatory character).

Prerequisite

A test assertion Prerequisite is a logical expression (similar to a Predicate) which further qualifies the Target for undergoing the core test (expressed by the Predicate) that addresses the Normative Statement. It may include references to the outcome of other test assertions. If the Prerequisite evaluates to "false" then the Target instance is not qualified for evaluation by the Predicate. Prerequisites have also been called “preconditions”.

In CAMP test assertions, when the Target is a message, the Prerequisite will often represent a sequence of messages that must have occurred, prior to the target message.