Serialization

1. Executive Summary

This document describes the standard for JSON serialization of Rune defined model objects. This format includes all relevant object information and improves efficiency, readability, maintainability, and interoperability.

The Rune serialization format is the default for models written in the Rune DSL.

Throughout this document, examples are drawn from the Rune demo model, and comparisons are made between the legacy format (the serialization format used by Rune models before this standard) and the Rune serialization format.

2. Principles

Principles of the design.

2.1 High level Goals

interoperability: users of the same version of the model should be able to exchange data regardless of their programming language
completeness: ability to represent the entire model
readability: serialized data should be human readable
compactness: serialized data should be as compact as possible

2.2 Design reference point

An ordered list of principles to serve as reference when reviewing the serialization design

object generation: serialized data should facilitate creation of Rune defined objects including enabling the "language's" polymorphic inheritance
model conformity: to the fullest extent possible, the serialized data should directly conform to the model. The reader has no obligation to keep fields that it does not recognise
error reporting: report all failures
atomic types serialization: to the extent possible, basic data types such as dates, times and others should be serialised according to well established standards/formats as for example ISO.

Out of scope

cross major version support: the current design does not address enabling serialization to support transformation across breaking changes

Some of the above may not be absolute and the design may have had to make compromises on the extent to which it meets the principles.

3. Overview

The following table illustrates all the special attributes that are provided as part of the serialization process. Please refer to the Implementation section for more details of where and how they are used.

New	Example	Model Syntax	Specification
`@model`	`"@model": "DEMO"`	N/A	See 4.1
`@version`	`"@version": "0.0.1"`	N/A	See 4.1
`@type`	`"@type": "demo.emissions.model.VehicleOwnership"`	N/A	See 4.1
`@data`	`"@data": "attribute-data"`	See 4.3	See 4.3
`@key`	`"@key": "abcd1234"`	See 4.4.1.1	See 4.4.1.1
`@ref`	`"@ref": "abcd1234"`	See 4.4.1.2	See 4.4.1.2
`@key:external`	`"@key:external": "my-external-key"`	See 4.4.1.1	See 4.4.1.1
`@ref:external`	`"@ref:external": "my-external-key"`	See 4.4.1.2	See 4.4.1.2
`@key:scoped`	`"@key:scoped": "my-scoped-key"`	See 4.4.2.1	See 4.4.2.1
`@ref:scoped`	`"@ref:scoped": "my-scoped-key"`	See 4.4.2.2	See 4.4.2.2
`@scheme`	`"@scheme": "urn:iso:std:iso:3166"`	See 4.4.3	See 4.4.3

The Examples section contains illustrations of all the special attributes.

4. Implementation

4.1 Include Model, Version and Type in Top level JSON

The serialized form contains the model, version and fully qualified type name. These will always appear at the top of the JSON.

New	Example	Desc
`@model`	`"@model": "DEMO"`	This is the short name for the model, as defined in a config file (see the demo model's rune-config.yml).
`@type`	`"@type": "demo.emissions.model.VehicleOwnership"`	This is formatted as the namespace followed by the type name in the model with the case matching the model (AKA "Fully Qualified Type Name")
`@version`	`"@version": "0.0.1"`	This is the release of the model, as defined by the model's published release (e.g. a GitHub Release)

{
  "@model" : "demo",
  "@type" : "demo.emissions.model.VehicleOwnership",
  "@version" : "0.0.1"
}

4.2 Include Type when required

When required, for example when a Choice type or Base class is used as an attribute, serialization includes @type to determine the subclass/choice.

For example, the demo model defines Vehicle as a choice between ElectricCar, CombustibleVehicle and Bicycle. When a VehicleOwnership object is serialized, @type identifies which alternative is present:

{
  "vehicle" : {
    "@type" : "demo.emissions.model.Car",
    "specification" : {...},
    "registrationID" : "X2",
    "vehicleTaxBand" : "BBand",
    "vehicleClassification" : "M1_Passengers",
    "fuelCapacity" : 200,
    "engine" : {...},
    "carType" : "Hatchback"
  }
}

4.3 Types with meta

The serialized form reflects that any attribute with a meta annotation (regardless of cardinality or type i.e. basic, complex or enum) will ALWAYS be an object.

This enables consistency, making it easier to understand the serialized format as we will have the same serialization rules for all types i.e.:

Single cardinality basic types
Multi cardinality basic types
Single cardinality complex types
Multi cardinality complex types
Single cardinality enumerations
Multi cardinality enumerations

For basic types and enumerations this will mean the serialized form has an additional wrapper, regardless of whether the meta is included in the data. Where required, the actual data (held in an additional "value" attribute in the legacy format) will now be included in an @data attribute.

For example, if the demo model's DrivingLicence type annotated its dateOfIssuance attribute with [metadata id]:

Rune Definition

type DrivingLicence:
    ...
    dateOfIssuance date (1..1)
        [metadata id]

Legacy format

   "drivingLicence": [
     {
       "dateOfIssuance": {
         "value": "2017-12-18",
         "meta": {
           "globalKey": "3f0b12"
         }
       }
     }
   ]

Rune serialization format

   "drivingLicence": [
     {
       "dateOfIssuance": {
         "@key": "3f0b12",
         "@data": "2017-12-18"
       }
     }
   ]

4.4 Referencing Model

The referencing mechanism in Rune model definitions uses keywords for keys and references. The following table compares how these keywords are serialized in each format:

Legacy Serialised Key / Reference	Rune Serialised Key / Reference
`globalKey` / `globalReference`	`@key` / `@ref`
`location, scope` / `address, scope`	`@key:scoped` / `@ref:scoped`
`externalKey` / `externalReference`	`@key:external` / `@ref:external`

NOTE 1: Where a key is required for a basic type the id annotation is used instead of key i.e. [metadata id] instead of [metadata key]. Both id and key annotations will result in @key being put into the serialized form.

NOTE 2: In the Rune serialization format the location, address and scope annotations now all converge on the use of @key:scoped and @ref:scoped. The external keys and refs (@key:external and @ref:external) will remain for now, but may also be able to be replaced by @key and @ref in the future. Serialization needs to support existing behaviour, whilst paving a way forward so all Rune referencing mechanisms can be unified.

4.4.1 Global/External References

References and external references follow the structure and naming in the model.

In the default implementation @key is a generated hash (as globalKey is in the legacy format) which is intended to be an identifier unique within the document. However, the implementation of globalKey/@key (i.e. how it is generated) can be overridden by the user application.

The external references i.e. externalKey/@key:external are user defined data, from another source.

Serialization is just taking the data in these special attributes, not defining it i.e. the content of @key and @key:external is outside the scope of serialization.

4.4.1.1 Global key

The demo model's VehicleOrderItem type is annotated with [metadata key], so it can be the target of a reference:

Rune Definition

type VehicleOrderItem:
    [metadata key]
    name string (1..1)
    quantity number (1..1)
    price number (1..1)
    currency CurrencyEnum (1..1)

Legacy format

  "orderItem": {
    "meta": {
      "globalKey": "b6bdbfc2",
      "externalKey": "item-1"
    },
    "name" : "OrderItemName001",
    "quantity" : 10,
    "price" : 115.65,
    "currency" : "EUR"
  }

Rune serialization format

  "orderItem" : {
    "@key" : "b6bdbfc2",
    "@key:external" : "item-1",
    "name" : "OrderItemName001",
    "quantity" : 10,
    "price" : 115.65,
    "currency" : "EUR"
  }

4.4.1.2 Global reference

The OrderConfirmation type's orderItemReference attribute is annotated with [metadata reference] and points at an VehicleOrderItem by its key:

Legacy format

    "orderItemReference": {
      "globalReference": "b6bdbfc2",
      "externalReference": "item-1"
    }

Rune serialization format

    "orderItemReference" : {
      "@ref" : "b6bdbfc2",
      "@ref:external" : "item-1"
    }

A multi-cardinality reference attribute is serialized as an array in which each entry is either a reference or a literal value. For example, if the model were extended with a type that keeps track of related orders by listing references to their identifiers:

Rune Definition

type OrderHistory:
    orderIdentifier string (1..*)
        [metadata reference]

Rune serialization format

    "orderIdentifier": [
      {
        "@ref": "gfkldd3k"
      },
      {
        "@data": "ORD-0099"
      }
    ]

The first entry references the @key of an identifier defined elsewhere in the document, while the second holds a literal value.

4.4.2 Scoped references

Scoped references also follow the structure in the model, and use @key:scoped and @ref:scoped.

Scoped references allow specific sections of a document to be referenced. The supported scoped references are:

location - Specifies this is the target of an internal reference i.e. this is the key @key:scoped
address - Specifies that this is an internal reference to an object that appears elsewhere i.e. this is the reference @ref:scoped

The scope annotation allows the scope of the reference to be defined e.g. to a specific type. However, the only scope available is DOCUMENT. This means the scope annotation keyword is not required.

More information on scoped references can be found in the Metadata documentation.

4.4.2.1 Location (key)

The demo model's VehicleOrder type marks its customer attribute as a location, so that other parts of the document can point at it:

Rune Definition

type VehicleOrder:
    ...
    customer Customer (1..1)
        [metadata location]

Legacy format

   "customer": {
     "value": {
        "fullName" : "John Doe"
     },
     "meta": {
        "location": [ {
           "scope": "DOCUMENT",
           "value": "customer-1"
         } ]
      }
   }

Rune serialization format

  "customer" : {
      "@key:scoped" : "customer-1",
      "fullName" : "John Doe"
  }

Where the annotated attribute is a complex type rather than a basic type, @key:scoped is merged into the serialized object alongside its other attributes and no @data wrapper is used — just as @key is in 4.4.1.1.

4.4.2.2 Address (reference)

The OrderConfirmation type's customerReference attribute is an address (i.e. reference) that points at the customer's customer location (i.e. key):

Rune Definition

type OrderConfirmation:
    ...
    customerReference Customer (1..1) 
        [metadata address "pointsTo"=VehicleOrder->customer] 

Legacy format

   "customerReference": {
     "address": {
       "scope": "DOCUMENT",
       "value": "customer-1"
     }
   }

Rune serialization format

   "customerReference": {
     "@ref:scoped": "customer-1"
   }

4.4.3 Scheme

Scheme gives control over the set of values that an attribute can take, without having to define this attribute as an enumeration in the model.

4.4.3.1 Single cardinality attribute with scheme

The demo model's DrivingLicence type annotates its countryOfIssuance attribute with [metadata scheme], allowing the country code to be qualified by the coding scheme it is drawn from:

Legacy format

   "countryOfIssuance": {
     "value": "GB",
     "meta": {
       "scheme": "urn:iso:std:iso:3166"
     }
   }

Rune serialization format

   "countryOfIssuance": {
     "@scheme": "urn:iso:std:iso:3166",
     "@data": "GB"
   }

4.4.3.2 Multiple cardinality (List) attribute, some with scheme

For example, if the driving licence's multi-cardinality vehicleEntitlement attribute were annotated with [metadata scheme], values may individually carry a scheme:

   "vehicleEntitlement": [
     {
      "@scheme": "urn:eu:directive:2007-46-EC",
      "@data": "M1_Passengers"
     },
     {
      "@data": "l3e_Motorcycle"
     }
  ]

5. Processing

The design includes additional enhancements intended to improve the generation and ingestion of the serialized form.

5.1 Generation

These details pertain to how the serialized form is to be generated i.e. the process of serialization.

5.1.1 @key and @ref

If @key is not referenced by an @ref then it will not be included in the serialized form.

This will remove clutter and make the referencing provided by @key/@ref have more value and be easier to use.

5.1.2 @key:external and @ref:external

The @key:external and @ref:external are user defined and will ALWAYS be included if defined.

This means it will be possible to have external keys that do not have a corresponding reference, and vice-versa.

5.1.3 @key:scoped and @ref:scoped

The @key:scoped and @ref:scoped are user defined and will ALWAYS be included if defined.

This means it will be possible to have scoped keys that do not have a corresponding reference, and vice-versa.

5.1.4 Null values

If a value is null then the attribute will not get written out.

If an array is null then it will also not get written out.

5.1.5 Multiple References

There is the confusing potential that an attribute can have more than one reference (i.e. more than one of @ref, @ref:external, and @ref:scoped).

If the multiple references point to different parts of the object, reference resolution and, therefore, validation is not possible. In this case, the outcome of deserialization depends on how strictly to conduct validation. Strict validation will fail. More permissive validation will give notice of conflicting references for later resolution but leave the object accessible in an "invalid" state.

If the multiple references point to the same part of the object, only one is needed. The others are superfluous and will be pruned during serialization according to the following:

@ref:scoped is viewed to have the most "value" and will be preserved in all cases where it is present. All other references will be removed.
If @ref:scoped is not present, @ref:external will be preserved as it is viewed to have greater "value" than @ref.

5.2 Ingestion

These details pertain to how the serialized form is to be ingested i.e. the process of deserialization.

5.2.1 Null values

When a null value is encountered it will be ignored, it will not be processed.

Null arrays will also be ignored.

5.3 Validation

Principles

Follow the Robustness Principle: "be conservative in what you do, be liberal in what you accept from others. It is often reworded as: be conservative in what you send, be liberal in what you accept."
Validation assumes that model is valid.

Validation Process

Check that the string is valid
Check that the string decodes to JSON; and
Check that the specified Rune type can be built from the JSON with three levels of validation:
- The structure is valid
- The types match the model definition
- All relevant constraints are met

5.3.1 Deserialization and Validation

Deserialization will provide a warning and discard any attributes that do not conform to the model.

By default, input data must conform to the constraints a model places on attribute values if and when those constraints exist. There is now a configuration option to relax this requirement in a manner which preserves the Robustness Principle.

The legacy process either fails (Python) or does not give a warning (Java) when it finds non-conforming attributes.

5.3.2 Validation and Serialization

Following the Robustness Principle to enable interoperability, by default an entity must be valid prior to serialization.

Since the legacy process can be less strict, there is now a configuration option to relax this constraint. This option will be marked as deprecated in a timeframe TBD.

5.4 Error and Warning Reporting

Errors and warnings from serialization/deserialization will be logged.

6. Examples

The demo model's demo.emissions.ingestion.json namespace defines a set of types that, together, exercise all of the meta and referencing features described in the previous sections.

An example of the JSON that corresponds to these types is then expressed to help illustrate how the serialized version would look.

Rune Definition

type VehicleOrder:
    orderId string (1..1)
        [metadata id]
    orderItem VehicleOrderItem (1..1)
    customer Customer (1..1)
        [metadata location]
    orderConfirmation OrderConfirmation (1..1)

type VehicleOrderItem:
    [metadata key]
    name string (1..1)
    quantity number (1..1)
    price number (1..1)
    currency CurrencyEnum (1..1)

type Customer:
    userId string (1..1)
        [metadata scheme]
    fullName string (1..1)

type OrderConfirmation:
    orderIdReference string (1..1)
        [metadata reference]
    orderItemReference VehicleOrderItem (1..1)
        [metadata reference]
    customerReference Customer (1..1)
        [metadata address "pointsTo"=VehicleOrder->customer]

Rune Serialized JSON format

{
  "@model" : "demo",
  "@type" : "demo.emissions.ingestion.json.VehicleOrder",
  "@version" : "1.2.3",
  "orderId" : {
    "@key" : "order-id-key",
    "@data" : "order_id_3"
  },
  "orderItem" : {
    "@key" : "order-item-key",
    "name" : "OQOWUEROQIWU001",
    "quantity" : 10,
    "price" : 115.65,
    "currency" : "EUR"
  },
  "customer" : {
    "@key:scoped" : "customer-key",
    "userId" : {
      "@scheme": "https://www.example.com/scheme1",
      "@data": "user_id_3"
    },
    "fullName" : "Laura Gray"
  },
  "orderConfirmation" : {
    "orderIdReference" : {
      "@ref" : "order-id-key"
    },
    "orderItemReference" : {
      "@ref" : "order-item-key"
    },
    "customerReference" : {
      "@ref:scoped" : "customer-key"
    }
  }
}

7. Appendix

Supporting and reference information.

7.1 JSONPath

JSONPath allows navigation through a JSON file. The serialized format that has been implemented was tested against JSONPath. It was found that in all cases except for one JSONPath could successfully traverse the JSON.

The exception case was if a special item was at the top level of the document. In this instance JSONPath failed to locate the item. This was found to be an issue with the JSONPath specification/implementation and had already been reported with error logs here and here

7.2 Backwards compatibility

As a reference, we are using the terminology defined here: Extending and Versioning Languages: Terminology

Key points from that document that relate to our context are:

A language change is backwards compatible if consumers of the revised language can correctly process all instances of the unrevised language. A software example is a word processor at version 5 being able to read and process version 4 documents. A schema example is a schema at version 5 being able to validate version 4 documents.
A language change is forwards compatible if consumers of the unrevised language can correctly process all instances of the revised language. An example is a word processing software at version 4 being able to read and process version 5 documents. A schema example is a schema at version 4 being able to validate version 5 documents.

1. Executive Summary​

2. Principles​

2.1 High level Goals​

2.2 Design reference point​

3. Overview​

4. Implementation​

4.1 Include Model, Version and Type in Top level JSON​

4.2 Include Type when required​

4.3 Types with meta​

4.4 Referencing Model​

4.4.1 Global/External References​

4.4.1.1 Global key​

4.4.1.2 Global reference​

4.4.2 Scoped references​

4.4.2.1 Location (key)​

4.4.2.2 Address (reference)​

4.4.3 Scheme​

4.4.3.1 Single cardinality attribute with scheme​

4.4.3.2 Multiple cardinality (List) attribute, some with scheme​

5. Processing​

5.1 Generation​

5.1.1 @key and @ref​

5.1.2 @key:external and @ref:external​

5.1.3 @key:scoped and @ref:scoped​

5.1.4 Null values​

5.1.5 Multiple References​

5.2 Ingestion​

5.2.1 Null values​

5.3 Validation​

5.3.1 Deserialization and Validation​

5.3.2 Validation and Serialization​

5.4 Error and Warning Reporting​

6. Examples​

7. Appendix​

7.1 JSONPath​

7.2 Backwards compatibility​

1. Executive Summary

2. Principles

2.1 High level Goals

2.2 Design reference point

3. Overview

4. Implementation

4.1 Include Model, Version and Type in Top level JSON

4.2 Include Type when required

4.3 Types with meta

4.4 Referencing Model

4.4.1 Global/External References

4.4.1.1 Global key

4.4.1.2 Global reference

4.4.2 Scoped references

4.4.2.1 Location (key)

4.4.2.2 Address (reference)

4.4.3 Scheme

4.4.3.1 Single cardinality attribute with scheme

4.4.3.2 Multiple cardinality (List) attribute, some with scheme

5. Processing

5.1 Generation

5.1.1 @key and @ref

5.1.2 @key:external and @ref:external

5.1.3 @key:scoped and @ref:scoped

5.1.4 Null values

5.1.5 Multiple References

5.2 Ingestion

5.2.1 Null values

5.3 Validation

5.3.1 Deserialization and Validation

5.3.2 Validation and Serialization

5.4 Error and Warning Reporting

6. Examples

7. Appendix

7.1 JSONPath

7.2 Backwards compatibility