Schema Assertion#
The schema
assertion verifies that the output of a mapping conforms to a specific schema. This may be useful at the
end of complex transformation chains to ensure a given data interface contract is met.
Example:#
kind: schema
mapping: some_mapping
ignoreTypes: true
columns:
col_1: string
col_2: int
kind: schema
mapping: some_mapping
ignoreNullability: false
schema:
kind: inline
fields:
- name: col_1
type: string
- name: col_2
type: int
A more complete example using the validate
target might look as follows:
targets:
validate_export_schema:
kind: validate
assertions:
- kind: schema
mapping: some_mapping
ignoreNullability: true
ignoreCase: false
ignoreOrder: true
schema:
kind: avro
file: ${project.basedir}/schema/export-schema.avsc
perform_export:
kind: relation
relation: export_relation
mapping: some_mapping
jobs:
export:
targets:
- validate_export_schema
- perform_export
This example will validate that the result of the mapping some_mapping
actually conforms to a schema in some Avro
schema definition. Since the validation is performed in the VALIDATE
phase, even before the CREATE
and BUILD
phase,
the job will fail if the schema doesn’t match and the export as defined in the target perform_export
won’t be
performed.
Of course, you can also use the schema
assertion within the Flowman testing framework,
such that the test will not be performed during the real execution of an export job, but as part of an integration
test, which is typically conducted only as part of a continuous integration (CI) process.
Fields#
kind
(mandatory) (type: string):schema
description
(optional) (type: string): A textual description of the assertionmapping
(required) (type: string): The name of the mapping which is to be tested.columns
(optional) (type: map:string): Specifies the list of column names (key) with their type (value)schema
(optional) (type: schema): As an alternative of specifying a list of columns you can also directly specify a schema.ignoreTypes
(optional) (type: boolean) (default: false): Ignore types, only field names will be compared.ignoreNullability
(optional) (type: boolean) (default: false): Ignore nullability of fields, only field names and types will be compared.ignoreCase
(optional) (type: boolean) (default: false): Ignore upper/lower case of field names.ignoreOrder
(optional) (type: boolean) (default: false): Ignore the order of fields. The schema matches the given schema as long as all fields are present. Otherwise, the order of all fields also has to match (which is the default)