Submodule experimental.config.utils.schema | Tarantool
Документация на русском языке
поддерживается сообществом

Submodule experimental.config.utils.schema

Since: 3.2.0

The experimental.config.utils.schema module is used to validate and process parts of cluster configurations that have arbitrary user-defined structures:

The module provides an API to get and set configuration values, filter and transform configuration data, and so on.

Важно

experimental.config.utils.schema is an experimental submodule and is subject to changes.

As an example, consider an application role that has a single configuration option – an HTTP endpoint address.

roles: [ http_api ]
roles_cfg:
  http_api: 'http://127.0.0.1:8080'

This is how you can use the experimental.config.utils.schema module to process the role configuration:

  1. Load the module:

    local schema = require('experimental.config.utils.schema')
    
  2. Define a schema – the root object that stores information about the role’s configuration – using schema.new(). The example below shows a schema that includes a single string option:

    local http_api_schema = schema.new('http_api', schema.scalar({ type = 'string' }))
    

    Learn more in Defining a schema.

  3. Use the validate() method of the schema object to validate configuration values against the schema. In case of a role, call this method inside the role’s validate() function:

    local function validate(cfg)
        http_api_schema:validate(cfg)
    end
    

    Learn more in Validating configuration.

  4. Refer to values of configuration options using the get() method inside the role’s apply() function. Learn more in Getting configuration values.

A configuration schema stores information about a user-defined configuration structure that can be passed inside an app.cfg or a roles_cfg section. It includes option names, types, hierarchy, and other aspects of a configuration.

To create a schema, use the schema.new() function. It has the following arguments:

  • Schema name – an arbitrary string to use as an identifier.
  • Root schema node – a table describing the hierarchical schema structure starting from the root.
  • (Optional) methods – user-defined functions that can be called on this schema object.

Schema nodes describe the hierarchy of options within a schema. There are two types of schema nodes:

  • Scalar nodes hold a single value of a supported primitive type. For example, a string configuration option of a role is a scalar node in its schema.
  • Composite nodes include multiple values in different forms: records, arrays, or maps.

A node can have annotations – named attributes that enable customization of its behavior, for example, setting a default value.

Scalar nodes hold a single value of a primitive type, for example, a string or a number. For the full list of supported scalar types, see Data types.

This configuration has one scalar node of the string type:

roles: [ http_api ]
roles_cfg:
  http_api: 'http://127.0.0.1:8080'

To define a scalar node in a schema, use schema.scalar(). The following schema can be used to process the configuration shown above:

local http_api_schema = schema.new('http_api', schema.scalar({ type = 'string' }))

If a scalar node has a limited set of allowed values, you can also define it with the schema.enum(). Pass the list of allowed values as its argument:

scheme = schema.enum({ 'http', 'https' }),

Примечание

Another way to restrict possible option values is the allowed_values built-in annotation.

Scalar nodes can have the following data types:

Scalar type Lua type Comment
string string  
number number  
integer number Only integer numbers
boolean boolean true or false
string, number or number, string string or number  
any Arbitrary Lua value May be used to declare an arbitrary value that doesn’t need validation.

Record is a composite node that includes a predefined set of other nodes, scalar or composite. In YAML, a record is represented as a node with nested fields. For example, the following configuration has a record node http_api with three scalar fields:

roles: [ http_api ]
roles_cfg:
  http_api:
    host: '127.0.0.1'
    port: 8080
    scheme: 'http'

To define a record node in a schema, use schema.record(). The following schema describes the configuration above:

local listen_address_schema = schema.new('listen_address', schema.record({
    scheme = schema.enum({ 'http', 'https' }),
    host = schema.scalar({ type = 'string' }),
    port = schema.scalar({ type = 'integer' })
}))

Records are also used to define nested schema nodes of non-primitive types. In the example below, the http_api node includes another record listen_address.

roles: [ http_api ]
roles_cfg:
  http_api:
    listen_address:
      host: '127.0.0.1'
      port: 8080
      scheme: 'http'

The following schema describes this configuration:

local listen_address_schema = schema.new('listen_address', schema.record({
    listen_address = schema.record({
        scheme = schema.enum({ 'http', 'https' }),
        host = schema.scalar({ type = 'string' }),
        port = schema.scalar({ type = 'integer' })
    })
}))

Array is a composite node type that includes a collection of items of the same type. The items can be either scalar or composite nodes.

In YAML, array items start with hyphens. For example, the following configuration includes an array named http_api. Each its item is a record with three fields: host, port, and scheme:

roles: [ http-api ]
roles_cfg:
  http-api:
  - host: '127.0.0.1'
    port: 8080
    scheme: 'http'
  - host: '127.0.0.1'
    port: 8443
    scheme: 'https'

To create an array node in a schema, use schema.array(). The following schema describes this configuration:

local listen_address_schema = schema.new('listen_address', schema.array({
    items = schema.record({
        scheme = schema.enum({ 'http', 'https' }),
        host = schema.scalar({ type = 'string' }),
        port = schema.scalar({ type = 'integer' })
    })
}))

There is also the schema.set() function that enables creating arrays with a limited set of allowed items.

Map is a composite node type that holds an arbitrary number of key-value pairs of predefined types.

In YAML, a map is represented as a node with nested fields. For example, the following configuration has the endpoints node:

roles: [ http_api ]
roles_cfg:
  http_api:
    host: '127.0.0.1'
    port: 8080
    scheme: 'http'
    endpoints:
      user: true
      order: true
      customer: false

To create a map node in a schema, use schema.map(). If this node is declared as a map as shown below, the endpoints section can include any number of options with arbitrary names and boolean values.

local listen_address_schema = schema.new('listen_address', schema.record({
    scheme = schema.enum({ 'http', 'https' }),
    host = schema.scalar({ type = 'string' }),
    port = schema.scalar({ type = 'integer' }),
    endpoints = schema.map({ key = schema.scalar({ type = 'string' }),
                             value = schema.scalar({ type = 'boolean' }) })
}))

Node annotations are named attributes that define its various aspects. For example, scalar nodes have a required annotation type that defines the node value type. Other annotations can, for example, set a node’s default value and a validation function, or store arbitrary user-provided data.

Annotations are passed in a table to the node creation function:

scheme = schema.scalar({
    type = 'string',
    allowed_values = { 'http', 'https' },
    default = 'http',
}),

Node annotations fall into three groups:

  • Built-in annotations are handled by the module. These are: type, validate, allowed_values, default and apply_default_if. Note that validate and allowed_values are used for validation only. default and apply_default_if can transform the configuration.
  • User-defined annotations add named node attributes that can be used in the application or role code.
  • Computed annotations allow access to annotations of other nodes throughout the schema.

Built-in annotations are interpreted by the module itself. There are the following built-in annotations:

  • type – the node value type. The type must be explicitly specified for scalar nodes, except for those created with schema.enum(). For composite nodes and scalar enums, the corresponding constructors schema.record(), schema.map(), schema.array(), schema.set(), and schema.enum() set the type automatically.
  • allowed_values – (optional) a list of possible node values.
  • validate – (optional) a validation function for the provided node value.
  • default – (optional) a value to use if the option is not specified in the configuration.
  • apply_default_if – (optional) a function that defines when to apply the default value.

Consider the following role configuration:

roles: [ http_api ]
roles_cfg:
  http_api:
    host: '127.0.0.1'
    port: 8080
    scheme: 'http'

The following schema uses built-in annotations default, allowed_values, and validate to define default and allowed option values and validation functions:

local listen_address_schema = schema.new('listen_address', schema.record({
    scheme = schema.scalar({
        type = 'string',
        allowed_values = { 'http', 'https' },
        default = 'http',
    }),
    host = schema.scalar({
        type = 'string',
        validate = validate_host,
        default = '127.0.0.1',
    }),
    port = schema.scalar({
        type = 'integer',
        validate = validate_port,
        default = 8080,
    }),
}))

Validation functions can look as follows:

local function validate_host(host, w)
    local host_pattern = "^(%d+)%.(%d+)%.(%d+)%.(%d+)$"
    if not host:match(host_pattern) then
        w.error("'host' should be a string containing a valid IP address, got %q", host)
    end
end

local function validate_port(port, w)
    if port <= 1 or port >= 65535 then
        w.error("'port' should be between 1 and 65535, got %d", port)
    end
end

A schema node can have user-defined annotations with arbitrary names. Such annotations are used to implement custom behavior. You can get their names and values from the schema and use in the role or application code.

Example: the env user-defined annotation is used to provide names of environment variables from which the configuration values can be taken.

local listen_address_schema = schema.new('listen_address', schema.record({
    scheme = schema.enum({ 'http', 'https' }, { env = 'HTTP_SCHEME' }),
    host = schema.scalar({ type = 'string', env = 'HTTP_HOST' }),
    port = schema.scalar({ type = 'integer', env = 'HTTP_PORT' })
}))

See the full sample here: Parsing environment variables.

Computed annotations enable access from a node to annotations of its ancestor nodes.

In the example below, the listen_address record validation function refers to the protocol annotation of its ancestor node:

local listen_address = schema.record({
    scheme = schema.enum({ 'http', 'https' }),
    host = schema.scalar({ type = 'string' }),
    port = schema.scalar({ type = 'integer' })
}, {
    validate = function(data, w)
        local protocol = w.schema.computed.annotations.protocol
        if protocol == 'iproto' and data.scheme ~= nil then
            w.error("iproto doesn't support 'scheme'")
        end
    end,
})

Примечание

If there are several ancestor nodes with this annotation, its value is taken from the closest one to the current node.

The following schema with listen_address passes the validation:

local http_listen_address_schema = schema.new('http_listen_address', schema.record({
    name = schema.scalar({ type = 'string' }),
    listen_address = listen_address,
}, {
    protocol = 'http',
}))

If this record is added to a schema with protocol = 'iproto', the listen_address validation fails with an error:

local iproto_listen_address_schema = schema.new('iproto_listen_address', schema.record({
    name = schema.scalar({ type = 'string' }),
    listen_address = listen_address,
}, {
    protocol = 'iproto',
}))

A schema can implement custom logic with methods – user-defined functions that can be called on this schema.

For example, this schema has the format method that returns its fields merged in a URI string:

local listen_address_schema = schema.new(
        "listen_address",
        schema.record(
                {
                    scheme = schema.enum({ "http", "https" }),
                    host = schema.scalar({ type = "string" }),
                    port = schema.scalar({ type = "integer" })
                }
        ),
        {
            methods = {
                format = function(_self, url)
                    return string.format("%s://%s:%d", url.scheme, url.host, url.port)
                end
            }
        }
)

The schema object’s validate() method performs all the necessary checks on the provided configuration. It validates the configuration structure, node types, allowed values, and other aspects of the schema.

When writing roles, call this function inside the role validation function:

local function validate(cfg)
    listen_address_schema:validate(cfg)
end

To get configuration values, use the schema object’s get() method. It takes the configuration and the full path to the node as arguments:

local function apply(cfg)
    local scheme = listen_address_schema:get(cfg, 'listen_address.scheme')
    local host = listen_address_schema:get(cfg, 'listen_address.host')
    local port = listen_address_schema:get(cfg, 'listen_address.port')
    log.info("HTTP API endpoint: %s://%s:%d", scheme, host, port)
end

The schema object has methods that transform configuration data based on the schema, for example, apply_default(), merge(), set().

The following sample shows how to apply default values from the schema to fill missing configuration fields:

local function apply(cfg)
    local cfg_with_defaults = listen_address_schema:apply_default(cfg)
    local scheme = listen_address_schema:get(cfg_with_defaults, 'scheme')
    local host = listen_address_schema:get(cfg_with_defaults, 'host')
    local port = listen_address_schema:get(cfg_with_defaults, 'port')
    log.info("HTTP API endpoint: %s://%s:%d", scheme, host, port)
end

The schema.fromenv() function allows getting configuration values from environment variables. The example below shows how to do this by adding a user-defined annotation env:

local listen_address_schema = schema.new('listen_address', schema.record({
    scheme = schema.enum({ 'http', 'https' }, { env = 'HTTP_SCHEME' }),
    host = schema.scalar({ type = 'string', env = 'HTTP_HOST' }),
    port = schema.scalar({ type = 'integer', env = 'HTTP_PORT' })
}))

local function collect_env_cfg()
    local res = {}
    for _, w in listen_address_schema:pairs() do
        local env_var = w.schema.env
        if env_var ~= nil then
            local value = schema.fromenv(env_var, os.getenv(env_var), w.schema)
            listen_address_schema:set(res, w.path, value)
        end
    end
    return res
end

The function also uses schema object methods:

  • pairs() to iterate over the schema nodes.
  • set() to assign configuration values.

Functions  
schema.array() Create an array node
schema.enum() Create an enum scalar node
schema.fromenv() Parse a value from an environment variable
schema.map() Create a map node
schema.new() Create a schema
schema.record() Create a record node
schema.scalar() Create a scalar node
schema.set() Create a set array node
schema_object  
schema_object:apply_default() Apply default values
schema_object:filter() Filter schema nodes
schema_object:get() Get specified configuration data
schema_object:map() Transform configuration data
schema_object:merge() Merge two configurations
schema_object:pairs() Walk over a configuration
schema_object:set() Set a configuration value
schema_object:validate() Validate a configuration against a schema
schema_object.methods User-defined methods
schema_object.name Schema name
schema_object.schema Schema nodes hierarchy
schema_node_annotation  
allowed_values Allowed node values
apply_default_if Condition to apply defaults
default Default node value
type Value type
validate Validation function
schema_node_object  
schema_node_object.allowed_values Allowed node values
schema_node_object.apply_default_if Condition to apply defaults
schema_node_object.computed Computed annotations
schema_node_object.default Default value
schema_node_object.fields Record node fields
schema_node_object.items Array node items
schema_node_object.type Scalar node type
schema_node_object.validate Validation function

schema.array(array_def)

Create an array node of a configuration schema.

Параметры:
Return:

the created array node as a table with the following fields:

  • type: array
  • items: a table describing an array item as a schema node
  • annotations, if provided in array_def
Rtype:

table

See also: Arrays

schema.enum(allowed_values, annotations)

A shortcut for creating a string scalar node with a limited set of allowed values.

Параметры:
Return:

the created scalar node as a table with the following fields:

  • type: string
  • allowed_values: allowed node values
  • annotations, if annotations is provided
Rtype:

table

See also: Scalar nodes

schema.fromenv(env_var_name, raw_value, schema_node)

Parse an environment variable as a value of the given schema node. The env_var_name parameter is used only for error messages. The value (raw_value) should be received using os.getenv() or os.environ().

How the raw value is parsed depends on the schema_node type:

  • Scalar:
    • string: return the value as is
    • number or integer: parse the value as a number or an integer
    • string, number: attempt to parse as a number; in case of a failure return the value as is
    • boolean: accept true and false (case-insensitively), or 1 and 0 for true and false values correspondingly
    • any: parse the value as a JSON
  • Map: parse either as JSON (if the raw value starts with {) or as a comma-separated string of key=value pairs: key1=value1,key2=value2
  • Array: parse either as JSON (if the raw value starts with [) or as a comma-separated string of items: item1,item2,item3

Примечание

Parsing records from environment variables is not supported.

Параметры:
Return:

the parsed value

Rtype:

table

See also: Parsing environment variables

schema.map(map_def)

Create a map node of a configuration schema.

Параметры:
Return:

the created map node as a table with the following fields:

  • type: map
  • key: map key type
  • value: map value type
  • annotations, if provided in map_def
Rtype:

table

See also: Maps

schema.new(schema_name, schema_node[, { methods = <...> }])

Create a schema object.

Параметры:
  • schema_name (string) – a name
  • schema_node (table) – a root schema node
  • methods (table) – methods
Return:

a new schema object (see schema_object) as a table with the following fields:

  • name: the schema name
  • schema: a table with schema nodes
  • methods: a table with user-provided methods
Rtype:

table

See also: Getting started with config.utils.schema

schema.record(fields[, annotations])

Create a record node of a configuration schema.

Параметры:
  • fields (table) –

    a table of fields in the following format:

    {
        [<field_name>] = <schema_node_object>,
        <...>
    }
    

    See also: schema_node_object.

  • annotations (table) – annotations (see Annotations)
Return:

the created record node as a table with the following fields:

  • type: record
  • fields: a table describing the record fields
  • annotations, if provided
Rtype:

table

See also: Records

schema.scalar(scalar_def)

Create a scalar node of a configuration schema.

Параметры:
Return:

the created scalar node as a table with the following fields:

  • type: the node type (see Data types)
  • annotations, if provided
Rtype:

table

See also: Scalar nodes

schema.set(allowed_values, annotations)

Shortcut for creating an array node of unique string values from the given list of allowed values.

Параметры:
  • allowed_values (table) – allowed values of array items
  • annotations (table) – annotations (see Annotations)
Return:

the created array node as a table with the following fields:

  • type: array
  • items: a table describing an array item as a schema node
  • validate: an auto-generated validation function that checks that the values don’t repeat
  • annotations, if provided
Rtype:

table

See also: Arrays

object schema_object
schema_object:apply_default(data)

Важно

data is assumed to be validated against the given schema.

Apply default values to scalar nodes. The functions takes the default built-in annotation values of the scalar nodes and applies them based on the apply_default_if annotation. If there is no apply_default_if annotation on a node, the default value is also applied.

Примечание

The method works for static defaults. To define a dynamic default value, use the map() method.

Параметры:
  • data (any) – configuration data
Return:

configuration data with applied schema defaults

See also: default, apply_default_if

schema_object:filter(data, f)

Важно

data is assumed to be validated against the given schema.

Filter data based on the schema annotations. The method returns an iterator by configuration nodes for which the given filter function f returns true.

The filter function f receives the following table as the argument:

w = {
    path = <array-like table>,
    schema = <schema node>,
    data = <data at the given path>,
}

The filter function returns a boolean value that is interpreted as «accepted» or «not accepted».

Example:

Calling a function on all schema nodes that have the my_annotation annotation defined:

s:filter(function(w)
    return w.schema.my_annotation ~= nil
end):each(function(w)
    do_something(w.data)
end)
Параметры:
  • data (any) – configuration data
  • f (function) – filter function
Return:

a luafun iterator

schema_object:get(data, path)

Важно

data is assumed to be validated against the given schema.

Get nested configuration values at the given path. The path can be either a dot-separated string (http.scheme) or an array-like table ({ 'http', 'scheme'}).

Example:

local scheme = listen_address_schema:get(cfg, 'listen_address.scheme')
Параметры:
  • data (any) – configuration data
  • path (string/table) –

    path to the target node as:

    • a string in the dot notation
    • an array-like table
Return:

data at the given path

See also: see Getting configuration values

schema_object:map(data, f, f_ctx)

Важно

data is assumed to be validated against the given schema.

Transform data by the given function. The data fields are transformed by the function passed in the second argument (f), while its structure remains unchanged.

The transformation function takes three arguments:

  • data – the configuration data
  • wwalkthrough node with the following fields:
    • w.schema – schema node
    • w.path – the path to the schema node
    • w.error() – a function for printing human-readable error messages
  • ctx – additional context for the transformation function. Can be used to provide values for a specific call.

An example of the transformation function:

local function f(data, w, ctx)
    if w.schema.type == 'string' and data ~= nil then
        return data:gsub('{{ *foo *}}', ctx.foo)
    end
    return data
end

The map() method traverses all fields of the schema records, even if they are nil or box.NULL in the provided configuration. This allows using this method to set computed default values for missing fields. Note that this is not the case for maps and arrays since the schema doesn’t define their fields to traverse.

Параметры:
  • data (any) – configuration data
  • f (function) – transformation function
  • f_ctx (any) – user-provided context for the transformation function
Return:

transformed configuration data

schema_object:merge(data_a, data_b)

Важно

data_a and data_b are assumed to be validated against the given schema.

Merge two configurations. The method merges configurations in a single node hierarchy, preferring the latter in case of a collision.

The following merge rules are used:

  • any present value is preferred over nil and box.NULL

  • box.NULL is preferred over nil

  • for scalar and array nodes, the right-hand value is used

    Примечание

    • Scalars of the any type are merged the same way as other scalars. They are not deeply merged even if they are tables.
    • Arrays are not concatenated. Left hand array items are discarded.
  • records and maps are deeply merged, that is, the merge is performed recursively for their nested nodes

Параметры:
  • data_a (any) – configuration data
  • data_b (any) – configuration data
Return:

merged configuration data

schema_object:pairs()

Walk over the schema and return scalar, array, and map schema nodes

Важно

The method doesn’t return record nodes.

Return:a luafun iterator

Example:

for _, w in schema:pairs() do
    local path = w.path
    local schema = w.schema
    -- <...>
end
schema_object:set(data, path, value)

Важно

data is assumed to be validated against the given schema. value is validated by the method before the assignment.

Set a given value at the given path in a configuration. The path can be either a dot-separated string (http.scheme) or an array-like table ({ 'http', 'scheme'}).

Параметры:
  • data (any) – configuration data
  • path (string/table) –

    path to the target node as:

    • a string in the dot notation
    • an array-like table
  • value (any) – new value
Return:

updated configuration data

Example: see Parsing environment variables

schema_object:validate(data)

Validate data against the schema. If the data doesn’t adhere to the schema, an error is raised.

The method performs the following checks:

  • field type checks: field values are checked against the schema node types
  • allowed values: if a node has the allowed_values annotations of schema nodes, the corresponding data field is checked against the allowed values list
  • validation functions: if a validation function is defined for a node (the validate annotation), it is executed to check that the provided value is valid.
Параметры:
  • data (any) – data

Example: see Annotations and Validating configuration

See also: allowed_values, validate

schema_object.methods

User-defined methods in the schema.

See also: User-defined methods

schema_object.name

Schema name.

schema_object.schema

Schema nodes hierarchy.

See also: schema_node_object

The following elements of tables passed as node constructor arguments are parsed by the modules as built-in annotations:

  • apply_default_if

    A boolean function that defines whether to apply the default value specified using default. If this function returns true on a provided configuration data, the node receives the default value upon the schema_object.apply_default() method call.

    The function takes two arguments:

    • data – the configuration data
    • wwalkthrough node with the following fields:
      • w.schema – schema node
      • w.path – the path to the schema node
      • w.error() – a function for printing human-readable error messages

    See also: schema_object:apply_default()

  • validate

    A function used to validate node data. The function must raise an error to fail the check. The function is called upon the schema_object:validate() function calls.

    The function takes two arguments:

    • data – the configuration data
    • wwalkthrough node with the following fields:
      • w.schema – schema node
      • w.path – the path to the schema node
      • w.error() – a function for printing human-readable error messages

    Example:

    A function that checks that a string is a valid IP address:

    local function validate_host(host, w)
        local host_pattern = "^(%d+)%.(%d+)%.(%d+)%.(%d+)$"
        if not host:match(host_pattern) then
            w.error("'host' should be a string containing a valid IP address, got %q", host)
        end
    end
    

    See also: schema_object:validate()

object schema_node_object
schema_node_object.allowed_values

A list of values allowed for the node. The values are taken from the allowed_values node annotation.

schema_node_object.apply_default_if

A function to define when to apply the default node value. The value is taken from the apply_default_if annotation.

schema_node_object.computed

computed.annotations stores the node’s computed annotations.

schema_node_object.default

Node’s default value. The value is taken from the default annotation.

schema_node_object.fields

Child nodes for record nodes. See also Records.

schema_node_object.items

Node items for array nodes. See also Arrays

schema_node_object.type

Node type for scalar nodes. See Data types

schema_node_object.validate

Node value validation function. The value is taken from the validate annotation.

Нашли ответ на свой вопрос?
Обратная связь