Submodule experimental.config.utils.schema
Since: 3.2.0
The experimental.config.utils.schema
module is used to validate and process
parts of cluster configurations that have arbitrary user-defined structures:
- app.cfg for applications loaded using the app option
- roles_cfg for custom roles developed as a part of a cluster application
The module provides an API to get and set configuration values, filter and transform configuration data, and so on.
Важно
experimental.config.utils.schema
is an experimental submodule and is subject to changes.
As an example, consider an application role that has a single configuration option – an HTTP endpoint address.
roles: [ http_api ]
roles_cfg:
http_api: 'http://127.0.0.1:8080'
This is how you can use the experimental.config.utils.schema
module to process
the role configuration:
Load the module:
local schema = require('experimental.config.utils.schema')
Define a schema – the root object that stores information about the role’s configuration – using schema.new(). The example below shows a schema that includes a single string option:
local http_api_schema = schema.new('http_api', schema.scalar({ type = 'string' }))
Learn more in Defining a schema.
Use the validate() method of the schema object to validate configuration values against the schema. In case of a role, call this method inside the role’s validate() function:
local function validate(cfg) http_api_schema:validate(cfg) end
Learn more in Validating configuration.
Refer to values of configuration options using the get() method inside the role’s apply() function. Learn more in Getting configuration values.
A configuration schema stores information about a user-defined configuration structure that can be passed inside an app.cfg or a roles_cfg section. It includes option names, types, hierarchy, and other aspects of a configuration.
To create a schema, use the schema.new() function. It has the following arguments:
- Schema name – an arbitrary string to use as an identifier.
- Root schema node – a table describing the hierarchical schema structure starting from the root.
- (Optional) methods – user-defined functions that can be called on this schema object.
Schema nodes describe the hierarchy of options within a schema. There are two types of schema nodes:
- Scalar nodes hold a single value of a supported primitive type. For example, a string configuration option of a role is a scalar node in its schema.
- Composite nodes include multiple values in different forms: records, arrays, or maps.
A node can have annotations – named attributes that enable customization of its behavior, for example, setting a default value.
Scalar nodes hold a single value of a primitive type, for example, a string or a number. For the full list of supported scalar types, see Data types.
This configuration has one scalar node of the string
type:
roles: [ http_api ]
roles_cfg:
http_api: 'http://127.0.0.1:8080'
To define a scalar node in a schema, use schema.scalar(). The following schema can be used to process the configuration shown above:
local http_api_schema = schema.new('http_api', schema.scalar({ type = 'string' }))
If a scalar node has a limited set of allowed values, you can also define it with the schema.enum(). Pass the list of allowed values as its argument:
scheme = schema.enum({ 'http', 'https' }),
Примечание
Another way to restrict possible option values is the allowed_values built-in annotation.
Scalar nodes can have the following data types:
Scalar type | Lua type | Comment |
---|---|---|
string |
string |
|
number |
number |
|
integer |
number |
Only integer numbers |
boolean |
boolean |
true or false |
string, number
or
number, string |
string or number |
|
any |
Arbitrary Lua value | May be used to declare an arbitrary value that doesn’t need validation. |
Record is a composite node that includes a predefined set of other nodes, scalar
or composite. In YAML, a record is represented as a node with nested fields.
For example, the following configuration has a record node http_api
with
three scalar fields:
roles: [ http_api ]
roles_cfg:
http_api:
host: '127.0.0.1'
port: 8080
scheme: 'http'
To define a record node in a schema, use schema.record(). The following schema describes the configuration above:
local listen_address_schema = schema.new('listen_address', schema.record({
scheme = schema.enum({ 'http', 'https' }),
host = schema.scalar({ type = 'string' }),
port = schema.scalar({ type = 'integer' })
}))
Records are also used to define nested schema nodes of non-primitive types. In the example
below, the http_api
node includes another record listen_address
.
roles: [ http_api ]
roles_cfg:
http_api:
listen_address:
host: '127.0.0.1'
port: 8080
scheme: 'http'
The following schema describes this configuration:
local listen_address_schema = schema.new('listen_address', schema.record({
listen_address = schema.record({
scheme = schema.enum({ 'http', 'https' }),
host = schema.scalar({ type = 'string' }),
port = schema.scalar({ type = 'integer' })
})
}))
Array is a composite node type that includes a collection of items of the same type. The items can be either scalar or composite nodes.
In YAML, array items start with hyphens. For example, the following configuration
includes an array named http_api
. Each its item is a record with three fields:
host
, port
, and scheme
:
roles: [ http-api ]
roles_cfg:
http-api:
- host: '127.0.0.1'
port: 8080
scheme: 'http'
- host: '127.0.0.1'
port: 8443
scheme: 'https'
To create an array node in a schema, use schema.array(). The following schema describes this configuration:
local listen_address_schema = schema.new('listen_address', schema.array({
items = schema.record({
scheme = schema.enum({ 'http', 'https' }),
host = schema.scalar({ type = 'string' }),
port = schema.scalar({ type = 'integer' })
})
}))
There is also the schema.set() function that enables creating arrays with a limited set of allowed items.
Map is a composite node type that holds an arbitrary number of key-value pairs of predefined types.
In YAML, a map is represented as a node with nested fields.
For example, the following configuration has the endpoints
node:
roles: [ http_api ]
roles_cfg:
http_api:
host: '127.0.0.1'
port: 8080
scheme: 'http'
endpoints:
user: true
order: true
customer: false
To create a map node in a schema, use schema.map().
If this node is declared as a map as shown below, the endpoints
section can include
any number of options with arbitrary names and boolean values.
local listen_address_schema = schema.new('listen_address', schema.record({
scheme = schema.enum({ 'http', 'https' }),
host = schema.scalar({ type = 'string' }),
port = schema.scalar({ type = 'integer' }),
endpoints = schema.map({ key = schema.scalar({ type = 'string' }),
value = schema.scalar({ type = 'boolean' }) })
}))
Node annotations are named attributes that define its various aspects. For example,
scalar nodes have a required annotation type
that defines the node value type.
Other annotations can, for example, set a node’s default value and a validation function,
or store arbitrary user-provided data.
Annotations are passed in a table to the node creation function:
scheme = schema.scalar({
type = 'string',
allowed_values = { 'http', 'https' },
default = 'http',
}),
Node annotations fall into three groups:
- Built-in annotations are handled by the module. These are:
type
,validate
,allowed_values
,default
andapply_default_if
. Note thatvalidate
andallowed_values
are used for validation only.default
andapply_default_if
can transform the configuration. - User-defined annotations add named node attributes that can be used in the application or role code.
- Computed annotations allow access to annotations of other nodes throughout the schema.
Built-in annotations are interpreted by the module itself. There are the following built-in annotations:
- type – the node value type.
The type must be explicitly specified for scalar nodes, except for those created with
schema.enum()
. For composite nodes and scalar enums, the corresponding constructorsschema.record()
,schema.map()
,schema.array()
,schema.set()
, andschema.enum()
set the type automatically. - allowed_values – (optional) a list of possible node values.
- validate – (optional) a validation function for the provided node value.
- default – (optional) a value to use if the option is not specified in the configuration.
- apply_default_if – (optional) a function that defines when to apply the default value.
Consider the following role configuration:
roles: [ http_api ]
roles_cfg:
http_api:
host: '127.0.0.1'
port: 8080
scheme: 'http'
The following schema uses built-in annotations default
, allowed_values
, and validate
to define default and allowed option values and validation functions:
local listen_address_schema = schema.new('listen_address', schema.record({
scheme = schema.scalar({
type = 'string',
allowed_values = { 'http', 'https' },
default = 'http',
}),
host = schema.scalar({
type = 'string',
validate = validate_host,
default = '127.0.0.1',
}),
port = schema.scalar({
type = 'integer',
validate = validate_port,
default = 8080,
}),
}))
Validation functions can look as follows:
local function validate_host(host, w)
local host_pattern = "^(%d+)%.(%d+)%.(%d+)%.(%d+)$"
if not host:match(host_pattern) then
w.error("'host' should be a string containing a valid IP address, got %q", host)
end
end
local function validate_port(port, w)
if port <= 1 or port >= 65535 then
w.error("'port' should be between 1 and 65535, got %d", port)
end
end
A schema node can have user-defined annotations with arbitrary names. Such annotations are used to implement custom behavior. You can get their names and values from the schema and use in the role or application code.
Example: the env
user-defined annotation is used to provide names
of environment variables from which the configuration values can be taken.
local listen_address_schema = schema.new('listen_address', schema.record({
scheme = schema.enum({ 'http', 'https' }, { env = 'HTTP_SCHEME' }),
host = schema.scalar({ type = 'string', env = 'HTTP_HOST' }),
port = schema.scalar({ type = 'integer', env = 'HTTP_PORT' })
}))
See the full sample here: Parsing environment variables.
Computed annotations enable access from a node to annotations of its ancestor nodes.
In the example below, the listen_address
record validation function refers to the
protocol
annotation of its ancestor node:
local listen_address = schema.record({
scheme = schema.enum({ 'http', 'https' }),
host = schema.scalar({ type = 'string' }),
port = schema.scalar({ type = 'integer' })
}, {
validate = function(data, w)
local protocol = w.schema.computed.annotations.protocol
if protocol == 'iproto' and data.scheme ~= nil then
w.error("iproto doesn't support 'scheme'")
end
end,
})
Примечание
If there are several ancestor nodes with this annotation, its value is taken from the closest one to the current node.
The following schema with listen_address
passes the validation:
local http_listen_address_schema = schema.new('http_listen_address', schema.record({
name = schema.scalar({ type = 'string' }),
listen_address = listen_address,
}, {
protocol = 'http',
}))
If this record is added to a schema with protocol = 'iproto'
, the listen_address
validation fails with an error:
local iproto_listen_address_schema = schema.new('iproto_listen_address', schema.record({
name = schema.scalar({ type = 'string' }),
listen_address = listen_address,
}, {
protocol = 'iproto',
}))
A schema can implement custom logic with methods – user-defined functions that can be called on this schema.
For example, this schema has the format
method that returns its fields merged in a URI string:
local listen_address_schema = schema.new(
"listen_address",
schema.record(
{
scheme = schema.enum({ "http", "https" }),
host = schema.scalar({ type = "string" }),
port = schema.scalar({ type = "integer" })
}
),
{
methods = {
format = function(_self, url)
return string.format("%s://%s:%d", url.scheme, url.host, url.port)
end
}
}
)
The schema object’s validate() method performs all the necessary checks on the provided configuration. It validates the configuration structure, node types, allowed values, and other aspects of the schema.
When writing roles, call this function inside the role validation function:
local function validate(cfg)
listen_address_schema:validate(cfg)
end
To get configuration values, use the schema object’s get() method. It takes the configuration and the full path to the node as arguments:
local function apply(cfg)
local scheme = listen_address_schema:get(cfg, 'listen_address.scheme')
local host = listen_address_schema:get(cfg, 'listen_address.host')
local port = listen_address_schema:get(cfg, 'listen_address.port')
log.info("HTTP API endpoint: %s://%s:%d", scheme, host, port)
end
The schema object has methods that transform configuration data based on the schema, for example, apply_default(), merge(), set().
The following sample shows how to apply default values from the schema to fill missing configuration fields:
local function apply(cfg)
local cfg_with_defaults = listen_address_schema:apply_default(cfg)
local scheme = listen_address_schema:get(cfg_with_defaults, 'scheme')
local host = listen_address_schema:get(cfg_with_defaults, 'host')
local port = listen_address_schema:get(cfg_with_defaults, 'port')
log.info("HTTP API endpoint: %s://%s:%d", scheme, host, port)
end
The schema.fromenv() function allows getting
configuration values from environment variables. The example below shows how to do
this by adding a user-defined annotation env
:
local listen_address_schema = schema.new('listen_address', schema.record({
scheme = schema.enum({ 'http', 'https' }, { env = 'HTTP_SCHEME' }),
host = schema.scalar({ type = 'string', env = 'HTTP_HOST' }),
port = schema.scalar({ type = 'integer', env = 'HTTP_PORT' })
}))
local function collect_env_cfg()
local res = {}
for _, w in listen_address_schema:pairs() do
local env_var = w.schema.env
if env_var ~= nil then
local value = schema.fromenv(env_var, os.getenv(env_var), w.schema)
listen_address_schema:set(res, w.path, value)
end
end
return res
end
The function also uses schema object methods:
Functions | |
schema.array() | Create an array node |
schema.enum() | Create an enum scalar node |
schema.fromenv() | Parse a value from an environment variable |
schema.map() | Create a map node |
schema.new() | Create a schema |
schema.record() | Create a record node |
schema.scalar() | Create a scalar node |
schema.set() | Create a set array node |
schema_object | |
schema_object:apply_default() | Apply default values |
schema_object:filter() | Filter schema nodes |
schema_object:get() | Get specified configuration data |
schema_object:map() | Transform configuration data |
schema_object:merge() | Merge two configurations |
schema_object:pairs() | Walk over a configuration |
schema_object:set() | Set a configuration value |
schema_object:validate() | Validate a configuration against a schema |
schema_object.methods | User-defined methods |
schema_object.name | Schema name |
schema_object.schema | Schema nodes hierarchy |
schema_node_annotation | |
allowed_values | Allowed node values |
apply_default_if | Condition to apply defaults |
default | Default node value |
type | Value type |
validate | Validation function |
schema_node_object | |
schema_node_object.allowed_values | Allowed node values |
schema_node_object.apply_default_if | Condition to apply defaults |
schema_node_object.computed | Computed annotations |
schema_node_object.default | Default value |
schema_node_object.fields | Record node fields |
schema_node_object.items | Array node items |
schema_node_object.type | Scalar node type |
schema_node_object.validate | Validation function |
-
schema.
array
(array_def)¶ Create an array node of a configuration schema.
Параметры: - array_def (
table
) –a table in the following format:
{ items = <schema_node_object>, <..annotations..> }
See also: schema_node_object, schema_node_annotation.
Return: the created array node as a table with the following fields:
type
:array
items
: a table describing an array item as a schema node- annotations, if provided in
array_def
Rtype: See also: Arrays
- array_def (
-
schema.
enum
(allowed_values, annotations)¶ A shortcut for creating a string scalar node with a limited set of allowed values.
Параметры: - allowed_values (
table
) – a list of enum members – values allowed for the node - annotations (
table
) – annotations (see schema_node_annotation)
Return: the created scalar node as a table with the following fields:
type
:string
allowed_values
: allowed node values- annotations, if
annotations
is provided
Rtype: See also: Scalar nodes
- allowed_values (
-
schema.
fromenv
(env_var_name, raw_value, schema_node)¶ Parse an environment variable as a value of the given schema node. The
env_var_name
parameter is used only for error messages. The value (raw_value
) should be received usingos.getenv()
oros.environ()
.How the raw value is parsed depends on the
schema_node
type:- Scalar:
string
: return the value as isnumber
orinteger
: parse the value as a number or an integerstring, number
: attempt to parse as a number; in case of a failure return the value as isboolean
: accepttrue
andfalse
(case-insensitively), or1
and0
fortrue
andfalse
values correspondinglyany
: parse the value as a JSON
- Map: parse either as JSON (if the raw value starts with
{
) or as a comma-separated string ofkey=value
pairs:key1=value1,key2=value2
- Array: parse either as JSON (if the raw value starts with
[
) or as a comma-separated string of items:item1,item2,item3
Примечание
Parsing records from environment variables is not supported.
Параметры: - env_var_name (
string
) – environment variable name to use for error messages - raw_value (
string
) – environment variable value - schema_node (
schema_node_object
) – a schema node (see schema_node_object)
Return: the parsed value
Rtype: See also: Parsing environment variables
- Scalar:
-
schema.
map
(map_def)¶ Create a map node of a configuration schema.
Параметры: - map_def (
table
) –a table in the following format:
{ key = <schema_node_object>, value = <schema_node_object>, <..annotations..> }
See also: schema_node_object, schema_node_annotation.
Return: the created map node as a table with the following fields:
type
:map
key
: map key typevalue
: map value type- annotations, if provided in
map_def
Rtype: See also: Maps
- map_def (
-
schema.
new
(schema_name, schema_node[, { methods = <...> }])¶ Create a schema object.
Параметры: Return: a new schema object (see schema_object) as a table with the following fields:
name
: the schema nameschema
: a table with schema nodesmethods
: a table with user-provided methods
Rtype: See also: Getting started with config.utils.schema
-
schema.
record
(fields[, annotations])¶ Create a record node of a configuration schema.
Параметры: - fields (
table
) –a table of fields in the following format:
{ [<field_name>] = <schema_node_object>, <...> }
See also: schema_node_object.
- annotations (
table
) – annotations (see Annotations)
Return: the created record node as a table with the following fields:
type
:record
fields
: a table describing the record fields- annotations, if provided
Rtype: See also: Records
- fields (
-
schema.
scalar
(scalar_def)¶ Create a scalar node of a configuration schema.
Параметры: - scalar_def (
table
) –a table in the following format:
{ type = <scalar_type>, <..annotations..> }
See also: schema_node_object, schema_node_annotation.
- type (
string
) – data type (see Data types) - annotations (
table
) – annotations (see Annotations)
Return: the created scalar node as a table with the following fields:
type
: the node type (see Data types)- annotations, if provided
Rtype: See also: Scalar nodes
- scalar_def (
-
schema.
set
(allowed_values, annotations)¶ Shortcut for creating an array node of unique string values from the given list of allowed values.
Параметры: - allowed_values (
table
) – allowed values of array items - annotations (
table
) – annotations (see Annotations)
Return: the created array node as a table with the following fields:
type
:array
items
: a table describing an array item as a schema nodevalidate
: an auto-generated validation function that checks that the values don’t repeat- annotations, if provided
Rtype: See also: Arrays
- allowed_values (
-
object
schema_object
¶ -
schema_object:
apply_default
(data)¶ Важно
data
is assumed to be validated against the given schema.Apply default values to scalar nodes. The functions takes the
default
built-in annotation values of the scalar nodes and applies them based on theapply_default_if
annotation. If there is noapply_default_if
annotation on a node, the default value is also applied.Примечание
The method works for static defaults. To define a dynamic default value, use the map() method.
Параметры: - data (
any
) – configuration data
Return: configuration data with applied schema defaults
See also: default, apply_default_if
- data (
-
schema_object:
filter
(data, f)¶ Важно
data
is assumed to be validated against the given schema.Filter data based on the schema annotations. The method returns an iterator by configuration nodes for which the given filter function
f
returnstrue
.The filter function
f
receives the following table as the argument:w = { path = <array-like table>, schema = <schema node>, data = <data at the given path>, }
The filter function returns a boolean value that is interpreted as «accepted» or «not accepted».
Example:
Calling a function on all schema nodes that have the
my_annotation
annotation defined:s:filter(function(w) return w.schema.my_annotation ~= nil end):each(function(w) do_something(w.data) end)
Параметры: - data (
any
) – configuration data - f (
function
) – filter function
Return: a luafun iterator
- data (
-
schema_object:
get
(data, path)¶ Важно
data
is assumed to be validated against the given schema.Get nested configuration values at the given path. The path can be either a dot-separated string (
http.scheme
) or an array-like table ({ 'http', 'scheme'}
).Example:
local scheme = listen_address_schema:get(cfg, 'listen_address.scheme')
Параметры: - data (
any
) – configuration data - path (
string/table
) –path to the target node as:
- a string in the dot notation
- an array-like table
Return: data at the given path
See also: see Getting configuration values
- data (
-
schema_object:
map
(data, f, f_ctx)¶ Важно
data
is assumed to be validated against the given schema.Transform data by the given function. The
data
fields are transformed by the function passed in the second argument (f
), while its structure remains unchanged.The transformation function takes three arguments:
data
– the configuration dataw
– walkthrough node with the following fields:w.schema
– schema nodew.path
– the path to the schema nodew.error()
– a function for printing human-readable error messages
ctx
– additional context for the transformation function. Can be used to provide values for a specific call.
An example of the transformation function:
local function f(data, w, ctx) if w.schema.type == 'string' and data ~= nil then return data:gsub('{{ *foo *}}', ctx.foo) end return data end
The
map()
method traverses all fields of the schema records, even if they arenil
orbox.NULL
in the provided configuration. This allows using this method to set computed default values for missing fields. Note that this is not the case for maps and arrays since the schema doesn’t define their fields to traverse.Параметры: - data (
any
) – configuration data - f (
function
) – transformation function - f_ctx (
any
) – user-provided context for the transformation function
Return: transformed configuration data
-
schema_object:
merge
(data_a, data_b)¶ Важно
data_a
anddata_b
are assumed to be validated against the given schema.Merge two configurations. The method merges configurations in a single node hierarchy, preferring the latter in case of a collision.
The following merge rules are used:
any present value is preferred over
nil
andbox.NULL
box.NULL
is preferred overnil
for scalar and array nodes, the right-hand value is used
Примечание
- Scalars of the
any
type are merged the same way as other scalars. They are not deeply merged even if they are tables. - Arrays are not concatenated. Left hand array items are discarded.
- Scalars of the
records and maps are deeply merged, that is, the merge is performed recursively for their nested nodes
Параметры: - data_a (
any
) – configuration data - data_b (
any
) – configuration data
Return: merged configuration data
-
schema_object:
pairs
()¶ Walk over the schema and return scalar, array, and map schema nodes
Важно
The method doesn’t return record nodes.
Return: a luafun iterator Example:
for _, w in schema:pairs() do local path = w.path local schema = w.schema -- <...> end
-
schema_object:
set
(data, path, value)¶ Важно
data
is assumed to be validated against the given schema.value
is validated by the method before the assignment.Set a given value at the given path in a configuration. The path can be either a dot-separated string (
http.scheme
) or an array-like table ({ 'http', 'scheme'}
).Параметры: - data (
any
) – configuration data - path (
string/table
) –path to the target node as:
- a string in the dot notation
- an array-like table
- value (
any
) – new value
Return: updated configuration data
Example: see Parsing environment variables
- data (
-
schema_object:
validate
(data)¶ Validate data against the schema. If the data doesn’t adhere to the schema, an error is raised.
The method performs the following checks:
- field type checks: field values are checked against the schema node types
- allowed values: if a node has the
allowed_values
annotations of schema nodes, the corresponding data field is checked against the allowed values list - validation functions: if a validation function is defined for a node
(the
validate
annotation), it is executed to check that the provided value is valid.
Параметры: - data (
any
) – data
Example: see Annotations and Validating configuration
See also: allowed_values, validate
-
schema_object.
methods
¶ User-defined methods in the schema.
See also: User-defined methods
-
schema_object.
name
¶ Schema name.
-
schema_object.
schema
¶ Schema nodes hierarchy.
See also: schema_node_object
-
The following elements of tables passed as node constructor arguments are parsed by the modules as built-in annotations:
allowed_values
A list of allowed values for a node.
See also: schema_object:validate()
apply_default_if
A boolean function that defines whether to apply the default value specified using
default
. If this function returnstrue
on a provided configuration data, the node receives the default value upon the schema_object.apply_default() method call.The function takes two arguments:
data
– the configuration dataw
– walkthrough node with the following fields:w.schema
– schema nodew.path
– the path to the schema nodew.error()
– a function for printing human-readable error messages
See also: schema_object:apply_default()
default
A default value to use for a scalar node if it’s not specified explicitly.
Example: see Transforming configuration
See also: schema_object:apply_default()
type
A schema node type.
See also: Data types
validate
A function used to validate node data. The function must raise an error to fail the check. The function is called upon the schema_object:validate() function calls.
The function takes two arguments:
data
– the configuration dataw
– walkthrough node with the following fields:w.schema
– schema nodew.path
– the path to the schema nodew.error()
– a function for printing human-readable error messages
Example:
A function that checks that a string is a valid IP address:
local function validate_host(host, w) local host_pattern = "^(%d+)%.(%d+)%.(%d+)%.(%d+)$" if not host:match(host_pattern) then w.error("'host' should be a string containing a valid IP address, got %q", host) end end
See also: schema_object:validate()
-
object
schema_node_object
¶ -
schema_node_object.
allowed_values
¶ A list of values allowed for the node. The values are taken from the allowed_values node annotation.
-
schema_node_object.
apply_default_if
¶ A function to define when to apply the default node value. The value is taken from the apply_default_if annotation.
-
schema_node_object.
computed
¶ computed.annotations
stores the node’s computed annotations.
-
schema_node_object.
type
¶ Node type for scalar nodes. See Data types
-