Module metrics

Collectors

Tarantool provides the following metric collectors:

A collector is a representation of one or more observations that change over time.

Labels

A label is a piece of metainfo that you associate with a metric in the key-value format. For details, see labels in Prometheus and tags in Graphite.

Labels are used to differentiate between the characteristics of a thing being measured. For example, in a metric associated with the total number of HTTP requests, you can represent methods and statuses as label pairs:

http_requests_total_counter:inc(1, { method = 'POST', status = '200' })

The example above allows extracting the following time series:

1. The total number of requests over time with method = "POST" (and any status).
2. The total number of requests over time with status = 500 (and any method).

Creating custom metrics

To create a custom metric, follow the steps below:

1. Create a metric

   To create a new metric, you need to call a function corresponding to the desired collector type. For example, call metrics.counter() or metrics.gauge() to create a new counter or gauge, respectively. In the example below, a new counter is created:

   local metrics = require('metrics')
   local bands_replace_count = metrics.counter('bands_replace_count', 'The number of data operations')
   

   This counter is intended to collect the number of data operations performed on the specified space.

   In the next example, a gauge is created:

   local metrics = require('metrics')
   local bands_waste_size = metrics.gauge('bands_waste_size', 'The size of memory wasted due to internal fragmentation')
   

2. Observe a value

   You can observe a value in two ways:

   • At the appropriate place, for example, in an API request handler or trigger. In this example below, the counter value is increased any time a data operation is performed on the bands space. To increase a counter value, counter_obj:inc() is called.

     local metrics = require('metrics')
     local bands_replace_count = metrics.counter('bands_replace_count', 'The number of data operations')
     local trigger = require('trigger')
     trigger.set(
             'box.space.bands.on_replace',
             'update_bands_replace_count_metric',
             function(_, _, _, request_type)
                 bands_replace_count:inc(1, { request_type = request_type })
             end
     )
     

   • At the time of requesting the data collected by metrics. In this case, you need to collect the required metric inside metrics.register_callback(). The example below shows how to use a gauge collector to measure the size of memory wasted due to internal fragmentation:

     local metrics = require('metrics')
     local bands_waste_size = metrics.gauge('bands_waste_size', 'The size of memory wasted due to internal fragmentation')
     metrics.register_callback(function()
         bands_waste_size:set(box.space.bands:stat()['tuple']['memtx']['waste_size'])
     end)
     

     To set a gauge value, gauge_obj:set() is called.

You can find the full example on GitHub: metrics_collect_custom.

Possible limitations

The module allows to add your own metrics, but there are some subtleties when working with specific tools.

When adding your custom metric, it’s important to ensure that the number of label value combinations is kept to a minimum. Otherwise, combinatorial explosion may happen in the timeseries database with metrics values stored. Examples of data labels:

• Labels in Prometheus
• Tags in InfluxDB

For example, if your company uses InfluxDB for metric collection, you can potentially disrupt the entire monitoring setup, both for your application and for all other systems within the company. As a result, monitoring data is likely to be lost.

Example:

local some_metric = metrics.counter('some', 'Some metric')

-- THIS IS POSSIBLE
local function on_value_update(instance_alias)
   some_metric:inc(1, { alias = instance_alias })
end

-- THIS IS NOT ALLOWED
local function on_value_update(customer_id)
   some_metric:inc(1, { customer_id = customer_id })
end

In the example, there are two versions of the function on_value_update. The top version labels the data with the cluster instance’s alias. Since there’s a relatively small number of nodes, using them as labels is feasible. In the second case, an identifier of a record is used. If there are many records, it’s recommended to avoid such situations.

The same principle applies to URLs. Using the entire URL with parameters is not recommended. Use a URL template or the name of the command instead.

In essence, when designing custom metrics and selecting labels or tags, it’s crucial to opt for a minimal set of values that can uniquely identify the data without introducing unnecessary complexity or potential conflicts with existing metrics and systems.

Creating custom plugins

Use the following API to create custom plugins:

• metrics.invoke_callbacks()
• metrics.collectors()
• collector_object

To create a plugin, you need to include the following in your main export function:

-- Invoke all callbacks registered via `metrics.register_callback(<callback-function>)`
metrics.invoke_callbacks()

-- Loop over collectors
for _, c in pairs(metrics.collectors()) do
    ...

    -- Loop over instant observations in the collector
    for _, obs in pairs(c:collect()) do
        -- Export observation `obs`
        ...
    end
end

See the source code of built-in plugins in the metrics GitHub repository.

metrics API

metrics.cfg([config])¶

Entrypoint to setup the module.

Parameters:

config (table) – module configuration options: cfg.include (string/table, default all): all to enable all supported default metrics, none to disable all default metrics, table with names of the default metrics to enable a specific set of metrics. cfg.exclude (table, default {}): a table containing the names of the default metrics that you want to disable. Has higher priority than cfg.include. cfg.labels (table, default {}): a table containing label names as string keys, label values as values. See also: Labels.

You can work with metrics.cfg as a table to read values, but you must call metrics.cfg{} as a function to update them.

Supported default metric names (for cfg.include and cfg.exclude tables):

• all (metasection including all metrics)
• network
• operations
• system
• replicas
• info
• slab
• runtime
• memory
• spaces
• fibers
• cpu
• vinyl
• memtx
• luajit
• clock
• event_loop
• config

See metrics reference for details. All metric collectors from the collection have metainfo.default = true.

cfg.labels are the global labels to be added to every observation.

Global labels are applied only to metric collection. They have no effect on how observations are stored.

Global labels can be changed on the fly.

label_pairs from observation objects have priority over global labels. If you pass label_pairs to an observation method with the same key as some global label, the method argument value will be used.

Note that both label names and values in label_pairs are treated as strings.

metrics.collect([opts])¶

Collect observations from each collector.

Parameters:	opts (table) – table of collect options: invoke_callbacks – if true, invoke_callbacks() is triggered before actual collect. default_only – if true, observations contain only default metrics (metainfo.default = true).

metrics.collectors()¶

List all collectors in the registry. Designed to be used in exporters.

Return:	A list of created collectors (see collector_object).

See also: Creating custom plugins

metrics.counter(name[, help, metainfo])¶

Register a new counter.

Parameters:	name (string) – collector name. Must be unique. help (string) – collector description. metainfo (table) – collector metainfo.
Return:	A counter object (see counter_obj).
Rtype:	counter_obj

See also: Creating custom metrics

metrics.enable_default_metrics([include, exclude])¶

Same as metrics.cfg{include=include, exclude=exclude}, but include={} is treated as include='all' for backward compatibility.

metrics.gauge(name[, help, metainfo])¶

Register a new gauge.

Parameters:	name (string) – collector name. Must be unique. help (string) – collector description. metainfo (table) – collector metainfo.
Return:	A gauge object (see gauge_obj).
Rtype:	gauge_obj

See also: Creating custom metrics

metrics.histogram(name[, help, buckets, metainfo])¶

Register a new histogram.

Parameters:	name (string) – collector name. Must be unique. help (string) – collector description. buckets (table) – histogram buckets (an array of sorted positive numbers). The infinity bucket (INF) is appended automatically. Default: {.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0, 7.5, 10.0, INF}. metainfo (table) – collector metainfo.
Return:	A histogram object (see histogram_obj).
Rtype:	histogram_obj

See also: Creating custom metrics

Note

A histogram is basically a set of collectors:

• name .. "_sum" – a counter holding the sum of added observations.
• name .. "_count" – a counter holding the number of added observations.
• name .. "_bucket" – a counter holding all bucket sizes under the label le (less or equal). To access a specific bucket – x (where x is a number), specify the value x for the label le.

metrics.invoke_callbacks()¶

Invoke all registered callbacks. Has to be called before each collect(). You can also use collect{invoke_callbacks = true} instead. If you’re using one of the default exporters, invoke_callbacks() will be called by the exporter.

See also: Creating custom plugins

metrics.register_callback(callback)¶

Register a function named callback, which will be called right before metric collection on plugin export.

Parameters:	callback (function) – a function that takes no parameters.

This method is most often used for gauge metrics updates.

Example:

local metrics = require('metrics')
local bands_waste_size = metrics.gauge('bands_waste_size', 'The size of memory wasted due to internal fragmentation')
metrics.register_callback(function()
    bands_waste_size:set(box.space.bands:stat()['tuple']['memtx']['waste_size'])
end)

See also: Custom metrics

metrics.set_global_labels(label_pairs)¶

Same as metrics.cfg{ labels = label_pairs }. Learn more in metrics.cfg().

metrics.summary(name[, help, objectives, params, metainfo])¶

Register a new summary. Quantile computation is based on the “Effective computation of biased quantiles over data streams” algorithm.

Parameters:	name (string) – collector name. Must be unique. help (string) – collector description. objectives (table) – a list of “targeted” φ-quantiles in the {quantile = error, ... } form. Example: {[0.5]=0.01, [0.9]=0.01, [0.99]=0.01}. The targeted φ-quantile is specified in the form of a φ-quantile and the tolerated error. For example, {[0.5] = 0.1} means that the median (= 50th percentile) is to be returned with a 10-percent error. Note that percentiles and quantiles are the same concept, except that percentiles are expressed as percentages. The φ-quantile must be in the interval [0, 1]. A lower tolerated error for a φ-quantile results in higher memory and CPU usage during summary calculation. params (table) – table of the summary parameters used to configuring the sliding time window. This window consists of several buckets to store observations. New observations are added to each bucket. After a time period, the head bucket (from which observations are collected) is reset, and the next bucket becomes the new head. This way, each bucket stores observations for max_age_time * age_buckets_count seconds before it is reset. max_age_time sets the duration of each bucket’s lifetime – that is, how many seconds the observations are kept before they are discarded. age_buckets_count sets the number of buckets in the sliding time window. This variable determines the number of buckets used to exclude observations older than max_age_time from the summary. The value is a trade-off between resources (memory and CPU for maintaining the bucket) and how smooth the time window moves. Default value: {max_age_time = math.huge, age_buckets_count = 1}. metainfo (table) – collector metainfo.
Return:	A summary object (see summary_obj).
Rtype:	summary_obj

See also: Creating custom metrics

Note

A summary represents a set of collectors:

• name .. "_sum" – a counter holding the sum of added observations.
• name .. "_count" – a counter holding the number of added observations.
• name holds all the quantiles under observation that find themselves under the label quantile (less or equal). To access bucket x (where x is a number), specify the value x for the label quantile.

metrics.unregister_callback(callback)¶

Unregister a function named callback that is called right before metric collection on plugin export.

Parameters:	callback (function) – a function that takes no parameters.

Example:

local cpu_callback = function()
    local cpu_metrics = require('metrics.psutils.cpu')
    cpu_metrics.update()
end

metrics.register_callback(cpu_callback)

-- after a while, we don't need that callback function anymore

metrics.unregister_callback(cpu_callback)

metrics.http_middleware API

metrics.http_middleware.build_default_collector(type_name, name[, help])¶

Register and return a collector for the middleware.

Parameters:	type_name (string) – collector type: histogram or summary. The default is histogram. name (string) – collector name. The default is http_server_request_latency. help (string) – collector description. The default is HTTP Server Request Latency.
Return:	A collector object

Possible errors:

• A collector with the same type and name already exists in the registry.

metrics.http_middleware.configure_default_collector(type_name, name, help)¶

Register a collector for the middleware and set it as default.

Parameters:	type_name (string) – collector type: histogram or summary. The default is histogram. name (string) – collector name. The default is http_server_request_latency. help (string) – collector description. The default is HTTP Server Request Latency.

Possible errors:

• A collector with the same type and name already exists in the registry.

metrics.http_middleware.get_default_collector()¶

Return the default collector. If the default collector hasn’t been set yet, register it (with default http_middleware.build_default_collector() parameters) and set it as default.

Return:	A collector object

metrics.http_middleware.set_default_collector(collector)¶

Set the default collector.

Parameters:	collector – middleware collector object

metrics.http_middleware.v1(handler, collector)¶

Latency measuring wrap-up for the HTTP ver. 1.x.x handler. Returns a wrapped handler.

Learn more in Collecting HTTP metrics.

Parameters:	handler (function) – handler function. collector – middleware collector object. If not set, the default collector is used (like in http_middleware.get_default_collector()).

Usage:

httpd:route(route, http_middleware.v1(request_handler, collector))

See also: Collecting HTTP metrics

Related objects

object collector_object¶

A collector object.

See also: Creating custom plugins

collector_object:collect()¶

Collect observations from this collector. To collect observations from each collector, use metrics.collectors().

collector_object:collect() is equivalent to the following code:

for _, c in pairs(metrics.collectors()) do
    for _, obs in ipairs(c:collect()) do
        ...  -- handle observation
    end
end

Return:	A concatenation of observation objects across all created collectors. { label_pairs: table, -- `label_pairs` key-value table timestamp: ctype<uint64_t>, -- current system time (in microseconds) value: number, -- current value metric_name: string, -- collector }
Rtype:	table

object counter_obj¶

A counter object.

counter_obj:inc(num, label_pairs)¶

Increment the observation for label_pairs. If label_pairs doesn’t exist, the method creates it.

See also: Labels

Parameters:	num (number) – increment value. label_pairs (table) – table containing label names as keys, label values as values. Note that both label names and values in label_pairs are treated as strings.

counter_obj:collect()¶

Return:	Array of observation objects for a given counter.

{
    label_pairs: table,          -- `label_pairs` key-value table
    timestamp: ctype<uint64_t>,  -- current system time (in microseconds)
    value: number,               -- current value
    metric_name: string,         -- collector
}

Rtype:	table

counter_obj:remove(label_pairs)¶

Remove the observation for label_pairs.

counter_obj:reset(label_pairs)¶

Set the observation for label_pairs to 0.

Parameters:	label_pairs (table) – table containing label names as keys, label values as values. Note that both label names and values in label_pairs are treated as strings.

object gauge_obj¶

gauge_obj:inc(num, label_pairs)¶

Increment the observation for label_pairs. If label_pairs doesn’t exist, the method creates it.

gauge_obj:dec(num, label_pairs)¶

Decrement the observation for label_pairs.

gauge_obj:set(num, label_pairs)¶

Set the observation for label_pairs to num.

gauge_obj:collect()¶

Get an array of observation objects for a given gauge. For the description of observation, see counter_obj:collect().

gauge_obj:remove(label_pairs)¶

Remove the observation for label_pairs.

object histogram_obj¶

histogram_obj:observe(num, label_pairs)¶

Record a new value in a histogram. This increments all bucket sizes under the labels le >= num and the labels that match label_pairs.

Parameters:	num (number) – value to put in the histogram. label_pairs (table) – table containing label names as keys, label values as values. All internal counters that have these labels specified observe new counter values. Note that both label names and values in label_pairs are treated as strings. See also: Labels.

histogram_obj:collect()¶

Return a concatenation of counter_obj:collect() across all internal counters of histogram_obj. For the description of observation, see counter_obj:collect().

histogram_obj:remove(label_pairs)¶

Works like the remove() function of a counter.

object registry¶

registry:unregister(collector)¶

Remove a collector from the registry.

Parameters:	collector (collector_obj) – the collector to be removed.

Example:

local collector = metrics.gauge('some-gauge')

-- after a while, we don't need it anymore

metrics.registry:unregister(collector)

registry:find(kind, name)¶

Find a collector in the registry.

Parameters:	kind (string) – collector kind (counter, gauge, histogram, or summary). name (string) – collector name.
Return:	A collector object or nil.
Rtype:	collector_obj

Example:

local collector = metrics.gauge('some-gauge')

collector = metrics.registry:find('gauge', 'some-gauge')

object summary_obj¶

summary_obj:observe(num, label_pairs)¶

Record a new value in a summary.

Parameters:

num (number) – value to put in the data stream. label_pairs (table) – a table containing label names as keys, label values as values. All internal counters that have these labels specified observe new counter values. You can’t add the "quantile" label to a summary. It is added automatically. If max_age_time and age_buckets_count are set, the observed value is added to each bucket. Note that both label names and values in label_pairs are treated as strings. See also: Labels.

summary_obj:collect()¶

Return a concatenation of counter_obj:collect() across all internal counters of summary_obj. For the description of observation, see counter_obj:collect(). If max_age_time and age_buckets_count are set, quantile observations are collected only from the head bucket in the sliding time window, not from every bucket. If no observations were recorded, the method will return NaN in the values.

summary_obj:remove(label_pairs)¶

Works like the remove() function of a counter.