User’s Guide / Monitoring / API reference
User’s Guide / Monitoring / API reference

API reference

API reference

Collectors

An application using the metrics module has 3 primitives (called “collectors”) at its disposal:

  • Counter
  • Gauge
  • Histogram

A collector represents one or more observations that are changing over time.

Counter

metrics.counter(name[, help])

Registers a new counter.

Parameters:
  • name (string) – Collector name. Must be unique.
  • help (string) – Help description.
Return:

Counter object

Rtype:

counter_obj

object counter_obj
counter_obj:inc(num, label_pairs)

Increments an observation under label_pairs. If label_pairs didn’t exist before, this creates it.

Parameters:
  • num (number) – Increase value.
  • label_pairs (table) – Table containing label names as keys, label values as values.
counter_obj:collect()
Return:Array of observation objects for the given counter.
{
    label_pairs: table,          -- `label_pairs` key-value table
    timestamp: ctype<uint64_t>,  -- current system time (in microseconds)
    value: number,               -- current value
    metric_name: string,         -- collector
}
Rtype:table

Gauge

metrics.gauge(name[, help])

Registers a new gauge. Returns a Counter object.

Parameters:
  • name (string) – Collector name. Must be unique.
  • help (string) – Help description.
Return:

Gauge object

Rtype:

gauge_obj

object gauge_obj
gauge_obj:inc(num, label_pairs)

Same as Counter inc().

gauge_obj:inc(num, label_pairs)

Same as inc(), but decreases the observation.

gauge_obj:set(num, label_pairs)

Same as inc(), but sets the observation.

gauge_obj:collect()

Returns an array of observation objects for the given gauge. For observation description, see counter_obj:collect().

Histogram

metrics.histogram(name[, help, buckets])

Registers a new histogram.

Parameters:
  • name (string) – Collector name. Must be unique.
  • help (string) – Help description.
  • buckets (table) – Histogram buckets (an array of sorted positive numbers). Infinity bucket (INF) is appended automatically. Default is {.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0, 7.5, 10.0, INF}.
Return:

Histogram object

Rtype:

histogram_obj

Note

The histogram is just a set of collectors:

  • name .. "_sum" - A counter holding the sum of added observations. Contains only an empty label set.
  • name .. "_count" - A counter holding number of added observations. Contains only an empty label set.
  • name .. "_bucket" - A counter holding all bucket sizes under the label le (low or equal). So to access a specific bucket x (x is a number), you should specify the value x for the label le.
object histogram_obj

Average

Can be used only as a collector for HTTP statistics (described below) and cannot be built explicitly.

object histogram_obj

Labels

All collectors support providing label_pairs on data modification. Labels are basically a metainfo that you associate with a metric in the format of key-value pairs. See tags in Graphite and labels in Prometheus. Labels are used to differentiate the characteristics of a thing being measured. For example, in a metric associated with the total number of http requests, you can use methods and statuses label pairs:

http_requests_total_counter:inc(1, {method = 'POST', status = '200'})

You don’t have to predefine labels in advance.

Using labels on your metrics allows you to later derive new time series (visualize their graphs) by specifying conditions on label values. In the example above, we could derive these time series:

  1. The total number of requests over time with method = “POST” (and any status).
  2. The total number of requests over time with status = 500 (and any method).

You can also set global labels by calling metrics.set_global_labels({ label = value, ...}).

metrics.enable_default_metrics()

Enables default metrics collections. Collects Tarantool metrics ported from https://github.com/tarantool/stat

Default metrics are:

  • fiber_count - Amount of fibers
  • fiber_csw - Fibers context switches count
  • fiber_memalloc - Fibers memory allocated
  • fiber_memused - Fibers memory used
  • info_lsn - Tarantool log sequence number
  • info_uptime - Tarantool uptime
  • info_memory_k - memory information from box.info.memory()
  • net_sent_total - Totally sent in bytes
  • net_received_total - Totally received in bytes
  • net_sent_rps - Sending RPS
  • net_received_rps - Receive RPS
  • net_connections_rps - Connection RPS (for Tarantool 2.2.0 and higher)
  • net_connections_total - Connections total amount (for Tarantool 2.2.0 and higher)
  • net_connections_current - Current connections amount
  • net_requests_rps - Requests RPS
  • net_requests_total - Requests total amount
  • net_requests_current - Pending requests
  • stats_op_total - Total amount of operations
  • stats_op_rps - Total RPS
  • replication_replica_i_lsn - lsn for replica i
  • replication_master_i_lsn - lsn for master i
  • runtime_k - runtime information from box.runtime.info()
  • slab_k - slab information from box.slab.info()
  • space_index_bsize - Index bsize
  • space_len - Space length (for memtx)
  • space_bsize - Space bsize (for memtx)
  • space_total_bsize - Space total bsize (for memtx)
  • space_count - Space count (for vinyl)
  • cfg_current_time - Tarantool cfg time
metrics.set_global_labels(label_pairs)

Set global labels that will be added to every observation.

Parameters:
  • label_pairs (table) – Table containing label names as string keys, label values as values (table).

Global labels are applied only on metrics collection and have no effect on how observations are stored.

Global labels can be changed on the fly.

Observation label_pairs has priority over global labels: if you pass label_pairs to an observation method with the same key as some global label, the method argument value will be used.

metrics.register_callback(callback)

Registers a function callback which will be called right before metrics collection on plugin export.

Parameters:
  • callback (function) – Function which takes no parameters.

Most common usage is for gauge metrics updates.

Collecting HTTP requests latency statistics

metrics also provides a middleware for monitoring HTTP (set by the http module) latency statistics.

metrics.http_middleware.configure_default_collector(type_name, name, help)

Registers a collector for the middleware and sets it as default.

Parameters:
  • type_name (string) – Collector type: “histogram” or “average”. Default is “histogram”.
  • name (string) – Collector name. Default is “http_server_request_latency”.
  • help (string) – Help description. Default is “HTTP Server Request Latency”.

If a collector with the same type and name already exists in the registry, throws an error.

metrics.http_middleware.build_default_collector(type_name, name[, help])

Registers a collector for the middleware and returns it.

Parameters:
  • type_name (string) – Collector type: “histogram” or “average”. Default is “histogram”.
  • name (string) – Collector name. Default is “http_server_request_latency”.
  • help (string) – Help description. Default is “HTTP Server Request Latency”.

If a collector with the same type and name already exists in the registry, throws an error.

metrics.http_middleware.set_default_collector(collector)

Sets the default collector.

Parameters:
  • collector – Middleware collector object.
metrics.http_middleware.get_default_collector()

Returns the default collector. If the default collector hasn’t been set yet, registers it (with default http_middleware.build_default_collector(...) parameters) and sets it as default.

metrics.http_middleware.v1(handler, collector)

Latency measure wrap-up for HTTP ver. 1.x.x handler. Returns a wrapped handler.

Parameters:
  • handler (function) – Handler function.
  • collector – Middleware collector object. If not set, uses the default collector (like in http_middleware.get_default_collector()).

Usage: httpd:route(route, http_middleware.v1(request_handler, collector))

For a more detailed example, see https://github.com/tarantool/metrics/blob/master/example/HTTP/latency_v1.lua

metrics.http_middleware.v2(collector)

Returns the latency measure middleware for HTTP ver. 2.x.x.

Parameters:
  • collector – Middleware collector object. If not set, uses the default collector (like in http_middleware.get_default_collector()).

Usage:

router = require('http.router').new()
router:route(route, request_handler)
router:use(http_middleware.v2(collector), {name = 'http_instrumentation'}) -- the second argument is optional, see HTTP docs

For a more detailed example, see https://github.com/tarantool/metrics/blob/master/example/HTTP/latency_v2.lua

CPU usage metrics

Collected metrics example

Prometheus query aggregated by thread name

sum by (thread_name) (idelta(tnt_cpu_thread[$__interval]))
  / scalar(idelta(tnt_cpu_total[$__interval]) / tnt_cpu_count)

Examples

Below are examples of using metrics primitives.

Notice that this usage is independent of export-plugins such as Prometheus / Graphite / etc. For documentation on plugins usage, see their the Metrics plugins section.

Using counters:

local metrics = require('metrics')

-- create a counter
local http_requests_total_counter = metrics.counter('http_requests_total')

-- somewhere in the HTTP requests middleware:
http_requests_total_counter:inc(1, {method = 'GET'})

Using gauges:

local metrics = require('metrics')

-- create a gauge
local cpu_usage_gauge = metrics.gauge('cpu_usage', 'CPU usage')

-- register a lazy gauge value update
-- this will be called whenever the export is invoked in any plugins
metrics.register_callback(function()
    local current_cpu_usage = math.random()
    cpu_usage_gauge:set(current_cpu_usage, {app = 'tarantool'})
end)

Using histograms:

local metrics = require('metrics')

-- create a histogram
local http_requests_latency_hist = metrics.histogram(
    'http_requests_latency', 'HTTP requests total', {2, 4, 6})

-- somewhere in the HTTP requests middleware:
local latency = math.random(1, 10)
http_requests_latency_hist:observe(latency)