Router API

Perform the initial cluster bootstrap and distribute all buckets across the replica sets.

Parameters:

• timeout — a number of seconds before ending a bootstrap attempt as unsuccessful. Recreate the cluster in case of bootstrap timeout.

• if_not_bootstrapped — by default is set to false that means raise an error, when the cluster is already bootstrapped. True means consider an already bootstrapped cluster a success.

Example:

 
vshard.router.bootstrap({timeout = 4, if_not_bootstrapped = true})

vshard.router.cfg(cfg)

Configure the database and start sharding for the specified router instance.

• cfg — a configuration table

Create a new router instance. vshard supports multiple routers in a single Tarantool instance. Each router can be connected to any vshard cluster, and multiple routers can be connected to the same cluster.

A router created via vshard.router.new() works in the same way as a static router, but the method name is preceded by a colon (vshard.router:method_name(...)), while for a static router the method name is preceded by a period (vshard.router.method_name(...)).

A static router can be obtained via the vshard.router.static() method and then used like a router created via the vshard.router.new() method.

• name — a router instance name. This name is used as a prefix in logs of the router and must be unique within the instance

• cfg — a configuration table

Return : a router instance, if created successfully; otherwise, nil and an error object

{options})

Call the function identified by function-name on the shard storing the bucket identified by bucket_id. See the Processing requests section for details on function operation.

• bucket_id — a bucket identifier

• mode — either a string = 'read'[|'write', or a map with mode='read'|](##SUBST##|'write', or a map with mode='read'|)'write' and/or prefer_replica=true[|false and/or balance=true|](##SUBST##|false and/or balance=true|)false.

• function_name — a function to execute

• argument_list — an array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds. If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• request_timeout (since vshard 0.1.28) –- timeout in seconds that serves as a protection against hung replicas. The parameter is used in read requests only (mode=read). It is necessary to pass the request_timeout and timeout parameters together, with the following requirement: timeout > request_timeout.

  The request_timeout parameter controls how much time a single request attempt may take. When this time is over (the TimedOut error is raised), the router retries this request on the next replica as long as the timeout value is not elapsed.

• other net.box options, such as is_async, buffer, on_push are also supported.

The mode parameter has two possible forms: a string or a map. Examples of the string form are: 'read', 'write'. Examples of the map form are: {mode='read'}, {mode='write'}, {mode='read', prefer_replica=true}, {mode='read', balance=true}, {mode='read', prefer_replica=true, balance=true}.

If 'write' is specified then the target is the master.

If prefer_replica=true is specified then the preferred target is one of the replicas, but the target is the master if there is no conveniently available replica.

It may be good to specify prefer_replica=true for functions which are expensive in terms of resource use, to avoid slowing down the master.

If balance=true then there is load balancing–-reads are distributed over all the nodes in the replica set in round-robin fashion, with a preference for replicas if prefer_replica=true is also set.

Return

: The original return value of the executed function, or nil and error object. The error object has a type attribute equal to ShardingError or one of the regular Tarantool errors (ClientError, OutOfMemory, SocketError, etc.).

 
`ShardingError` is returned on errors specific for sharding: themaster is missing, wrong bucket id, etc. It has an attribute codecontaining one of the values from the `vshard.error.code.*` LUAtable, an optional attribute containing a message with thehuman-readable error description, and other attributes specific forthe error code.

Examples:

To call customer_add function from vshard/example, say:

 
vshard.router.call(100,                   'write',                   'customer_add',                   {{customer_id = 2, bucket_id = 100, name = 'name2', accounts = {}}},                   {timeout = 5})-- or, the same thing but with a map for the second argumentvshard.router.call(100,                   {mode='write'},                   'customer_add',                   {{customer_id = 2, bucket_id = 100, name = 'name2', accounts = {}}},                   {timeout = 5})

vshard.router.callro(bucket_id, function_name, {argument_list}, {options})

Call the function identified by function-name on the shard storing the bucket identified by bucket_id, in read-only mode (similar to calling vshard.router.call with mode='read'). See the Processing requests section for details on function operation.

• bucket_id — a bucket identifier

• function_name — a function to execute

• argument_list — an array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds.If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• request_timeout (since vshard 0.1.28) –- timeout in seconds that serves as a protection against hung replicas. It is necessary to pass the request_timeout and timeout parameters together, with the following requirement: timeout > request_timeout. The request_timeout parameter controls how much time a single request attempt may take. When this time is over (the TimedOut error is raised), the router retries this request on the next replica as long as the timeout value is not elapsed.

• other net.box options, such as is_async, buffer, on_push are also supported.

Return

The original return value of the executed function, or nil and error object. The error object has a type attribute equal to ShardingError or one of the regular Tarantool errors (ClientError, OutOfMemory, SocketError, etc.).

ShardingError is returned on errors specific for sharding: the replica set is not available, the master is missing, wrong bucket id, etc. It has an attribute code containing one of the values from the vshard.error.code.* LUA table, an optional attribute containing a message with the human-readable error description, and other attributes specific for this error code.

vshard.router.callrw(bucket_id, function_name, {argument_list}, {options})

Call the function identified by function-name on the shard storing the bucket identified by bucket_id, in read-write mode (similar to calling vshard.router.call with mode='write'). See the Processing requests section for details on function operation.

• bucket_id — a bucket identifier

• function_name — a function to execute

• argument_list — an array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds. If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• other net.box options, such as is_async, buffer, on_push are also supported.

Return

The original return value of the executed function, or nil and error object. The error object has a type attribute equal to ShardingError or one of the regular Tarantool errors (ClientError, OutOfMemory, SocketError, etc.).

ShardingError is returned on errors specific for sharding: the replica set is not available, the master is missing, wrong bucket id, etc. It has an attribute code containing one of the values from the vshard.error.code.* LUA table, an optional attribute containing a message with the human-readable error description, and other attributes specific for this error code.

vshard.router.callre(bucket_id, function_name, {argument_list}, {options})

Call the function identified by function-name on the shard storing the bucket identified by bucket_id, in read-only mode (similar to calling vshard.router.call with mode='read'), with preference for a replica rather than a master (similar to calling vshard.router.call with prefer_replica = true). See the Processing requests section for details on function operation.

• bucket_id — a bucket identifier

• function_name — a function to execute

• argument_list — an array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds. If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• request_timeout (since vshard 0.1.28) –- timeout in seconds that serves as a protection against hung replicas. It is necessary to pass the request_timeout and timeout parameters together, with the following requirement: timeout > request_timeout. The request_timeout parameter controls how much time a single request attempt may take. When this time is over (the TimedOut error is raised), the router retries this request on the next replica as long as the timeout value is not elapsed.

• other net.box options, such as is_async, buffer, on_push are also supported.

Return

The original return value of the executed function, or nil and error object. The error object has a type attribute equal to ShardingError or one of the regular Tarantool errors (ClientError, OutOfMemory, SocketError, etc.).

ShardingError is returned on errors specific for sharding: the replica set is not available, the master is missing, wrong bucket id, etc. It has an attribute code containing one of the values from the vshard.error.code.* LUA table, an optional attribute containing a message with the human-readable error description, and other attributes specific for this error code.

vshard.router.callbro(bucket_id, function_name, {argument_list}, {options})

This has the same effect as vshard.router.call() with mode parameter = {mode='read', balance=true}.

vshard.router.callbre(bucket_id, function_name, {argument_list}, {options})

This has the same effect as vshard.router.call() with mode parameter = {mode='read', balance=true, prefer_replica=true}.

The function implements consistent map-reduce over the entire cluster. Consistency means:

• All the data was accessible.
• The data was not migrated between physical storages during the map requests execution.

The function can be helpful if you need to access:

• all the data in the cluster
• a vast number of buckets scattered over the instances in case their individual vshard.router.call() takes up too much time.

The function is called on the master node of each replica set with the given arguments.

• function_name — a function to call on the storages (masters of all replica sets)

• argument_list — an array of the function's arguments

• options — none

• timeout – a request timeout, in seconds. The timeout is for the entire map_callrw(), including all its stages.

• return_raw – the net.box option implemented in Tarantool since version 2.10.0. If set to true, net.box returns the response data wrapped in a MessagePack object instead of decoding it to Lua. For more details, see the Return section below.

Return

• On success: a map with replica set UUIDs (keys) and results of the function_name (values).

   
  {uuid1 = {res1}, uuid2 = {res2}, ...}

  If the function returns nil or box.NULL from one of the storages, it will not be present in the resulting map.

  If the return_raw option is used, the result is a map of the following format: {[replicaset_uuid] = msgpack.object} where msgpack.object is an object that stores a MessagePack array with the results returned from the storage map function.

  The option use case is the same as in using net.box: to avoid decoding of the call results into Lua. The option can be helpful if a router is used as a proxy and results received from a storage are big.

  Example:

   
  local res = vshard.router.map_callrw('my_func', args, {..., return_raw = true})for replicaset_uuid, msgpack_value in pairs(res) do    log.info('Replicaset %s returned %s', replicaset_uuid,             msgpack_value:decode())end

  This is an illustration of the option usage. Normally, you don't need to use return_raw if you call the decode() function.

• On failure: nil, error object, and optional replica set UUID where the error occurred. UUID will not be returned if the error is not related to a particular replica set. For instance, the method fails if not all buckets were found, even if all replica sets were scanned successfully. Handling the result looks like this:

   
  res, err, uuid = vshard.router.map_callrw(...)if not res then    -- Error.    -- 'err' - error object. 'uuid' - optional UUID of replica set    -- where the error happened.    ...else    -- Success.    for uuid, value in pairs(res) do        ...    endend

  If the return_raw option is used, the result on failure is the same as described above.

Map-Reduce in vshard can be divided into three stages: Ref, Map, and Reduce.

Ref and Map. map_callrw() combines both the Ref and the Map stages. The Ref stage ensures data consistency while executing the user's function (function_name) on all nodes. Keep in mind that consistency is incompatible with rebalancing (it breaks data consistency). Map-reduce and rebalancing are mutually exclusive, they compete for the cluster time. Any bucket move makes the sender and receiver nodes inconsistent, so it is impossible to call a function on them to access all the data without vshard.storage.bucket_ref(). It makes the Ref stage intricate, as it should work together with the rebalancer to ensure they do not block each other.

For this, the storage has a special scheduler for bucket moves and storage refs. Storage ref is a volatile counter defined on each instance. It is incremented when a map-reduce request comes and decremented when it ends. Storage ref pins the entire instance with all its buckets, not just a single bucket (like bucket ref).

The scheduler shares storage time between bucket moves and storage refs fairly. The distribution depends on how long and frequent the moves and refs are. It can be configured using the storage options sched_move_quota and sched_ref_quota. Keep in mind that the scheduler configuration may affect map-reduce requests if used during rebalancing.

During the Map stage, map_callrw() sends map requests one by one to many servers. On success, the function returns a map. The map is a set of "key–-value" pairs. The keys are replica set UUIDs, and the values are the results of the user's function–-function_name.

Reduce. The Reduce stage is not performed by vshard. It is what the user's code does with the results of map_callrw().

vshard.router.route(bucket_id)

Return the replica set object for the bucket with the specified bucket id value.

• bucket_id — a bucket identifier

Return : a replica set object

Example:

 
replicaset = vshard.router.route(123)

vshard.router.routeall()

Return all available replica set objects.

Return : a map of the following type: {UUID = replicaset}

Rtype : a map of replica set objects

Example:

 
function selectall()    local resultset = {}    shards, err = vshard.router.routeall()    if err ~= nil then        error(err)    end    for uid, replica in pairs(shards) do        local set = replica:callro('box.space.*space-name*:select', {{}, {limit=10}}, {timeout=5})        for _, item in ipairs(set) do            table.insert(resultset, item)        end    end    table.sort(resultset, function(a, b) return a[1] < b[1] end)    return resultsetend

vshard.router.bucket_id(key)

Deprecated. Logs a warning when used because it is not consistent for cdata numbers.

In particular, it returns 3 different values for normal Lua numbers like 123, for unsigned long long cdata (like 123ULL, or ffi.cast('unsigned long long',123)), and for signed long long cdata (like 123LL, or ffi.cast('long long', 123)). And it is important.

 
vshard.router.bucket_id(123)vshard.router.bucket_id(123LL)vshard.router.bucket_id(123ULL)

For float and double cdata (ffi.cast('float', number), ffi.cast('double', number)) these functions return different values even for the same numbers of the same floating point type. This is because tostring() on a floating point cdata number returns not the number, but a pointer at it. Different on each call.

vshard.router.bucket_id_strcrc32() behaves exactly the same, but does not log a warning. In case you need that behavior.

Calculate the bucket id using a simple built-in hash function.

• key — a hash key. This can be any Lua object (number, table, string).

Return : a bucket identifier

Rtype : number

Example:

 
tarantool> vshard.router.bucket_count()---- 3000...tarantool> vshard.router.bucket_id_strcrc32("18374927634039")---- 2032...tarantool> vshard.router.bucket_id_strcrc32(18374927634039)---- 2032...tarantool> vshard.router.bucket_id_strcrc32("test")---- 1216...tarantool> vshard.router.bucket_id_strcrc32("other")---- 2284...

vshard.router.bucket_id_mpcrc32(key)

This function is safer than bucket_id_strcrc32. It takes a CRC32 from a MessagePack encoded value. That is, bucket id of integers does not depend on their Lua type. In case of a string key, it does not encode it into MessagePack, but takes a hash right from the string.

• key — a hash key. This can be any Lua object (number, table, string).

Return : a bucket identifier

Rtype : number

However it still may return different values for not equal floating point types. That is, ffi.cast('float', number) may be reflected into a bucket id not equal to ffi.cast('double', number). This can't be fixed, because a float value, even being casted to double, may have a garbage tail in its fraction.

Floating point keys should not be used to calculate a bucket id, usually.

Be very careful in case you store floating point types in a space. When data is returned from a space, it is cast to Lua number. And if that value had an empty fraction part, it will be treated as an integer by bucket_id_mpcrc32(). So you need to do explicit casts in such cases. Here is an example of the problem:

 
tarantool> s = box.schema.create_space('test', {format = {{'id', 'double'}}}); _ = s:create_index('pk')---...tarantool> inserted = ffi.cast('double', 1)---...-- Value is stored as doubletarantool> s:replace({inserted})---- [1]...-- But when returned to Lua, stored as Lua number, not cdata.tarantool> returned = s:get({inserted}).id---...tarantool> type(returned), returned---- number- 1...tarantool> vshard.router.bucket_id_mpcrc32(inserted)---- 1411...tarantool> vshard.router.bucket_id_mpcrc32(returned)---- 1614...

vshard.router.bucket_count()

Return the total number of buckets specified in vshard.router.cfg().

Return : the total number of buckets

Rtype : number

 
tarantool> vshard.router.bucket_count()---- 10000...

vshard.router.sync(timeout)

Wait until the dataset is synchronized on replicas.

• timeout — a timeout, in seconds

Returns

true if the dataset was synchronized successfully; or nil and err explaining why the dataset cannot be synchronized.

vshard.router.discovery_wakeup()

Force wakeup of the bucket discovery fiber.

vshard.router.discovery_set(mode)

Turn on/off the background discovery fiber used by the router to find buckets.

• mode — working mode of a discovery fiber. There are three modes: on, off and once

When the mode is on (default), the discovery fiber works during all the lifetime of the router. Even after all buckets are discovered, it will still come to storages and download their buckets with some big period (DISCOVERY_IDLE_INTERVAL). This is useful if the bucket topology changes often and the number of buckets is not big. The router will keep its route table up to date even when no requests are processed.

When the mode is off, discovery is disabled completely.

When the mode is once, discovery starts and finds the locations of all buckets, and then the discovery fiber is terminated. This is good for a large bucket count and for clusters, where rebalancing is rare.

The method is good to enable/disable discovery after the router is already started, but discovery is enabled by default. You may want to never enable it even for a short time–-then specify the discovery_mode option in the configuration. It takes the same values as vshard.router.discovery_set({mode}).

You may decide to turn off discovery or make it once if you have many routers, or tons of buckets (hundreds of thousands and more), and you see that the discovery process consumes notable CPU % on routers and storages. In that case it may be wise to turn off the discovery when there is no rebalancing in the cluster. And turn it on for new routers, as well as for all routers when rebalancing is started.

vshard.router.info({options})

Return information about each instance. Since vshard v.0.1.22, the function also accepts options, which can be used to get additional information.

• options — none

• with_services –- a bool value. If set to true, the function returns information about the background services (such as discovery, master search, or failover) that are working on the current instance.

Return

Replica set parameters:

• replica set uuid
• master instance parameters
• replica instance parameters

Instance parameters:

• uri–-URI of the instance
• uuid–-UUID of the instance
• status–-status of the instance (available, unreachable, missing)
• network_timeout–-a timeout for the request. The value is updated automatically on each 10th successful request and each 2nd failed request.

Bucket parameters:

• available_ro – the number of buckets known to the router and available for read requests
• available_rw – the number of buckets known to the router and available for read and write requests
• unreachable – the number of buckets known to the router but unavailable for any requests
• unknown – the number of buckets whose replica sets are not known to the router

Service parameters:

• name – service name. Possible values: discovery, failover, master_search.
• status – service status. Possible values: ok, error.
• error – error message that appears on the error status.
• activity – service state. It shows what the service is currently doing (for example, updating replicas).
• status_idx – incrementing counter of the status changes. The ok status is updated on every successful iteration of the service. The error status is updated only when it is fixed.

Example:

 
tarantool> vshard.router.info()---- replicasets:    ac522f65-aa94-4134-9f64-51ee384f1a54:      replica: &0        network_timeout: 0.5        status: available        uri: storage@127.0.0.1:3303        uuid: 1e02ae8a-afc0-4e91-ba34-843a356b8ed7      uuid: ac522f65-aa94-4134-9f64-51ee384f1a54      master: *0    cbf06940-0790-498b-948d-042b62cf3d29:      replica: &1        network_timeout: 0.5        status: available        uri: storage@127.0.0.1:3301        uuid: 8a274925-a26d-47fc-9e1b-af88ce939412      uuid: cbf06940-0790-498b-948d-042b62cf3d29      master: *1  bucket:    unreachable: 0    available_ro: 0    unknown: 0    available_rw: 3000  status: 0  alerts: ...tarantool> vshard.router.info({with_services = true})---<all info from vshard.router.info()>  services:    failover:      status_idx: 2      error:      activity: idling      name: failover      status: ok    discovery:      status_idx: 2      error: Error during discovery: TimedOut      activity: idling      name: discovery      status: error...

vshard.router.buckets_info()

Return information about each bucket. Since a bucket map can be huge, only the required range of buckets can be specified.

• offset — the offset in a bucket map of the first bucket to show

• limit — the maximum number of buckets to show

Return : a map of the following type: {bucket_id = 'unknown'/replicaset_uuid}

 
tarantool> vshard.router.buckets_info()---- - uuid: aaaaaaaa-0000-4000-a000-000000000000    status: available_rw  - uuid: aaaaaaaa-0000-4000-a000-000000000000    status: available_rw  - uuid: aaaaaaaa-0000-4000-a000-000000000000    status: available_rw  - uuid: bbbbbbbb-0000-4000-a000-000000000000    status: available_rw  - uuid: bbbbbbbb-0000-4000-a000-000000000000    status: available_rw  - uuid: bbbbbbbb-0000-4000-a000-000000000000    status: available_rw  - uuid: bbbbbbbb-0000-4000-a000-000000000000    status: available_rw...

vshard.router.enable()

Since vshard v.0.1.21. Manually allow access to the router API, revert vshard.router.disable().

vshard.router.disable()

Since vshard v.0.1.21. Manually restrict access to the router API. When the API is disabled, all its methods throw a Lua error, except vshard.router.cfg(), vshard.router.new(), vshard.router.enable() and vshard.router.disable(). The error object's name attribute is ROUTER_IS_DISABLED.

The router is enabled by default. However, it is automatically and forcefully disabled until the configuration is finished, as accessing the router's methods at that time is not safe.

Manual disabling can be used, for example, if some preparatory work needs to be done after calling vshard.router.cfg() but before the router's methods are available. It will look like this:

 
vshard.router.disable()vshard.router.cfg(...)-- Some preparatory work here ...vshard.router.enable()-- vshard.router's methods are available now

replicaset_object method call(function_name, {argument_list}, {options})

Call a function on a nearest available master (distances are defined using replica.zone and cfg.weights matrix) with specified arguments.

• function_name — function to execute

• argument_list — array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds. If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• other net.box options, such as is_async, buffer, on_push are also supported.

Returns

none

• result of function_name on success
• nil, err otherwise

method callrw(function_name, {argument_list}, {options})

Call a function on a nearest available master (distances are defined using replica.zone and cfg.weights matrix) with a specified arguments.

• function_name — function to execute

• argument_list — array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds. If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• other net.box options, such as is_async, buffer, on_push are also supported.

Returns

none

• result of function_name on success
• nil, err otherwise

 
tarantool> local bucket = 1; return vshard.router.callrw(         >     bucket,         >     'box.space.actors:insert',         >     {{         >         1, bucket, 'Renata Litvinova',         >         {theatre="Moscow Art Theatre"}         >     }},         >     {timeout=5}         > )

Call a function on the nearest available replica (distances are defined using replica.zone and cfg.weights matrix) with specified arguments. It is recommended to use replicaset_object:callro() for calling only read-only functions, as the called functions can be executed not only on a master, but also on replicas.

• function_name — function to execute

• argument_list — array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds. If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• other net.box options, such as is_async, buffer, on_push are also supported.

Returns

none

• result of function_name on success
• nil, err otherwise

Call a function on the nearest available replica (distances are defined using replica.zone and cfg.weights matrix) with specified arguments, with preference for a replica rather than a master (similar to calling vshard.router.call with prefer_replica = true). It is recommended to use replicaset_object:callre() for calling only read-only functions, as the called function can be executed not only on a master, but also on replicas.

• function_name — function to execute

• argument_list — array of the function's arguments

• options — none

• timeout –- a request timeout, in seconds. If the router cannot identify a shard with the specified bucket_id, it will retry until the timeout is reached.

• other net.box options, such as is_async, buffer, on_push are also supported.

Returns

none

• result of function_name on success
• nil, err otherwise

vshard.router.master_search_wakeup()

Automated master discovery works in its own fiber on a router, which is activated only if at least one replica set is configured to look for the master (the master parameter is set to auto). The fiber wakes up within a certain period. But it is possible to wake it up on demand by using this function.

Manual fiber wakeup can help speed up tests for master change. Another use case is performing some actions with a router in the router console.

The function does nothing if master search is not configured for any replica set.

Returns

none

vshard.router.bucket_discovery(bucket_id)

Search for the bucket in the whole cluster. If the bucket is not found, it is likely that it does not exist. The bucket might also be moved during rebalancing and currently is in the RECEIVING state.

• bucket_id — a bucket identifier