Module msgpack

The msgpack module decodes raw MsgPack strings by converting them to Lua objects, and encodes Lua objects by converting them to raw MsgPack strings. Tarantool makes heavy internal use of MsgPack because tuples in Tarantool are stored as MsgPack arrays.

Besides, starting from version 2.10.0, the msgpack module enables creating a specific userdata Lua object – MsgPack object. The MsgPack object stores arbitrary MsgPack data, and can be created from any Lua object including another MsgPack object and from a raw MsgPack string. The MsgPack object has its own set of methods and iterators.

Below is a list of msgpack members and related objects.

Convert a Lua object to a raw MsgPack string.

Parameters:

• lua_value — either a scalar value or a Lua table value.

Returns

the original contents formatted as a raw MsgPack string;

Return type

raw MsgPack string

Convert a Lua object to a raw MsgPack string in an ibuf, which is a buffer such as buffer.ibuf() creates. As with encode(lua_value), the result is a raw MsgPack string, but it goes to the ibuf output instead of being returned.

Parameters:

• lua_value (lua-object) — either a scalar value or a Lua table value.

• ibuf (buffer) — (output parameter) where the result raw MsgPack string goes

Returns

number of bytes in the output

Return type

raw MsgPack string

Example using buffer.ibuf() and ffi.string() and string.hex(): The result will be '91a161' because 91 is the MessagePack encoding of "fixarray size 1", a1 is the MessagePack encoding of "fixstr size 1", and 61 is the UTF-8 encoding of 'a':

 
ibuf = require('buffer').ibuf()msgpack_string_size = require('msgpack').encode({'a'}, ibuf)msgpack_string = require('ffi').string(ibuf.rpos, msgpack_string_size)string.hex(msgpack_string)

Convert a raw MsgPack string to a Lua object.

Parameters:

• msgpack_string (string) — a raw MsgPack string.

• start_position (integer) — where to start, minimum = 1, maximum = string length, default = 1.

Returns

none

• (if msgpack_string is a valid raw MsgPack string) the original contents of msgpack_string, formatted as a Lua object, usually a Lua table, (otherwise) a scalar value, such as a string or a number;
• "next_start_position". If decode stops after parsing as far as byte N in msgpack_string, then "next_start_position" will equal N + 1, and decode(msgpack_string, next_start_position) will continue parsing from where the previous decode stopped, plus 1. Normally decode parses all of msgpack_string, so "next_start_position" will equal string.len(msgpack_string) + 1.

Return type

Lua object and number

Example: The result will be ['a'] and 4:

 
msgpack_string = require('msgpack').encode({'a'})require('msgpack').decode(msgpack_string, 1)

Convert a raw MsgPack string, whose address is supplied as a C-style string pointer such as the rpos pointer which is inside an ibuf such as buffer.ibuf() creates, to a Lua object. A C-style string pointer may be described as cdata<char *> or cdata<const char *>.

Parameters:

• C_style_string_pointer (buffer) — a pointer to a raw MsgPack string.

• size (integer) — number of bytes in the raw MsgPack string

Returns

none

• (if C_style_string_pointer points to a valid raw MsgPack string) the original contents of msgpack_string, formatted as a Lua object, usually a Lua table, (otherwise) a scalar value, such as a string or a number;
• returned_pointer = a C-style pointer to the byte after what was passed, so that C_style_string_pointer + size = returned_pointer

Return type

table and C-style pointer to after what was passed

Example using buffer.ibuf and pointer arithmetic: The result will be ['a'] and 3 and true:

 
ibuf = require('buffer').ibuf()msgpack_string_size = require('msgpack').encode({'a'}, ibuf)a, b = require('msgpack').decode(ibuf.rpos, msgpack_string_size)a, b - ibuf.rpos, msgpack_string_size == b - ibuf.rpos

Input and output are the same as for decode(string).

Input and output are the same as for decode(C_style_string_pointer), except that size is not needed. Some checking is skipped, and decode_unchecked(C_style_string_pointer) can operate with string pointers to buffers which decode(C_style_string_pointer) cannot handle. For an example see the buffer module.

Call the MsgPuck's mp_decode_array function and return the array size and a pointer to the first array component. A subsequent call to msgpack_decode can decode the component instead of the whole array.

Parameters:

• byte-array — a pointer to a raw MsgPack string.

• size — a number greater than or equal to the string's length

Returns

none

• the size of the array;
• a pointer to after the array header.

Example:

 
-- Example of decode_array_header-- Suppose we have the raw data '\x93\x01\x02\x03'.-- \x93 is MsgPack encoding for a header of a three-item array.-- We want to skip it and decode the next three items.msgpack = require('msgpack');ffi = require('ffi');x, y = msgpack.decode_array_header(ffi.cast('char*', '\x93\x01\x02\x03'), 4)a = msgpack.decode(y, 1);b = msgpack.decode(y + 1, 1);c = msgpack.decode(y + 2, 1);a, b, c-- The result is: 1,2,3.

Call the MsgPuck's mp_decode_map function and return the map size and a pointer to the first map component. A subsequent call to msgpack_decode can decode the component instead of the whole map.

Parameters:

• byte-array — a pointer to a raw MsgPack string.

• size — a number greater than or equal to the raw MsgPack string's length

Returns

none

• the size of the map;
• a pointer to after the map header.

Example:

 
-- Example of decode_map_header-- Suppose we have the raw data '\x81\xa2\x41\x41\xc3'.-- '\x81' is MsgPack encoding for a header of a one-item map.-- We want to skip it and decode the next map item.msgpack = require('msgpack');ffi = require('ffi')x, y = msgpack.decode_map_header(ffi.cast('char*', '\x81\xa2\x41\x41\xc3'), 5)a = msgpack.decode(y, 3);b = msgpack.decode(y + 3, 1)x, a, b-- The result is: 1,"AA", true.

__serialize parameter

The MsgPack output structure can be specified with the __serialize parameter:

• 'seq', 'sequence', 'array' – table encoded as an array
• 'map', 'mappping' – table encoded as a map
• function – the meta-method called to unpack the serializable representation of table, cdata, or userdata objects

Serializing 'A' and 'B' with different __serialize values brings different results. To show this, here is a routine which encodes {'A','B'} both as an array and as a map, then displays each result in hexadecimal.

 
function hexdump(bytes)    local result = ''    for i = 1, #bytes do        result = result .. string.format("%x", string.byte(bytes, i)) .. ' '    end    return resultendmsgpack = require('msgpack')m1 = msgpack.encode(setmetatable({'A', 'B'}, {                             __serialize = "seq"                          }))m2 = msgpack.encode(setmetatable({'A', 'B'}, {                             __serialize = "map"                          }))print('array encoding: ', hexdump(m1))print('map encoding: ', hexdump(m2))

Result:

 
**array** encoding: 92 a1 41 a1 42**map** encoding:   82 01 a1 41 02 a1 42

The MsgPack Specification page explains that the first encoding means:

 
fixarray(2), fixstr(1), "A", fixstr(1), "B"

and the second encoding means:

 
fixmap(2), key(1), fixstr(1), "A", key(2), fixstr(2), "B"

Here are examples for all the common types, with the Lua-table representation on the left, with the MsgPack format name and encoding on the right.

Common Types and MsgPack Encodings

\Y{0.2}Y{0.8}|

Change MsgPack configuration settings.

The values are all either integers or boolean true/false.

Sparse arrays features

During encoding, the MsgPack encoder tries to classify tables into one of four kinds:

• map - at least one table index is not unsigned integer
• regular array - all array indexes are available
• sparse array - at least one array index is missing
• excessively sparse array - the number of values missing exceeds the configured ratio

An array is excessively sparse when all the following conditions are met:

• encode_sparse_ratio > 0
• max(table) > encode_sparse_safe
• max(table) > count(table) * encode_sparse_ratio

MsgPack encoder never considers an array to be excessively sparse when encode_sparse_ratio = 0. The encode_sparse_safe limit ensures that small Lua arrays are always encoded as sparse arrays. By default, attempting to encode an excessively sparse array generates an error. If encode_sparse_convert is set to true, excessively sparse arrays will be handled as maps.

msgpack.cfg() example 1:

If msgpack.cfg.encode_invalid_numbers = true (the default), then NaN and Inf are legal values. If that is not desirable, then ensure that msgpack.encode() does not accept them, by saying msgpack.cfg{encode_invalid_numbers = false}, thus:

 
tarantool> msgpack = require('msgpack'); msgpack.cfg{encode_invalid_numbers = true}---...tarantool> msgpack.decode(msgpack.encode{1, 0 / 0, 1 / 0, false})---- [1, -nan, inf, false]- 22...tarantool> msgpack.cfg{encode_invalid_numbers = false}---...tarantool> msgpack.decode(msgpack.encode{1, 0 / 0, 1 / 0, false})---- error: ... number must not be NaN or Inf'...

msgpack.cfg() example 2:

To avoid generating errors on attempts to encode unknown data types as userdata/cdata, you can use this code:

 
tarantool> httpc = require('http.client').new()---...tarantool> msgpack.encode(httpc.curl)---- error: unsupported Lua type 'userdata'...tarantool> msgpack.cfg{encode_use_tostring = true}---...tarantool> msgpack.encode(httpc.curl)---- !!binary tnVzZXJkYXRhOiAweDAxMDU5NDQ2Mzg=...

Similar configuration settings exist for JSON and YAML.

A value comparable to Lua "nil" which may be useful as a placeholder in a tuple.

Example

 
tarantool> msgpack = require('msgpack')---...tarantool> y = msgpack.encode({'a',1,'b',2})---...tarantool> z = msgpack.decode(y)---...tarantool> z[1], z[2], z[3], z[4]---- a- 1- b- 2...tarantool> box.space.tester:insert{20, msgpack.NULL, 20}---- [20, null, 20]...

Since: 2.10.0

Encode an arbitrary Lua object into the MsgPack format.

Parameters:

• lua_value (lua-object) — a Lua object of any type.

Returns

encoded MsgPack data encapsulated in a MsgPack object.

Return type

userdata

Example:

 
local msgpack = require('msgpack')-- Create a MsgPack object from a Lua object of any typelocal mp_from_number = msgpack.object(123)local mp_from_string = msgpack.object('hello world')local mp_from_array = msgpack.object({ 10, 20, 30 })local mp_from_table = msgpack.object({ band_name = 'The Beatles', year = 1960 })local mp_from_tuple = msgpack.object(box.tuple.new{1, 'The Beatles', 1960})

Since: 2.10.0

Create a MsgPack object from a raw MsgPack string.

• msgpack_string (string) — a raw MsgPack string.

Returns

a MsgPack object

Return type

userdata

Example:

 
local msgpack = require('msgpack')-- Create a MsgPack object from a raw MsgPack stringlocal raw_mp_string = msgpack.encode({ 10, 20, 30 })local mp_from_mp_string = msgpack.object_from_raw(raw_mp_string)

Since: 2.10.0

Create a MsgPack object from a raw MsgPack string. The address of the MsgPack string is supplied as a C-style string pointer such as the rpos pointer inside an ibuf that the buffer.ibuf() creates. A C-style string pointer may be described as cdata<char *> or cdata<const char *>.

• C_style_string_pointer (buffer) — a pointer to a raw MsgPack string.

• size (integer) — number of bytes in the raw MsgPack string.

Returns

a MsgPack object

Return type

userdata

Example:

 
local msgpack = require('msgpack')-- Create a MsgPack object from a raw MsgPack string using bufferlocal buffer = require('buffer')local ibuf = buffer.ibuf()msgpack.encode({ 10, 20, 30 }, ibuf)local mp_from_mp_string_pt = msgpack.object_from_raw(ibuf.buf, ibuf:size())

Since: 2.10.0

Check if the given argument is a MsgPack object.

• some_agrument — any argument.

Returns

true if the argument is a MsgPack object; otherwise, false

Return type

boolean

Example:

 
local msgpack = require('msgpack')local mp_from_string = msgpack.object('hello world')-- Check if the given argument is a MsgPack objectlocal mp_is_object = msgpack.is_object(mp_from_string) -- Returns truelocal string_is_object = msgpack.is_object('hello world') -- Returns false

msgpack_object A MsgPack object that stores arbitrary MsgPack data. To create a MsgPack object from a Lua object or string, use the following methods:

• msgpack.object
• msgpack.object_from_raw

If a MsgPack object stores an array, it can be inserted into a database space:

 
box.space.bands:insert(msgpack.object({1, 'The Beatles', 1960}))

method decode()

Since: 2.10.0

Decode MsgPack data in the MsgPack object.

Returns

a Lua object

Return type

Lua object

Example

 
local msgpack = require('msgpack')local mp_from_number = msgpack.object(123)local mp_from_string = msgpack.object('hello world')-- Decode MsgPack datalocal mp_number_decoded = mp_from_number:decode() -- Returns 123local mp_string_decoded = mp_from_string:decode() -- Returns 'hello world'

Since: 2.10.0

Create an iterator over the MsgPack data.

Returns

an iterator object over the MsgPack data

Return type

userdata

method msgpack_object[key]

Since: 2.11.0

Get an element of the MsgPack array by the specified index key. You can also use the get(key) method to get an array element.

The index key used to get the array element might be one of the following:

• if a MsgPack object is an array, the key is an integer value (starting with 1) that specifies the element index.
• if a MsgPack object is an associative array, key is the string value that specifies the element key. In this case, you can also access the array element using dot notation (msgpack_object.<key>).

If the specified key is missing in the array, msgpack_object[key] returns nil.

Example

 
local msgpack = require('msgpack')local mp_from_array = msgpack.object({ 10, 20, 30 })local mp_from_table = msgpack.object({ band_name = 'The Beatles', year = 1960 })local mp_from_tuple = msgpack.object(box.tuple.new{1, 'The Beatles', 1960})-- Get MsgPack data by the specified index or keylocal mp_array_get_by_index = mp_from_array[1] -- Returns 10local mp_table_get_by_key = mp_from_table['band_name'] -- Returns 'The Beatles'local mp_table_get_by_nonexistent_key = mp_from_table['rating'] -- Returns nillocal mp_tuple_get_by_index = mp_from_tuple[3] -- Returns 1960

Since: 2.11.0

Get an element of the MsgPack array by the specified index key. You can also use the indexed notation msgpack_object[key] to get an array element.

Parameters:

• key (number/string) — the index key used to get the array element, which might be one of the following:

• if a MsgPack object is an array, the key is an integer value (starting with 1) that specifies the element index.

• if a MsgPack object is an associative array, key is the string value that specifies the element key.

Returns

an element of the MsgPack array. If the specified key is missing in the array, get returns nil.

iterator_object An iterator over a MsgPack array.

method decode_array_header()

Since: 2.10.0

Decode a MsgPack array header under the iterator cursor and advance the cursor. After calling this function, the iterator points to the first element of the array or to the value following the array if the array is empty.

Returns

number of elements in the array

Return type

number

Possible errors: raise an error if the type of the value under the iterator cursor is not MP_ARRAY.

Example

 
local msgpack = require('msgpack')local mp_array = msgpack.object({ 10, 20, 30, 40 })local mp_array_iterator = mp_array:iterator()local size = mp_array_iterator:decode_array_header()  -- returns 4local first = mp_array_iterator:decode()              -- returns 10local second = mp_array_iterator:decode()             -- returns 20mp_array_iterator:skip()                              -- returns none, skips 30local fourth = mp_array_iterator:decode()             -- returns 40

method decode_map_header()

Since: 2.10.0

Decode a MsgPack map header under the iterator cursor and advance the cursor. After calling this function, the iterator points to the first key stored in the map or to the value following the map if the map is empty.

Returns

number of key-value pairs in the map

Return type

number

Possible errors: raise an error if the type of the value under the iterator cursor is not MP_MAP.

Example

 
local msgpack = require('msgpack')local mp_map = msgpack.object({ foo = 123 })local mp_map_iterator = mp_map:iterator()local size = mp_map_iterator:decode_map_header() -- returns 1local first = mp_map_iterator:decode()           -- returns 'foo'local second = mp_map_iterator:decode()          -- returns '123'

method decode()

Since: 2.10.0

Decode a MsgPack value under the iterator cursor and advance the cursor.

Returns

a Lua object corresponding to the MsgPack value

Return type

Lua object

Possible errors: raise a Lua error if there's no data to decode.

Example

 
local msgpack = require('msgpack')local mp_array = msgpack.object({ 10, 20, 30, 40 })local mp_array_iterator = mp_array:iterator()local size = mp_array_iterator:decode_array_header()  -- returns 4local first = mp_array_iterator:decode()              -- returns 10local second = mp_array_iterator:decode()             -- returns 20mp_array_iterator:skip()                              -- returns none, skips 30local fourth = mp_array_iterator:decode()             -- returns 40

method take()

Since: 2.10.0

Return a MsgPack value under the iterator cursor as a MsgPack object without decoding and advance the cursor. The method doesn't copy MsgPack data. Instead, it takes a reference to the original object.

Possible errors: raise a Lua error if there's no data to decode.

Example

 
local msgpack = require('msgpack')local mp_array = msgpack.object({ 10, 20, 30 })local mp_array_iterator = mp_array:iterator()local size = mp_array_iterator:decode_array_header()  -- returns 3local first = mp_array_iterator:decode()              -- returns 10mp_array_iterator:skip()                              -- returns none, skips 20local mp_value_under_cursor = mp_array_iterator:take()local third = mp_value_under_cursor:decode()          -- returns 30

method take_array(count)

Since: 2.10.0

Copy the specified number of MsgPack values starting from the iterator's cursor position to a new MsgPack array object and advance the cursor.

• count (number) — the number of MsgPack values to copy

Returns

a new MsgPack object

Possible errors: raise a Lua error if there aren't enough values to decode. In this case, the iterator's cursor position doesn't change.

Example

 
local msgpack = require('msgpack')local mp_array = msgpack.object({ 10, 20, 30, 40 })local mp_array_iterator = mp_array:iterator()local size = mp_array_iterator:decode_array_header()  -- returns 4local first = mp_array_iterator:decode()              -- returns 10local mp_array_new = mp_array_iterator:take_array(2)local mp_array_new_decoded = mp_array_new:decode()    -- returns {20, 30}local fourth = mp_array_iterator:decode()             -- returns 40

method skip()

Since: 2.10.0

Advance the iterator cursor by skipping one MsgPack value under the cursor. Returns nothing.

Possible errors: raise a Lua error if there's no data to skip.

Example

 
local msgpack = require('msgpack')local mp_array = msgpack.object({ 10, 20, 30, 40 })local mp_array_iterator = mp_array:iterator()local size = mp_array_iterator:decode_array_header()  -- returns 4local first = mp_array_iterator:decode()              -- returns 10local second = mp_array_iterator:decode()             -- returns 20mp_array_iterator:skip()                              -- returns none, skips 30local fourth = mp_array_iterator:decode()             -- returns 40