Модуль msgpack

Общие сведения

The msgpack module decodes raw MsgPack strings by converting them to Lua objects, and encodes Lua objects by converting them to raw MsgPack strings. Tarantool makes heavy internal use of MsgPack because tuples in Tarantool are stored as MsgPack arrays.

Definitions: MsgPack is short for MessagePack. A «raw MsgPack string» is a byte array formatted according to the MsgPack specification including type bytes and sizes. The type bytes and sizes can be made displayable with string.hex(), or the raw MsgPack strings can be converted to Lua objects with msgpack methods.

Указатель

Ниже приведен перечень всех функций и элементов модуля msgpack.

msgpack.encode(lua_value)

Convert a Lua object to a raw MsgPack string.

Параметры:	lua_value – скалярное значение или значение из Lua-таблицы.
возвращает:	the original contents formatted as a raw MsgPack string;
тип возвращаемого значения:
	raw MsgPack string

msgpack.encode(lua_value, ibuf)

Convert a Lua object to a raw MsgPack string in an ibuf, which is a buffer such as buffer.ibuf() creates. As with encode(lua_value), the result is a raw MsgPack string, but it goes to the ibuf output instead of being returned.

Параметры:	lua_value (lua-object) – скалярное значение или значение из Lua-таблицы. ibuf (buffer) – (output parameter) where the result raw MsgPack string goes
возвращает:	number of bytes in the output
тип возвращаемого значения:
	raw MsgPack string

Example using buffer.ibuf() and ffi.string() and string.hex(): The result will be „91a161“ because 91 is the MessagePack encoding of «fixarray size 1», a1 is the MessagePack encoding of «fixstr size 1», and 61 is the UTF-8 encoding of „a“:

ibuf = require('buffer').ibuf()
msgpack_string_size = require('msgpack').encode({'a'}, ibuf)
msgpack_string = require('ffi').string(ibuf.rpos, msgpack_string_size)
string.hex(msgpack_string)

msgpack.decode(msgpack_string[, start_position])

Convert a raw MsgPack string to a Lua object.

Параметры:	msgpack_string (string) – a raw MsgPack string. start_position (integer) – откуда начинать, минимальное значение = 1, максимальное = длина строки, по умолчанию = 1.
возвращает:	(if msgpack_string is a valid raw MsgPack string) the original contents of msgpack_string, formatted as a Lua object, usually a Lua table, (otherwise) a scalar value, such as a string or a number; «next_start_position». Если расшифровка decode останавливается после разбора байта N в msgpack_string, то «next_start_position» = N + 1, а decode(msgpack_string, next_start_position) продолжит разбор с места остановки предыдущего decode плюс 1. Как правило, decode разбирает всю строку msgpack_string, поэтому «next_start_position» будет равняться string.len(msgpack_string) + 1.
тип возвращаемого значения:
	Lua object and number

Example: The result will be [„a“] and 4:

msgpack_string = require('msgpack').encode({'a'})
require('msgpack').decode(msgpack_string, 1)

msgpack.decode(C_style_string_pointer, size)

Convert a raw MsgPack string, whose address is supplied as a C-style string pointer such as the rpos pointer which is inside an ibuf such as buffer.ibuf() creates, to a Lua object. A C-style string pointer may be described as cdata<char *> or cdata<const char *>.

Параметры:	C_style_string_pointer (buffer) – a pointer to a raw MsgPack string. size (integer) – number of bytes in the raw MsgPack string
возвращает:	(if C_style_string_pointer points to a valid raw MsgPack string) the original contents of msgpack_string, formatted as a Lua object, usually a Lua table, (otherwise) a scalar value, such as a string or a number; returned_pointer = a C-style pointer to the byte after what was passed, so that C_style_string_pointer + size = returned_pointer
тип возвращаемого значения:
	table and C-style pointer to after what was passed

Example using buffer.ibuf and pointer arithmetic: The result will be [„a“] and 3 and true:

ibuf = require('buffer').ibuf()
msgpack_string_size = require('msgpack').encode({'a'}, ibuf)
a, b = require('msgpack').decode(ibuf.rpos, msgpack_string_size)
a, b - ibuf.rpos, msgpack_string_size == b - ibuf.rpos

msgpack.decode_unchecked(msgpack_string[, start_position])

Input and output are the same as for decode(string).

msgpack.decode_unchecked(C_style_string_pointer)

Input and output are the same as for decode(C_style_string_pointer), except that size is not needed. Some checking is skipped, and decode_unchecked(C_style_string_pointer) can operate with string pointers to buffers which decode(C_style_string_pointer) cannot handle. For an example see the buffer module.

msgpack.decode_array_header(byte-array, size)

Call the mp_decode_array function in the MsgPuck library and return the array size and a pointer to the first array component. A subsequent call to msgpack_decode can decode the component instead of the whole array.

Параметры:	byte-array – a pointer to a raw MsgPack string. size – a number greater than or equal to the string’s length
возвращает:	the size of the array; a pointer to after the array header.

-- Example of decode_array_header
-- Suppose we have the raw data '\x93\x01\x02\x03'.
-- \x93 is MsgPack encoding for a header of a three-item array.
-- We want to skip it and decode the next three items.
msgpack=require('msgpack'); ffi=require('ffi')
x,y=msgpack.decode_array_header(ffi.cast('char*','\x93\x01\x02\x03'),4)
a=msgpack.decode(y,1);b=msgpack.decode(y+1,1);c=msgpack.decode(y+2,1);
a,b,c
-- The result will be: 1,2,3.

msgpack.decode_map_header(byte-array, size)

Call the mp_decode_map function in the MsgPuck library and return the map size and a pointer to the first map component. A subsequent call to msgpack_decode can decode the component instead of the whole map.

Параметры:	byte-array – a pointer to a raw MsgPack string. size – a number greater than or equal to the raw MsgPack string’s length
возвращает:	the size of the map; a pointer to after the map header.

-- Example of decode_map_header
-- Suppose we have the raw data '\x81\xa2\x41\x41\xc3'.
-- \x81 is MsgPack encoding for a header of a one-item map.
-- We want to skip it and decode the next map item.
msgpack=require('msgpack'); ffi=require('ffi')
x,y=msgpack.decode_map_header(ffi.cast('char*','\x81\xa2\x41\x41\xc3'),5)
a=msgpack.decode(y,3);b=msgpack.decode(y+3,1)
x,a,b
-- The result will be: 1,"AA", true.

__serialize parameter:

Структуру MsgPack-вывода можно указать с помощью __serialize:

• „seq“, „sequence“, „array“ - table encoded as an array
• „map“, „mappping“ - table encoded as a map
• function - the meta-method called to unpack serializable representation of table, cdata or userdata objects

Serializing „A“ and „B“ with different __serialize values brings different results. To show this, here is a routine which encodes {'A','B'} both as an array and as a map, then displays each result in hexadecimal.

function hexdump(bytes)
    local result = ''
    for i = 1, #bytes do
        result = result .. string.format("%x", string.byte(bytes, i)) .. ' '
    end
    return result
end

msgpack = require('msgpack')
m1 = msgpack.encode(setmetatable({'A', 'B'}, {
                             __serialize = "seq"
                          }))
m2 = msgpack.encode(setmetatable({'A', 'B'}, {
                             __serialize = "map"
                          }))
print('array encoding: ', hexdump(m1))
print('map encoding: ', hexdump(m2))

Результат:

array encoding: 92 a1 41 a1 42
map encoding:   82 01 a1 41 02 a1 42

The MsgPack Specification page explains that the first encoding means:

fixarray(2), fixstr(1), "A", fixstr(1), "B"

а значение второго результата кодирования:

fixmap(2), key(1), fixstr(1), "A", key(2), fixstr(2), "B"

Ниже приведены примеры всех стандартных типов: слева отображение в Lua-таблице, а справа – имя и кодировка в формате MsgPack.

msgpack.cfg(table)

Some MsgPack configuration settings can be changed.

The values are all either integers or boolean true/false.

Характеристика	Значение по умолчанию	Назначение
cfg.encode_max_depth	128	Max recursion depth for encoding
cfg.encode_deep_as_nil	false	A flag saying whether to crop tables with nesting level deeper than cfg.encode_max_depth. Not-encoded fields are replaced with one null. If not set, too high nesting is considered an error.
cfg.encode_invalid_numbers	true	A flag saying whether to enable encoding of NaN and Inf numbers
cfg.encode_load_metatables	true	A flag saying whether the serializer will follow __serialize metatable field
cfg.encode_use_tostring	false	A flag saying whether to use tostring() for unknown types
cfg.encode_invalid_as_nil	false	A flag saying whether to use NULL for non-recognized types
cfg.encode_sparse_convert	true	A flag saying whether to handle excessively sparse arrays as maps. See detailed description below
cfg.encode_sparse_ratio	2	1/encode_sparse_ratio is the permissible percentage of missing values in a sparse array
cfg.encode_sparse_safe	10	A limit ensuring that small Lua arrays are always encoded as sparse arrays (instead of generating an error or encoding as a map)
cfg.decode_invalid_numbers	true	A flag saying whether to enable decoding of NaN and Inf numbers
cfg.decode_save_metatables	true	A flag saying whether to set metatables for all arrays and maps

Sparse arrays features:

During encoding, the MsgPack encoder tries to classify tables into one of four kinds:

• map - at least one table index is not unsigned integer
• regular array - all array indexes are available
• sparse array - at least one array index is missing
• excessively sparse array - the number of values missing exceeds the configured ratio

An array is excessively sparse when all the following conditions are met:

• encode_sparse_ratio > 0
• max(table) > encode_sparse_safe
• max(table) > count(table) * encode_sparse_ratio

MsgPack encoder will never consider an array to be excessively sparse when encode_sparse_ratio = 0. The encode_sparse_safe limit ensures that small Lua arrays are always encoded as sparse arrays. By default, attempting to encode an excessively sparse array will generate an error. If encode_sparse_convert is set to true, excessively sparse arrays will be handled as maps.

msgpack.cfg() example 1:

If msgpack.cfg.encode_invalid_numbers = true (the default), then NaN and Inf are legal values. If that is not desirable, then ensure that msgpack.encode() will not accept them, by saying msgpack.cfg{encode_invalid_numbers = false}, thus:

tarantool> msgpack = require('msgpack'); msgpack.cfg{encode_invalid_numbers = true}
---
...
tarantool> msgpack.decode(msgpack.encode{1, 0 / 0, 1 / 0, false})
---
- [1, -nan, inf, false]
- 22
...
tarantool> msgpack.cfg{encode_invalid_numbers = false}
---
...
tarantool> msgpack.decode(msgpack.encode{1, 0 / 0, 1 / 0, false})
---
- error: ... number must not be NaN or Inf'
...

msgpack.cfg example 2:

To avoid generating errors on attempts to encode unknown data types as userdata/cdata, you can use this code:

tarantool> httpc = require('http.client').new()
---
...

tarantool> msgpack.encode(httpc.curl)
---
- error: unsupported Lua type 'userdata'
...

tarantool> msgpack.encode(httpc.curl, {encode_use_tostring=true})
---
- '"userdata: 0x010a4ef2a0"'
...

Similar configuration settings exist for JSON and YAML.

msgpack.NULL

Значение, сопоставимое с нулевым значением «nil» в языке Lua, которое можно использовать в качестве объекта-заполнителя в кортеже.

Пример:

tarantool> msgpack = require('msgpack')
---
...
tarantool> y = msgpack.encode({'a',1,'b',2})
---
...
tarantool> z = msgpack.decode(y)
---
...
tarantool> z[1], z[2], z[3], z[4]
---
- a
- 1
- b
- 2
...
tarantool> box.space.tester:insert{20, msgpack.NULL, 20}
---
- [20, null, 20]
...