Module merger | Tarantool
Tarantool
Check out the new release policy

Module merger

The merger module takes a stream of tuples and provides access to them as tables.

The four functions for creating a merger object instance are:

The methods for using a merger object are:

merger.new_tuple_source(gen, param, state)

Create a new merger instance from a tuple source.

A tuple source just returns one tuple.

The generator function gen() allows creation of multiple tuples via an iterator.

The gen() function should return:

  • state, tuple each time it is called and a new tuple is available,
  • nil when no more tuples are available.
Parameters:
  • gen – function for iteratively returning tuples
  • param – parameter for the gen function
Return:

merger-object a merger object

Example: see merger_object:pairs() method.

merger.new_buffer_source(gen, param, state)

Create a new merger instance from a buffer source.

Parameters and return: same as for merger.new_tuple_source.

To set up a buffer, or a series of buffers, use the buffer module.

merger.new_table_source(gen, param, state)

Create a new merger instance from a table source.

Parameters and return: same as for merger.new_tuple_source.

Example: see merger_object:select() method.

merger.new(key_def, sources, options)

Create a new merger instance from a merger source.

A merger source is created from a key_def object and a set of (tuple or buffer or table or merger) sources. It performs a kind of merge sort. It chooses a source with a minimal / maximal tuple on each step, consumes a tuple from this source, and repeats.

Parameters:
  • key_def – object created with key_def
  • source – parameter for the gen() function
  • optionsreverse=true if descending, false or nil if ascending
Return:

merger-object a merger object

A key_def can be cached across requests with the same ordering rules (typically these would be requests accessing the same space).

Example: see merger_object:pairs() method.

object merger_object

A merger object is an object returned by:

It has methods:

merger_object:select([buffer[, limit]])

Access the contents of a merger object with familiar select syntax.

Parameters:
Return:

a table of tuples, similar to what select would return

Example with new_table_source():

-- Source via new_table_source, simple generator function
-- tarantool> s:select()
-- ---
-- - - [100]
--   - [200]
-- ...
merger=require('merger')
k=0
function merger_function(param)
  k = k + 1
  if param[k] == nil then return nil end
  return box.NULL, param[k]
  end
chunks={}
chunks[1] = {{100}} chunks[2] = {{200}} chunks[3] = nil
s = merger.new_table_source(merger_function, chunks)
s:select()
merger_object:pairs()

The pairs() method (or the equivalent ipairs() alias method) returns a luafun iterator. It is a Lua iterator, but also provides a set of handy methods to operate in functional style.

Parameters:
  • tuple (table) – tuple or Lua table with field contents
Return:

the tuples that can be found with a standard pairs() function

Example with new_tuple_source():

-- Source via new_tuple_source, from a space of tables
-- The result will look like this:
-- tarantool> so:pairs():totable()
-- ---
-- - - [100]
--   - [200]
-- ...
merger = require('merger')
box.schema.space.create('s')
box.space.s:create_index('i')
box.space.s:insert({100})
box.space.s:insert({200})
so = merger.new_tuple_source(box.space.s:pairs())
so:pairs():totable()

Example with two mergers:

-- Source via key_def, and table data

-- Create the key_def object
merger = require('merger')
key_def_lib = require('key_def')
key_def = key_def_lib.new({{
    fieldno = 1,
    type = 'string',
}})
-- Create the table source
data = {{'a'}, {'b'}, {'c'}}
source = merger.new_source_fromtable(data)
i1 = merger.new(key_def, {source}):pairs()
i2 = merger.new(key_def, {source}):pairs()
-- t1 will be 'a' (tuple 1 from merger 1)
t1 = i1:head():totable()
-- t3 will be 'c' (tuple 3 from merger 2)
t3 = i2:head():totable()
-- t2 will be 'b' (tuple 2 from merger 1)
t2 = i1:head():totable()
-- i1:is_null() will be true (merger 1 ends)
i1:is_null()
-- i2:is_null() will be true (merger 2 ends)
i2:is_null()

More examples:

See https://github.com/Totktonada/tarantool-merger-examples which, in addition to discussing the merger API in detail, shows Lua code for handling many more situations than are in this manual’s brief examples.

Found what you were looking for?
Feedback