Репликация¶
Replication allows multiple Tarantool instances to work on copies of the same databases. The databases are kept in synch because each instance can communicate its changes to all the other instances. A pack of instances which share the same databases are a «replica set». Each instance in a replica set also has a numeric identifier which is unique within the replica set, known as the «instance id» or «instance id».
To set up replication, it’s necessary to set up the master instances which make the original data-change requests, set up the replica instances which copy data-change requests from masters, and establish procedures for recovery from a degraded state.
Архитектура механизма репликации¶
Чтобы знать о всех изменениях на стороне главного сервера, каждая реплика непрерывно опрашивает главный сервер на предмет обновлений в его WAL-файле (write ahead log) и применяет эти обновления на своей стороне. Каждая запись в WAL-файле представляет собой один запрос на изменение данных (например, INSERT, UPDATE или DELETE) и присвоенный данной записи номер (LSN = log sequence number). Номера присваиваются в порядке возрастания. По сути, репликация в Tarantool’е является построчной: все команды на изменение данных полностью детерминированы, и каждая такая команда относится только к одному кортежу.
Вызовы хранимых Lua-процедур фиксируются не в WAL-файле, а в журнале событий (event log). Таким образом гарантируется, что не детерминированное поведение логики на Lua не приведет к рассинхронизации реплицированных данных.
A change to a temporary space is not written to the write-ahead log, and therefore is not replicated.
Setting up a master¶
Чтобы настроить возможность установки соединения для реплик, на стороне главного сервера требуется лишь указать значение для параметра «listen» в init-запросе box.cfg. Например, box.cfg{listen=3301}. Когда URI для прослушивания задан, главный сервер готов принимать запросы на соединение от любого количества реплик. Каждая реплика при этом находится в некотором статусе репликации.
Настройка сервера-реплики¶
An instance requires a valid snapshot (.snap) file. A snapshot file is created
for a instance the first time that box.cfg occurs for it. If this first
box.cfg request occurs without a «replication source» clause, then the
instance is a master and starts its own new replica set with a new unique UUID.
If this first box.cfg request occurs with a «replication source» clause,
then the instance is a replica and its snapshot file, along with the replica-set
information, is constructed from the write-ahead logs of the master.
Therefore, to start replication, specify the replication source in the
replication_source parameter of a box.cfg
request. When a replica contacts a master for the first time, it becomes part of
a replica set. On subsequent occasions, it should always contact a master in the
same replica set.
После установки соединения с главным сервером реплика запрашивает у него все изменения, чьи LSN-номера в WAL-файле больше номера последнего локального изменения на реплике. Поэтому WAL-файлы на главном сервере нужно хранить до тех пор, пока все реплики не применят изменения из этих WAL-файлов на своей стороне. Состояние реплики можно «обнулить», удалив все файлы репликации (.snap-файл со снимком и .xlog-файлы с записями WAL) и запустив сервер снова. Реплика при этом возьмет все кортежи с главного сервера и придет в синхронизированное состояние. Обратите внимание, что такая процедура «обнуления» сработает, только если на главном сервере будут доступны все нужные WAL-файлы.
Примечание
Параметры репликации можно менять на лету, что позволяет назначать реплику на роль главного сервера и наоборот. Для этого используется запрос box.cfg.
Примечание
Реплика не берет настройки конфигурации с главного сервера, например настройки запуска фоновой программы для работы со снимками на главном сервере. Чтобы получить те же настройки на реплике, нужно задать их явным образом.
Примечание
Replication requires privileges. Privileges for accessing spaces could be granted directly to the user who will start the replica. However, it is more usual to grant privileges for accessing spaces to a role, and then grant the role to the user who will start the replica.
Восстановление после сбоя¶
«Сбой» — это ситуация, когда главный сервер становится недоступен вследствие проблем с оборудованием, сетевых неполадок или программной ошибки. У реплики нет способа автоматически обнаружить, что связь с главным сервером утеряна насовсем, поскольку причины сбоя и окружение, в котором развернута репликация, могут быть очень разными. Поэтому обнаруживать сбой должен человек.
However, once a master failure is detected, the recovery is simple: declare
that the replica is now the new master, by saying
box.cfg{... listen=URI ...}
Then, if there are updates on the old master that were not propagated before
the old master went down, they would have to be re-applied manually.
Quick startup of a new simple two-instance replica set¶
Step 1. Start the first instance thus:
box.cfg{listen = *uri#1*}
-- в этом запросе можно задать больше ограничений
box.schema.user.grant('guest', 'read,write,execute', 'universe')
box.snapshot()
… Now a new replica set exists.
Step 2. Check where the second instance’s files will go by looking at its directories (snap_dir for snapshot files, wal_dir for .xlog files). They must be empty - when the second instance joins for the first time, it has to be working with a clean state so that the initial copy of the first instance’s databases can happen without conflicts.
Step 3. Start the second instance thus:
box.cfg{
listen = uri#2,
replication_source = uri#1
}
… where uri#1 = the URI that the first instance is listening on.
Вот и всё.
In this configuration, the first instance is the «master» and the second instance is the «replica». Henceforth every change that happens on the master will be visible on the replica. A simple two-instance replica set with the master on one computer and the replica on a different computer is very common and provides two benefits: FAILOVER (because if the master goes down then the replica can take over), or LOAD BALANCING (because clients can connect to either the master or the replica for select requests). Sometimes the replica may be configured with the additional parameter read_only = true.
Мониторинг действий реплики¶
In box.info there is a box.info.replication.status field:
«off», «stopped», «connecting», «auth», «follow», or «disconnected».
If a replica’s status is «follow», then there will be more fields –
the list is in the section Submodule box.info.
In the log there is a record of replication activity. If a primary instance is started with:
box.cfg{
<...>,
logger = *имя_файла_для_ведения_журнала*,
<...>
}
то на каждую установку/потерю соединения реплики с главным сервером в журнале будут появляться строчки со словом «relay».
Предотвращение дублирующихся действий¶
Предположим, что реплика пытается сделать нечто, что уже было сделано на главном сервере. Например:
box.schema.space.create('X')
Это приведет к ошибке «Space X exists» («Пространство X уже существует»). В данном частном случае можно скорректировать инструкцию следующим образом:
box.schema.space.create('X', {if_not_exists=true})
Но существует и более общее решение: использовать метод box.once(key, function). Если box.once() был вызван ранее с тем же значением параметра key, то функция function игнорируется; в противном случае функция function будет выполнена. Поэтому действия, которые должны совершаться только один раз за время текущей сессии репликации, нужно помещать в функцию и вызывать ее с помощью метода box.once(). Например:
function f()
box.schema.space.create('X')
end
box.once('space_creator', f)
Репликация по схеме master-master¶
In the simple master-replica configuration, the master’s changes are seen by the replica, but not vice versa, because the master was specified as the sole replication source. In the master-master configuration, also sometimes called multi-master configuration, it’s possible to go both ways. Starting with the simple configuration, the first instance has to say:
box.cfg{ replication_source = uri#2 }
Этот запрос можно выполнить в любой момент, т.к. параметр replication_source можно задавать на ходу.
In this configuration, both instances are «masters» and both instances are «replicas». Henceforth every change that happens on either instance will be visible on the other. The failover benefit is still present, and the load-balancing benefit is enhanced (because clients can connect to either instance for data-change requests as well as select requests).
If two operations for the same tuple take place «concurrently» (which can
involve a long interval because replication is asynchronous), and one of
the operations is delete or replace, there is a possibility that
instances will end up with different contents.
Ответы на вопросы «Что если?»¶
| Q: | What if there are more than two instances with master-master? |
|---|---|
| A: | On each instance, specify the replication source for all the others. For example, instance #3 would have a request:
box.cfg{ replication_source = {uri1}, {uri2} }
|
| Q: | What if an instance should be taken out of the replica set? |
| A: | For a replica, run |
| Q: | What if an instance leaves the replica set? |
| A: | The other instances carry on. If the wayward instance rejoins, it will receive all the updates that the other instances made while it was away. |
| Q: | What if two instances both change the same tuple? |
| A: | The last changer wins. For example, suppose that instance#1 changes the tuple, then instance#2 changes the tuple. In that case instance#2’s change overrides whatever instance#1 did. In order to keep track of who came last, Tarantool implements a vector clock. |
| Q: | What if two instances both insert the same tuple? |
| A: | If a master tries to insert a tuple which a replica has inserted already, this is an example of a severe error. Replication stops. It will have to be restarted manually. |
| Q: | What if a master disappears and the replica must take over? |
| A: | A message will appear on the replica stating that the connection is
lost. The replica must now become independent, which can be done by
saying |
| Q: | What if it’s necessary to know what replica set an instance is in? |
| A: | The identification of the replica set is a UUID which is generated when the
first master starts for the first time. This UUID is stored in a tuple
of the box.space._schema system space. So to
see it, say: |
| Q: | What if it’s necessary to know what other instances belong in the replica set? |
| A: | The universal identification of an instance is a UUID in
|
| Q: | What if one of the instance’s files is corrupted or deleted? |
| A: | Stop the instance, destroy all the database files (the ones with extension
«snap» or «xlog» or «.inprogress»), restart the instance, and catch up
with the master by contacting it again (just say
|
| Q: | What if replication causes security concerns? |
| A: | Prevent unauthorized replication sources by associating a password with
every user that has access privileges for the relevant spaces, and every
user that has a replication role. That
way, the URI for the replication source parameter will always have to have
the long form |
| Q: | What if advanced users want to understand better how it all works? |
| A: | See the description of instance startup with replication in the Internals section. |
Практическое руководство по репликации¶
After following the steps here, an administrator will have experience creating a replica set and adding a replica.
Запустите два терминала, каждый в своем окне, и расположите их рядом на экране. (Далее в примерах оба терминала показаны в виде закладок. Щелкните на заголовок закладки — «Terminal #1» или «Terminal #2», — чтобы увидеть вывод на соответствующем терминале.)
В первом терминале (Terminal #1) выполните следующие команды:
$ # Terminal 1
$ mkdir -p ~/tarantool_test_node_1
$ cd ~/tarantool_test_node_1
$ rm -R ~/tarantool_test_node_1/*
$ ~/tarantool/src/tarantool
tarantool> box.cfg{listen = 3301}
tarantool> box.schema.user.create('replicator', {password = 'password'})
tarantool> box.schema.user.grant('replicator','execute','role','replication')
tarantool> box.space._cluster:select({0}, {iterator = 'GE'})
The result is that a new replica set is configured, and the instance’s UUID is displayed. Now the screen looks like this: (except that UUID values are always different):
$ # Terminal 1
$ mkdir -p ~/tarantool_test_node_1
$ cd ~/tarantool_test_node_1
$ rm -R ./*
$ ~/tarantool/src/tarantool
$ ~/tarantool/src/tarantool: version 1.6.9-1724-g033ed69
type 'help' for interactive help
tarantool> box.cfg{listen = 3301}
<... ...>
tarantool> box.schema.user.create('replicator', {password = 'password'})
<...> I> creating ./00000000000000000000.xlog.inprogress'
---
...
tarantool> box.schema.user.grant('replicator','execute','role','replication')
---
...
tarantool> box.space._cluster:select({0}, {iterator = 'GE'})
---
- - [1, '6190d919-1133-4452-b123-beca0b178b32']
...
$
Во втором терминале (Terminal #2) выполните следующие команды:
$ # Terminal 2
$ mkdir -p ~/tarantool_test_node_2
$ cd ~/tarantool_test_node_2
$ rm -R ~/tarantool_test_node_2/*
$ ~/tarantool/src/tarantool
tarantool> box.cfg{
> listen = 3302,
> replication_source = 'replicator:password@localhost:3301'
> }
tarantool> box.space._cluster:select({0}, {iterator = 'GE'})
The result is that a replica is set up. Messages appear on Terminal #1 confirming that the replica has connected and that the WAL contents have been shipped to the replica. Messages appear on Terminal #2 showing that replication is starting. Also on Terminal#2 the _cluster UUID values are displayed, and one of them is the same as the _cluster UUID value that was displayed on Terminal #1, because both instances are in the same replica set.
$ # Terminal 1
$ mkdir -p ~/tarantool_test_node_1
<... ...>
tarantool> box.space._cluster:select({0}, {iterator = 'GE'})
---
- - [1, '6190d919-1133-4452-b123-beca0b178b32']
...
tarantool>
<...> [11031] relay/127.0.0.1:58734/101/main I> recovery start
<...> [11031] relay/127.0.0.1:58734/101/main I> recovering from `./00000000000000000000.snap'
<...> [11031] relay/127.0.0.1:58734/101/main I> snapshot sent
<...> [11031] relay/127.0.0.1:58734/101/main I> recover from `./00000000000000000000.xlog'
<...> [11031] relay/127.0.0.1:58734/101/main recovery.cc:211 W> file
`./00000000000000000000.xlog` wasn't correctly closed
<...> [11031] relay/127.0.0.1:58734/102/main I> recover from `./00000000000000000000.xlog'
$ # Terminal 2
$ mkdir -p ~/tarantool_test_node_2
$ cd ~/tarantool_test_node_2
$ rm -R ./*
$ ~/tarantool/src/tarantool
/home/username/tarantool/src/tarantool: version 1.6.9-1724-g033ed69
type 'help' for interactive help
tarantool> box.cfg{
> listen = 3302,
> replication_source = 'replicator:password@localhost:3301'
> }
<...>
<...> [11243] main/101/interactive I> mapping 1073741824 bytes for tuple arena...
<...> [11243] main/101/interactive C> starting replication from localhost:3301
<...> [11243] main/102/applier/localhost:3301 I> connected to 1.6.9 at 127.0.0.1:3301
<...> [11243] main/102/applier/localhost:3301 I> authenticated
<...> [11243] main/102/applier/localhost:3301 I> downloading a snapshot from 127.0.0.1:3301
<...> [11243] main/102/applier/localhost:3301 I> done
<...> [11243] snapshot/101/main I> creating `./00000000000000000000.snap.inprogress'
<...> [11243] snapshot/101/main I> saving snapshot `./00000000000000000000.snap.inprogress'
<...> [11243] snapshot/101/main I> done
<...> [11243] iproto I> binary: started
<...> [11243] iproto I> binary: bound to 0.0.0.0:3302
<...> [11243] wal/101/main I> creating `./00000000000000000000.xlog.inprogress'
<...> [11243] main/101/interactive I> ready to accept requests
tarantool> box.space._cluster:select({0}, {iterator = 'GE'})
- - [1, '6190d919-1133-4452-b123-beca0b178b32']
- [2, '236230b8-af3e-406b-b709-15a60b44c20c']
...
В первом терминале (Terminal #1) выполните следующие запросы:
tarantool> s = box.schema.space.create('tester')
tarantool> i = s:create_index('primary', {})
tarantool> s:insert{1, 'Tuple inserted on Terminal #1'}
Теперь вывод на экране выглядит следующим образом:
<... ...>
tarantool>
tarantool>
<...> [11031] relay/127.0.0.1:58734/101/main I> recovery start
<...> [11031] relay/127.0.0.1:58734/101/main I> recovering from `./00000000000000000000.snap'
<...> [11031] relay/127.0.0.1:58734/101/main I> snapshot sent
<...> [11031] relay/127.0.0.1:58734/101/main I> recover from `./00000000000000000000.xlog'
<...> [11031] relay/127.0.0.1:58734/101/main recovery.cc:211 W> file
`./00000000000000000000.xlog` wasn't correctly closed
<...> [11031] relay/127.0.0.1:58734/102/main I> recover from `./00000000000000000000.xlog'
tarantool> s = box.schema.space.create('tester')
---
...
tarantool> i = s:create_index('primary', {})
---
...
tarantool> s:insert{1, 'Tuple inserted on Terminal #1'}
---
- [1, 'Tuple inserted on Terminal #1']
...
$ # Terminal 2
$ mkdir -p ~/tarantool_test_node_2
$ cd ~/tarantool_test_node_2
$ rm -R ./*
$ ~/tarantool/src/tarantool
/home/username/tarantool/src/tarantool: version 1.6.9-1724-g033ed69
type 'help' for interactive help
tarantool> box.cfg{
> listen = 3302,
> replication_source = 'replicator:password@localhost:3301'
> }
<...>
<...> [11243] main/101/interactive I> mapping 1073741824 bytes for tuple arena...
<...> [11243] main/101/interactive C> starting replication from localhost:3301
<...> [11243] main/102/applier/localhost:3301 I> connected to 1.6.9 at 127.0.0.1:3301
<...> [11243] main/102/applier/localhost:3301 I> authenticated
<...> [11243] main/102/applier/localhost:3301 I> downloading a snapshot from 127.0.0.1:3301
<...> [11243] main/102/applier/localhost:3301 I> done
<...> [11243] snapshot/101/main I> creating `./00000000000000000000.snap.inprogress'
<...> [11243] snapshot/101/main I> saving snapshot `./00000000000000000000.snap.inprogress'
<...> [11243] snapshot/101/main I> done
<...> [11243] iproto I> binary: started
<...> [11243] iproto I> binary: bound to 0.0.0.0:3302
<...> [11243] wal/101/main I> creating `./00000000000000000000.xlog.inprogress'
<...> [11243] main/101/interactive I> ready to accept requests
tarantool> box.space._cluster:select({0}, {iterator = 'GE'})
- - [1, '6190d919-1133-4452-b123-beca0b178b32']
- [2, '236230b8-af3e-406b-b709-15a60b44c20c']
...
В первом терминале успешно отработали операции CREATE и INSERT. Но во втором терминале ничего не произошло.
Во втором терминале (Terminal #2) выполните следующие запросы:
tarantool> s = box.space.tester
tarantool> s:select({1}, {iterator = 'GE'})
tarantool> s:insert{2, 'Tuple inserted on Terminal #2'}
Теперь вывод на экране выглядит следующим образом:
<... ...>
tarantool>
tarantool>
<...> [11031] relay/127.0.0.1:58734/101/main I> recovery start
<...> [11031] relay/127.0.0.1:58734/101/main I> recovering from `./00000000000000000000.snap'
<...> [11031] relay/127.0.0.1:58734/101/main I> snapshot sent
<...> [11031] relay/127.0.0.1:58734/101/main I> recover from `./00000000000000000000.xlog'
<...> [11031] relay/127.0.0.1:58734/101/main recovery.cc:211 W> file
`./00000000000000000000.xlog` wasn't correctly closed
<...> [11031] relay/127.0.0.1:58734/102/main I> recover from `./00000000000000000000.xlog'
tarantool> s = box.schema.space.create('tester')
---
...
tarantool> i = s:create_index('primary', {})
---
...
tarantool> s:insert{1, 'Tuple inserted on Terminal #1'}
---
- [1, 'Tuple inserted on Terminal #1']
...
<... ...>
tarantool> box.space._cluster:select({0}, {iterator = 'GE'})
- - [1, '6190d919-1133-4452-b123-beca0b178b32']
- [2, '236230b8-af3e-406b-b709-15a60b44c20c']
...
tarantool> s = box.space.tester
---
...
tarantool> s:select({1}, {iterator = 'GE'})
---
- - [1, 'Tuple inserted on Terminal #1']
...
tarantool> s:insert{2, 'Tuple inserted on Terminal #2'}
---
- [2, 'Tuple inserted on Terminal #2']
...
Во втором терминале успешно отработали операции SELECT и INSERT. Но в первом терминале ничего не произошло.
В первом терминале (Terminal #1) выполните следующие запросы и команды:
$ os.exit()
$ ls -l ~/tarantool_test_node_1
$ ls -l ~/tarantool_test_node_2
Now Tarantool #1 is stopped. Messages appear on Terminal #2 announcing that fact.
The ls -l commands show that both instances have made snapshots, which have
similar sizes because they both contain the same tuples.
<... ...>
tarantool> s:insert{1, 'Tuple inserted on Terminal #1'}
---
- [1, 'Tuple inserted on Terminal #1']
...
$ ls -l ~/tarantool_test_node_1
tarantool> os.exit()
total 8
-rw-rw-r-- 1 4925 May 5 11:12 00000000000000000000.snap
-rw-rw-r-- 1 634 May 5 11:45 00000000000000000000.xlog
$ ls -l ~/tarantool_test_node_2/
total 8
-rw-rw-r-- 1 4925 May 5 11:20 00000000000000000000.snap
-rw-rw-r-- 1 704 May 5 11:38 00000000000000000000.xlog
$
<... ...>
tarantool> s:select({1}, {iterator = 'GE'})
---
- - [1, 'Tuple inserted on Terminal #1']
...
tarantool> s:insert{2, 'Tuple inserted on Terminal #2'}
---
- [2, 'Tuple inserted on Terminal #2']
...
tarantool>
<...> [25579] main/103/replica/localhost:3301 I> can't read row
<...> [25579] main/103/replica/localhost:3301 !> SystemError
unexpected EOF when reading from socket,
called on fd 10, aka 127.0.0.1:50884, peer of 127.0.0.1:3301: Broken pipe
<...> [25579] main/103/replica/localhost:3301 I> will retry every 1 second
Во втором терминале (Terminal #2) проигнорируйте сообщения об ошибках и выполните следующие запросы:
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
tarantool> box.space.tester:insert{3, 'Another'}
Теперь вывод на экране выглядит следующим образом (сообщения об ошибках мы не приводим):
<... ...>
tarantool> s:insert{1, 'Tuple inserted on Terminal #1'}
---
- [1, 'Tuple inserted on Terminal #1']
...
$ ls -l ~/tarantool_test_node_1
tarantool> os.exit()
total 8
-rw-rw-r-- 1 4925 May 5 11:12 00000000000000000000.snap
-rw-rw-r-- 1 634 May 5 11:45 00000000000000000000.xlog
$ ls -l ~/tarantool_test_node_2/
total 8
-rw-rw-r-- 1 4925 May 5 11:20 00000000000000000000.snap
-rw-rw-r-- 1 704 May 5 11:38 00000000000000000000.xlog
$
<... ...>
tarantool> s:insert{2, 'Tuple inserted on Terminal #2'}
---
- [2, 'Tuple inserted on Terminal #2']
...
tarantool>
<...> [11243] main/105/applier/localhost:3301 I> can't read row
<...> [11243] main/105/applier/localhost:3301 coio.cc:352 !> SystemError
unexpected EOF when reading from socket,
called on fd 6, aka 127.0.0.1:58734, peer of 127.0.0.1:3301: Broken pipe
<...> [11243] main/105/applier/localhost:3301 I> will retry every 1 second
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
---
- - [1, 'Tuple inserted on Terminal #1']
- [2, 'Tuple inserted on Terminal #2']
...
tarantool> box.space.tester:insert{3, 'Another'}
---
- [3, 'Another']
...
Запросы SELECT и INSERT во втором терминале отработали несмотря на то, что сервер в первом терминале остановлен.
В первом терминале (Terminal #1) выполните следующие запросы:
$ ~/tarantool/src/tarantool
tarantool> box.cfg{listen = 3301}
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
Теперь вывод на экране выглядит следующим образом:
<... ...>
tarantool> s:insert{1, 'Tuple inserted on Terminal #1'}
---
- [1, 'Tuple inserted on Terminal #1']
...
$ ls -l ~/tarantool_test_node_1
tarantool> os.exit()
total 8
-rw-rw-r-- 1 4925 May 5 11:12 00000000000000000000.snap
-rw-rw-r-- 1 634 May 5 11:45 00000000000000000000.xlog
$ ls -l ~/tarantool_test_node_2/
total 8
-rw-rw-r-- 1 4925 May 5 11:20 00000000000000000000.snap
-rw-rw-r-- 1 704 May 5 11:38 00000000000000000000.xlog
$ ~/tarantool/src/tarantool
<...>
tarantool> box.cfg{listen = 3301}
<...> [22612] main/101/interactive I> ready to accept requests
<... ...>
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
- - [1, 'Tuple inserted on Terminal #1']
...
<... ...>
tarantool> s:insert{2, 'Tuple inserted on Terminal #2'}
---
- [2, 'Tuple inserted on Terminal #2']
...
tarantool>
<...> [25579] main/103/replica/localhost:3301 I> can't read row
<...> [25579] main/103/replica/localhost:3301 !> SystemError
unexpected EOF when reading from socket,
called on fd 10, aka 127.0.0.1:50884, peer of 127.0.0.1:3301: Broken pipe
<...> [25579] main/103/replica/localhost:3301 I> will retry every 1 second
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
---
- - [1, 'Tuple inserted on Terminal #1']
- [2, 'Tuple inserted on Terminal #2']
...
tarantool> box.space.tester:insert{3, 'Another'}
---
- [3, 'Another']
...
<...> [11243] main/105/applier/localhost:3301 I> connected to 1.6.9 at 127.0.0.1:3301
<...> [11243] main/105/applier/localhost:3301 I> authenticated
The master has reconnected to the replica set, and has NOT found what the replica wrote while the master was away. That is not a surprise – the replica has not been asked to act as a replication source.
В первом терминале (Terminal #1) введите:
tarantool> box.cfg{
> replication_source = 'replicator:password@localhost:3302'
> }
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
Теперь вывод на экране выглядит следующим образом:
<... ...>
$ ~/tarantool/src/tarantool
<...>
tarantool> box.cfg{listen = 3301}
<...> [22612] main/101/interactive I> ready to accept requests
<... ...>
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
---
- - [1, 'Tuple inserted on Terminal #1']
...
tarantool> box.cfg{
> replication_source = 'replicator:password@localhost:3302'
> }
[28987] main/101/interactive C> starting replication from localhost:3302
---
...
[22612] main/101/interactive C> starting replication from localhost:3302
[22612] main/101/interactive I> set 'replication' configuration
option to "replicator:password@localhost:3302"
[22612] main/104/applier/localhost:3302 I> connected to 1.6.9 at 127.0.0.1:3302
[22612] main/104/applier/localhost:3302 I> authenticated
[22612] wal/101/main I> creating `./00000000000000000008.xlog.inprogress'
[22612] relay/127.0.0.1:33510/102/main I> done `./00000000000000000000.xlog'
[22612] relay/127.0.0.1:33510/102/main I> recover from `./00000000000000000008.xlog'
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
---
- - [1, 'Tuple inserted on Terminal #1']
- [2, 'Tuple inserted on Terminal #2']
- [3, 'Another']
...
<... ...>
tarantool> s:insert{2, 'Tuple inserted on Terminal #2'}
---
- [2, 'Tuple inserted on Terminal #2']
...
tarantool>
<...> [25579] main/103/replica/localhost:3301 I> can't read row
<...> [25579] main/103/replica/localhost:3301 !> SystemError
unexpected EOF when reading from socket,
called on fd 10, aka 127.0.0.1:50884, peer of 127.0.0.1:3301: Broken pipe
<...> [25579] main/103/replica/localhost:3301 I> will retry every 1 second
tarantool> box.space.tester:select({0}, {iterator = 'GE'})
---
- - [1, 'Tuple inserted on Terminal #1']
- [2, 'Tuple inserted on Terminal #2']
...
tarantool> box.space.tester:insert{3, 'Another'}
---
- [3, 'Another']
...
<...> [11243] main/105/applier/localhost:3301 I> connected to 1.6.9 at 127.0.0.1:3301
<...> [11243] main/105/applier/localhost:3301 I> authenticated
tarantool>
<...> [11243] relay/127.0.0.1:36150/101/main I> recover from `./00000000000000000000.xlog'
<...> [11243] relay/127.0.0.1:36150/101/main recovery.cc:211 W> file
`./00000000000000000000.xlog` wasn't correctly closed
<...> [11243] relay/127.0.0.1:36150/102/main I> recover from `./00000000000000000000.xlog'
This shows that the two instances are once again in synch, and that each instance sees what the other instance wrote.
Чтобы удалить все тестовые данные, выполните «os.exit()» на обоих терминалах, а затем на каждом из них выполните следующие команды:
$ cd ~
$ rm -R ~/tarantool_test_node_1
$ rm -R ~/tarantool_test_node_2