Replication requests and responses

This section describes internal requests and responses that happen during replication. Each of them is distinguished by the header, containing a unique IPROTO_REQUEST_TYPE value. These values and the corresponding packet body structures are considered below.

Connectors and clients do not need to send replication packets.

The master also sends heartbeat messages to the replicas. The heartbeat message's IPROTO_REQUEST_TYPE is 0.

Below are details on individual replication requests. For synchronous replication requests, see internals-iproto-replication-synchronous.

Once in replication_timeout seconds, a master sends a heartbeat message to a replica, and the replica sends a response. Both messages' IPROTO_REQUEST_TYPE is IPROTO_OK. IPROTO_TIMESTAMP is a float-64 MP_DOUBLE 8-byte timestamp.

Since version 2.11, both messages have an optional field in the body that contains the IPROTO_VCLOCK_SYNC key. The master's heartbeat has no body if the IPROTO_VCLOCK_SYNC key is omitted.

The message from master to a replica:

The response from the replica:

The tutorial Understanding the binary protocol shows actual byte codes of the above heartbeat examples.

Code: 0x41.

To join a replica set, an instance must send an initial IPROTO_JOIN request to the master instance of the replica set:

The instance that receives the request sends the following messages in response:

1. Its vclock:

   

2. (Optional) A sequence of requests with information required for instance initialization:

   • an IPROTO_JOIN_META request
   • an IPROTO_RAFT request with IPROTO_RAFT_TERM and IPROTO_RAFT_VOTE fields
   • an IPROTO_RAFT_PROMOTE request
   • an IPROTO_JOIN_SNAPSHOT request

   This step applies if the IPROTO_SERVER_VERSION specified in the request is 2.10 or later.

3. A number of INSERT requests (with additional LSN and ServerID). This way, the data is updated on the instance that sent the IPROTO_JOIN request. The instance should not reply to these INSERT requests.

4. The new vclock's MP_MAP in a response similar to the one above.

5. A number of INSERT, REPLACE, UPDATE, UPSERT, and DELETE requests. This way, the instance that is joining the replica set receives data updates that happened during the join stage.

6. The new vclock's MP_MAP in a response similar to the one above.

Code: 0x42.

If IPROTO_JOIN was successful, the initiator instance must send an IPROTO_SUBSCRIBE request to all the nodes listed in its box.cfg.replication:

After a successful IPROTO_SUBSCRIBE request, the instance must process every request that could come from other masters. Each master's request includes a vclock pair corresponding to that master – its instance ID and its LSN, independent of other masters.

IPROTO_ID_FILTER (0x51) is an optional key used in the SUBSCRIBE request followed by an array of ids of instances whose rows won't be relayed to the replica. The field is encoded only when the ID list is not empty.

Code: 0x45.

To join a replica set as an anonymous replica, an instance must send an initial IPROTO_FETCH_SNAPSHOT request to the master instance of the replica set:

To learn about anonymous replicas, see replication.anon.

The instance that receives the request sends the following messages in response:

1. Its vclock:

2. (Optional) A sequence of requests with information required for instance initialization:

   • an IPROTO_JOIN_META request
   • an IPROTO_RAFT request with IPROTO_RAFT_TERM and IPROTO_RAFT_VOTE fields
   • an IPROTO_RAFT_PROMOTE request
   • an IPROTO_JOIN_SNAPSHOT request

   This step applies if the IPROTO_SERVER_VERSION specified in the request is 2.10 or later.

3. A number of INSERT requests (with additional LSN and ServerID). This way, the data is updated on the instance that sent the IPROTO_JOIN request. The instance should not reply to these INSERT requests.

4. The new vclock's MP_MAP in a response similar to the one above.

Code: 0x46.

To register an anonymous replica in a replica set so that it's not anonymous anymore, it must send an IPROTO_REGISTER request to a master node of the replica set:

The instance that receives the request sends the following messages in response:

1. A number of INSERT, REPLACE, UPDATE, UPSERT, and DELETE requests. This way, the instance that is registering in the replica set receives data updates that happened since the time it fetched the snapshot.
2. The new vclock's MP_MAP.

Technically, subsequent IPROTO_FETCH_SNAPSHOT and IPROTO_REGISTER requests are equivalent to IPROTO_JOIN.

Code: 0x47.

When an instance receives an IPOTO_JOIN or IPROTO_FETCH_SNAPSHOT request, its responses include the information required for the instance initialization: current Raft term, current state of synchronous transaction queue. Before sending this information, the instance sends an IPROTO_JOIN_META request with an empty body:

Learn more in IPROTO_JOIN

Code: 0x48.

An instance that has received an IPROTO_JOIN or IPROTO_FETCH_SNAPSHOT request sends an IPROTO_JOIN_SNAPSHOT request with an empty body after it completes sending the instance initialization information.

Learn more in IPROTO_JOIN

Code: 0x44.

When connecting for replication, an instance sends an IPROTO_VOTE request. It has no body:

IPROTO_VOTE is critical during replica set bootstrap. The response to this request is IPROTO_BALLOT.

Code: 0x29.

This value of IPROTO_REQUEST_TYPE indicates a message sent in response to IPROTO_VOTE (not to be confused with the key IPROTO_RAFT_VOTE).

IPROTO_BALLOT and IPROTO_VOTE are critical during replica set bootstrap. IPROTO_BALLOT corresponds to a map containing the following fields:

IPROTO_BALLOT_REGISTERED_REPLICA_UUIDS has the MP_ARRAY type. The array contains MP_STR elements.

Code: 0x1e.

A node broadcasts the IPROTO_RAFT request to all the replicas connected to it when the RAFT state of the node changes. It can be any actions changing the state, like starting a new election, bumping the term, voting for another node, becoming the leader, and so on.

If there should be a response, for example, in case of a vote request to other nodes, the response will also be an IPROTO_RAFT message. In this case, the node should be connected as a replica to another node from which the response is expected because the response is sent via the replication channel. In other words, there should be a full-mesh connection between the nodes.

IPROTO_REPLICA_ID is the ID of the replica from which the request came.

Code: 0x1f.

See box.ctl.promote().

In the header:

• IPROTO_REPLICA_ID is the replica ID of the node that sent the request.
• IPROTO_LSN is the actual LSN of the promote operation as recorded in the WAL.

In the body:

• IPROTO_REPLICA_ID is the replica ID of the previous synchronous queue owner.
• IPROTO_LSN is the LSN of the last operation on the previous synchronous queue owner.
• IPROTO_TERM is the term in which the node that sent the request becomes the synchronous queue owner. This term corresponds to the value of box.info.synchro.queue.term on the instance.

Code: 0x20.

See box.ctl.demote().

In the header:

• IPROTO_REPLICA_ID is the replica ID of the node that sent the request.
• IPROTO_LSN is the actual LSN of the demote operation as recorded in the WAL.

In the body:

• IPROTO_REPLICA_ID is the replica ID of the node that sent the request (same as the value in the header).
• IPROTO_LSN is the LSN of the last synchronous transaction recorded in the node's WAL.
• IPROTO_TERM is the term in which the queue becomes empty.

Code: 0x28.

This message is used in replication connections between Tarantool nodes in synchronous replication. It is not supposed to be used by any client applications in their regular connections.

This message confirms that the transactions that originated from the instance with id = IPROTO_REPLICA_ID (body) have achieved quorum and can be committed, up to and including LSN = IPROTO_LSN (body).

The body is a 2-item map:

In the header:

• IPROTO_REPLICA_ID is the ID of the replica that sends the confirm message.
• IPROTO_LSN is the LSN of the confirmation action.

In the body:

• IPROTO_REPLICA_ID is the ID of the instance from which the transactions originated.
• IPROTO_LSN is the LSN up to which the transactions should be confirmed.

Prior to Tarantool 2.10.0, IPROTO_RAFT_CONFIRM was called IPROTO_CONFIRM.

Code: 0x29.

This message is used in replication connections between Tarantool nodes in synchronous replication. It is not supposed to be used by any client applications in their regular connections.

This message says that the transactions that originated from the instance with id = IPROTO_REPLICA_ID (body) couldn't achieve quorum for some reason and should be rolled back, down to LSN = IPROTO_LSN (body) and including it.

The body is a 2-item map:

In the header:

• IPROTO_REPLICA_ID is the ID of the replica that sends the rollback message.
• IPROTO_LSN is the LSN of the rollback action.

In the body:

• IPROTO_REPLICA_ID is the ID of the instance from which the transactions originated.
• IPROTO_LSN is the LSN starting with which all pending synchronous transactions should be rolled back.

Prior to Tarantool 2.10.0, IPROTO_RAFT_ROLLBACK was called IPROTO_ROLLBACK.