Graceful shutdown protocol

Overview

Since 2.10.0.

The graceful shutdown protocol is a mechanism that helps to prevent data loss in requests in case of a shutdown command. According to the protocol, when a server receives an os.exit() command or a SIGTERM signal, it does not exit immediately. Instead of that, first, the server stops listening for new connections. Then, the server sends the shutdown packets to all connections that support the graceful shutdown protocol. When a client is notified about the upcoming server exit, it stops serving any new requests and waits for active requests to complete before closing the connections. Once all connections are terminated, the server will be shut down.

The protocol uses the event subscription system. That is, the feature is available if the server supports the box.shutdown event and IPROTO_WATCH. For more information about it, see reference for the event watchers and the corresponding page in the Binary Protocol section.

How the graceful shutdown works

The shutdown protocol works in the following way:

1. First, the server receives a shutdown request. It can be either an os.exit() command or a SIGTERM signal.
2. Then the box.shutdown event is generated. The server broadcasts it to all subscribed remote watchers (see IPROTO_WATCH). That is, the server calls box.broadcast(‘box.shutdown’, true) from the box.ctl.on_shutdown() trigger callback. Once this is done, the server stops listening for new connections.
3. From now on, the server waits until all subscribed connections are terminated.
4. At the same time, the client gets the box.shutdown event and shuts the connection down gracefully.
5. After all connections are closed, the server will be stopped. Otherwise, a timeout occurs, and the Tarantool exits immediately. You can set up the required timeout with the set_on_shutdown_timeout() function.