How to launch a technological product with a throughput of 100 K+ RPS
cover

How to launch
a technology
product with
100K+ RPS throughput

VK Process Mining is an Enterprise solution by a Russian company that helps manage business processes and indicators in real time. The platform examines business processes to look for discrepancies with regulations and unnecessary approvals. The product is based on the Tarantool in-memory computation platform.

Scroll

Task and requirements

Initially, the VK Process Mining process analytics platform appeared as a custom solution for a specific project. None of the existing products met the customer’s requirements, so the development of the solution was delegated to VK. After the project was implemented, VK Process Mining became part of the VK Digital Technologies product portfolio.

One of the tasks that the developers faced from the very beginning was horizontal scaling of the solution. When implementing a platform, customers often start with one process, such as procurement analytics or logistics, and then expand the solution to others.

As the project progresses, it is necessary to break down processes into different data models over different periods and observe dynamic changes over time. At the same time, corporate information systems constantly generate new data that also needs to be analyzed. Therefore, VK Process Mining needed a cluster solution that would meet the following requirements:

  • Horizontal scaling
  • Processing large amounts of data
  • Analytics based on thousands of attributes from various systems
  • Real-time data loading

Solution

Solution

Solution

icon

The development team considered several approaches:

  • PostgreSQL-based solutions, including cluster solutions.
  • Graph DBMSs. At the time of the development of this analytical platform, not a single graph DBMS could be fully scaled without significant modifications.
  • ClickHouse with column-oriented storage. Columnar DBMSs do not provide sufficient performance.
  • The Tarantool in-memory computation platform.

Solution

icon

The developers of VK Process Mining chose a solution based on Tarantool Cartridge — the Tarantool framework for scaling. It allows creating and configuring clusters of multiple Tarantool instances. The Tarantool cluster combines an application server with a DBMS.

In particular, all business logic is based on Cartridge. Aside from the classic «processing» and «storage» roles, a Tarantool cluster also has «worker» and «router» nodes. These roles enable storage and processing of data in Tarantool, retrieving data from external sources, as well as running background processes, machine learning, and forecasting jobs.

Why are we talking about sources — in the plural? Typically, business processes take place across several information systems. That’s why VK Process Mining includes an ETL (Extract, Transform, Load) module, also based on Tarantool. It allows extracting data from external systems according to configured rules, converting them into chains of events, and adding them to the system.

icon icon
Scroll

Architectural
solutions

VK Process Mining has a classic web application with frontend, backend, and a data storage. Tarantool works as a web server, and provides high performance. It can withstand a load of hundreds of thousands of RPS. Additionally, there is an HTTP balancer between the system and the user.

The solution uses a hybrid storage. The main data with ready-made chains of events is stored in RAM for quick access. Tarantool saves the data pumped through ETL into a disk storage system.

Results

300+ million

Events processed without performance decrease in the client project

100K+ RPS

Speed of receiving data from an external source and its turnover

6 months

From the beginning of product development to the first industrial implementation

Get a consultation

Order a demo

Thank you for your request

Tarantool experts will
contact you shortly