Seminar 9: Consistency and the Database Module
Goal: Embark on the exploration of consistency in distributed systems, with a focus on designing a robust database module.
Introduction
In Session 9 of our Distributed Systems course, we shift our focus to implementing the database system within our distributed architecture. With the election mechanism in place for mutual exclusion, enabling the creation of the order execution layer, we now address the crucial services required for actual order execution, including the database, payments service, notifications, etc... Given the anticipated high system load, we prioritize a distributed database solution with replication, for ensuring reliability and availability. However, this introduces consistency challenges that demand attention. Our goal is to design a distributed database system that maintains consistency while ensuring robust performance under demanding conditions.
The focus of this entire Checkpoint will be on the order execution tasks. If you have already accomplished the required functionalities for the order validation tasks, feel free to comment out the code and docker-compose configuration of previous services, namely the code within the orchestrator that triggers the fraud-detection, books-suggestions, and transaction-verification services, as well as the specification of these services within docker-compose. For development and testing purposes, we will create only valid orders entering the queue and being executed. Later in the final Checkpoint, we will again connect the entire set of services.
In this task, you should now think of designing and implementing a distributed database system with two basic operations: Read and Write. This database should be a Key-Value database that will simply store the stocks of our available books, the key being the title and the value being the stock. Our main focus is the database functionality, not the design of the database model, but you are free to expand it with more information such as the price per book and other data points you may consider relevant. This database system should be replicated to ensure better reliability and availability, but you must come with a strategy for achieving the best possible level of consistency. Here are the concrete tasks:
Task
Extend your system with the following functionality and new services:
- Books Database: Create a new gRPC service - the books database - which will have two atomic operations, Read and Write:
- Replication: This service should be replicated, at least 3 times, meaning, there should be at least 3 instances of the database running at every moment. These instances may implement and run the same code, which internally should consist of a basic key-value structure. The gRPC interface, however, should allow for Read and Write operations across the whole distributed database system, abstracting from the client process (the one querying the database) the fact that it’s a distributed database. For that, we need to employ a consistency protocol.
- Consistency: Explore the different strategies for consistency protocols or consensus mechanisms learned in the lectures, choose one that suits some minimum consistency levels, e.g. sequential consistency, and proceed with the implementation. Examples are Primary-based protocols, State-machine Replication, or Chain Replication. There is no right or wrong approach, but be aware of the trade-offs of your design decisions, especially related to trading between consistency and availability, the read/write loads on specific parts of the database system, or the fault-tolerance of the protocol, among other considerations.
- Order Executor: The executor services, for each order, should update the database system by performing a series of Reads and Writes, e.g. to read the current value of the stock of a book and calculate and write its new value into the database.
- Bonus Points: Implement more database operations, based on Read and Write, to abstract from the executors the complexity of frequent operations. For instance, you may implement operations like DecrementStock (and IncrementStock) or comparison operations, as part of your database system. More bonus points will be awarded for other more operations that you may think of.
- Bonus Points: How do we deal with concurrent writes by different clients? Think of a solution for the problem of two simultaneous orders trying to update the stocks of the same book. You may combine your solution of this bonus task with the previous bonus task, for instance, within the scope of a new extended operation like DecrementStock, that concurrent order executors may invoke.