current position:Home>How to develop a distributed memory database (1)

How to develop a distributed memory database (1)

2022-05-15 07:35:38smartguys

How to develop a distributed memory database ( One )

   At present, there are many commercial in memory databases (timesten, atibase), Many open source distributed physical databases , However, there are few mature distributed memory databases . Of course mysql cluster Count as a , But it is controlled by oracle, Really want to use it for business , The cost should not be low . We have been using in memory databases for nearly 15 Years of history , With the development of distributed system , The distributed of memory database is also put on the agenda . Based on the current technical reserve , We believe that we have the ability to build a ( Real time billing system )、 reliable 、 Commercial level distributed memory database —— Call her... For the time being mdbcluster.

One 、 Goal conception :

  mdbcluster It should have the following functions :

  1.  Have the basic ability of an in memory database .

  2.  Data node containerization

  3.  Support data fragmentation

  4.  The node supports horizontal expansion and contraction ( Business interruption )

  5.  Fast recovery of failed nodes

  6.  Database operation interface

  7. data 3 Copies , And support distributed data consistency

  8.  The node supports online capacity expansion

  9.  Database cluster management

  10.  Database cluster monitoring and reporting

Two 、 How to achieve

  1.  Find an open source stand-alone memory database

   We're not going to develop from scratch , There's no point in building wheels over and over again . So find one that you can control 、 ordinary 、 A stand-alone memory database with good performance has become the best choice . I admit there are great risks here , If this step goes wrong , It may lead to the failure of the whole project .FastDB Into our eyes .

   Let's start with the advantages :

    a) FastDB yes C++ Written , I believe the performance should be fast , Preliminary experiments support 20W/S update operation .

    b)  Support the basic operation of complete transaction and database , It is of great benefit to our ability expansion later .

    c)  The ability to quickly recover from failures , Because it is intended to run in a container environment , This is crucial .

   shortcoming :

    a) When updating, you need to lock the library , You heard me , It's not a row lock , It's not a watch lock , It's a library lock . He simply supports reading 、 Write separation , And sex can be good . Considering that nodes can be expanded horizontally in the future , It doesn't matter that there is an upper limit on the processing capacity of a single node .

    b)  Support operation api special . It's not universal SQL,NoSQL,NDBAPI wait . Fortunately, we have rich experience in this field , Design a set of adapter It's not too difficult .

    c)  I haven't thought of ……

  2.  Design the overall architecture

  a) MDBProxy Responsible for communicating with clients , Forward request message , Handle fragment routing .

  b) MDBAgent Responsible for data writing , Due to the data consistency involving multiple pieces of data , You need to submit the data in the second stage of processing at this node .

  c) MDBRNode Responsible for reading data , It can improve the speed of multiple processes .

  d) MDBWNode Responsible for data writing operation , One table, one process , Lifting performance .

  e)  Other distributed nodes are not drawn yet , To be updated later .

3.  Interface design

   After coming up with the overall architecture , The first problem is the design of database interface . Databases usually have a connection manager , Responsible for managing all clients accessing the database , And support connection pool . Considering the complexity of this , We decided to use the interface of the database http2 agreement . There are several advantages to doing so :

  a) We have ready-made http2 The client of 、 Implementation scheme of server side , Reduce secondary development .

  b) Future trends , Strong commonality , In addition to C++ Interface , There may be JAVA Language interface .

  c)  After several years of use , We found that http2 Is powerful , such as tls,server push And other functions may be used in the future .

   Preliminary idea : When the client sends a request , stay URL You should bring a piece of hash value , With the type of operation (I、U、D、S). such MDBProxy You can use this to forward data , Without unpacking . And the package adopts JSON Format , Because the data of the client is usually small .JSON The format is conducive to fast query .

   When the server returns the query package , If the test performance is not poor , Also consider using JSON, To facilitate extension . otherwise , Consider a Protocol Buffer, compressed data , Maximize performance .

copyright notice
author[smartguys],Please bring the original link to reprint, thank you.

Random recommended