current position：Home>How to develop a distributed memory database (1)
How to develop a distributed memory database (1)
How to develop a distributed memory database ( One )
At present, there are many commercial in memory databases (timesten, atibase), Many open source distributed physical databases , However, there are few mature distributed memory databases . Of course mysql cluster Count as a , But it is controlled by oracle, Really want to use it for business , The cost should not be low . We have been using in memory databases for nearly 15 Years of history , With the development of distributed system , The distributed of memory database is also put on the agenda . Based on the current technical reserve , We believe that we have the ability to build a ( Real time billing system )、 reliable 、 Commercial level distributed memory database —— Call her... For the time being mdbcluster.
One 、 Goal conception ：
mdbcluster It should have the following functions ：
1. Have the basic ability of an in memory database .
2. Data node containerization
3. Support data fragmentation
4. The node supports horizontal expansion and contraction ( Business interruption )
5. Fast recovery of failed nodes
6. Database operation interface
7. data 3 Copies , And support distributed data consistency
8. The node supports online capacity expansion
9. Database cluster management
10. Database cluster monitoring and reporting
Two 、 How to achieve
1. Find an open source stand-alone memory database
We're not going to develop from scratch , There's no point in building wheels over and over again . So find one that you can control 、 ordinary 、 A stand-alone memory database with good performance has become the best choice . I admit there are great risks here , If this step goes wrong , It may lead to the failure of the whole project .FastDB Into our eyes .
Let's start with the advantages ：
a) FastDB yes C++ Written , I believe the performance should be fast , Preliminary experiments support 20W/S update operation .
b) Support the basic operation of complete transaction and database , It is of great benefit to our ability expansion later .
c) The ability to quickly recover from failures , Because it is intended to run in a container environment , This is crucial .
a) When updating, you need to lock the library , You heard me , It's not a row lock , It's not a watch lock , It's a library lock . He simply supports reading 、 Write separation , And sex can be good . Considering that nodes can be expanded horizontally in the future , It doesn't matter that there is an upper limit on the processing capacity of a single node .
b) Support operation api special . It's not universal SQL,NoSQL,NDBAPI wait . Fortunately, we have rich experience in this field , Design a set of adapter It's not too difficult .
c) I haven't thought of ……
2. Design the overall architecture
a) MDBProxy Responsible for communicating with clients , Forward request message , Handle fragment routing .
b) MDBAgent Responsible for data writing , Due to the data consistency involving multiple pieces of data , You need to submit the data in the second stage of processing at this node .
c) MDBRNode Responsible for reading data , It can improve the speed of multiple processes .
d) MDBWNode Responsible for data writing operation , One table, one process , Lifting performance .
e) Other distributed nodes are not drawn yet , To be updated later .
3. Interface design
After coming up with the overall architecture , The first problem is the design of database interface . Databases usually have a connection manager , Responsible for managing all clients accessing the database , And support connection pool . Considering the complexity of this , We decided to use the interface of the database http2 agreement . There are several advantages to doing so ：
a) We have ready-made http2 The client of 、 Implementation scheme of server side , Reduce secondary development .
b) Future trends , Strong commonality , In addition to C++ Interface , There may be JAVA Language interface .
c) After several years of use , We found that http2 Is powerful , such as tls,server push And other functions may be used in the future .
Preliminary idea ： When the client sends a request , stay URL You should bring a piece of hash value , With the type of operation （I、U、D、S）. such MDBProxy You can use this to forward data , Without unpacking . And the package adopts JSON Format , Because the data of the client is usually small .JSON The format is conducive to fast query .
When the server returns the query package , If the test performance is not poor , Also consider using JSON, To facilitate extension . otherwise , Consider a Protocol Buffer, compressed data , Maximize performance .
author[smartguys],Please bring the original link to reprint, thank you.
The sidebar is recommended
- SQLite3 minimalist Tutorial & go operating data structures using SQLite memory mode
- Penetration test - DNS rebinding
- The pytoch loading model only imports some layer weights, that is, it skips the method of specifying the network layer
- Parameter and buffer in pytoch model
- torch. nn. functional. Interpolate function
- Specify the graphics card during pytorch training
- [paper notes] Dr TANet: dynamic receptive temporary attention network for street scene change detection
- [MQ] achieve mq-08- configuration optimization from scratch fluent
- New signs are taking place in the Internet industry, and a new transformation has begun
- ACL 2022 | visual language pre training for multimodal attribute level emotion analysis
guess what you like
Cvpr2022 ｜ latest progress in small sample behavior recognition strm framework, spatio-temporal relationship modeling is still the top priority
Hallucinations in large models
Is it safe to open an account online? Which of the top ten securities companies are state-owned enterprises?
[encapsulation tips] encapsulation of list processing function
Start with Google sea entrepreneurship accelerator - recruitment and start
Hard core preview in May! Lecture tomorrow night: virtio virtualization technology trend and DPU practice | issue 16
Druid source code reading 1 - get connection and release connection
Graduation summary of actual combat training camp
Public offering "imported products" temporarily hit the reef? The first foreign-funded public offering BlackRock fund has a lot of bad thoughts or a lot of things. It is acclimatized and the performance of the two products is poor
Introduction and installation of selenium module, use of coding platform, use of XPath, use of selenium to crawl JD product information, and introduction and installation of sketch framework
- Financial IT architecture - Analysis of cloud native architecture of digital bank
- [paper notes] lsnet: extreme light weight Siamese network for change detection in remote sensing image
- Mock tool equivalent to Fiddler!
- Write a program, input several integers (separated by commas) and count the number of occurrences of each integer.
- Inventory a voice conversion library
- Technology selection of micro service registration center: which of the five mainstream registration centers is the most popular?
- Summary of root cause analysis ideas and methods ｜ ensuring it system and its stability
- JS custom string trim method
- Web3: the golden age of Creator economy
- Introduction and installation of selenium module, use of coding platform, use of XPath, use of selenium to crawl JD product information, and introduction and installation of sketch framework
- Basics: a 5-makefile example
- Database connection pool Druid source code learning (V)
- Check the box to prevent bubbling
- Click on the box to change color
- Local style and global style of uniapp
- LeetCode. 2233. Maximum product after K increases (math / minimum heap)
- Overview, BGP as, BGP neighbor, BGP update source, BGP TTL, BGP routing table, BGP synchronization
- Routing policy and routing control
- Principle and configuration of IS-IS
- C - no AC this summer
- Basic operation of linked list (complete code)
- Thread control - thread waiting, thread termination, thread separation
- Key points of acupuncture and moxibustion
- Module product and problem solution of Luogu p2260 [Tsinghua training 2012]
- Review points of Geodesy
- Summary of review points of Geodesy
- Arrangement of geodetic knowledge points
- Review key points of basic geodesy
- Luogu p2522 [haoi2011] problem B solution
- [app test] summary of test points
- Version management tool - SVN
- JDBC ~ resultset, use of resultsetmetadata, ORM idea, arbitrary field query of any table (JDBC Implementation)
- This article takes you to understand can bus
- Gear monthly update April
- Gear monthly update April
- Convert timestamp to formatted date JS
- The time stamp shows how many minutes ago and how many days ago the JS was processed
- Luogu p2216 [haoi2007] ideal square problem solution
- Miscellaneous questions