current position：Home>Baidu simian: talk about persistence mechanism and rdb/aof application scenario analysis!
Baidu simian: talk about persistence mechanism and rdb/aof application scenario analysis!
2022-06-24 12:44:24【Java program ape】
One 、Redis How persistence works ？
What is persistence ？Redis The real question of the interview , Simply put, it is to put the data into the device that will not lose the data after power failure , That is, we usually understand the hard disk .
First of all, let's see what the database does when it writes , There are five main processes ：
- The client sends the write operation to the server （ The data is in the client's memory ）.
- The database server receives the data of write request （ The data is in the memory of the server ）.
- Server call write This system call , Write data to disk （ The data is in the buffer of the system memory ）.
- The operating system transfers the data in the buffer to the disk controller （ Data in disk cache ）.
- The disk controller writes data to the physical media of the disk （ The data is really on disk ）.
The writing operations are as follows 5 A process , Let's combine the above 5 A process to see the various levels of failure ：
- When the database system fails , At this time, the system kernel is still intact . So at this time, as long as we finish executing the 3 Step , So data is safe , Because the subsequent operating system will complete the following steps , Make sure that the data will end up on disk .
- When the system is powered off , This time up 5 All the caches mentioned in item will be invalidated , And the database and the operating system will stop working . So only when the data is finished 5 Step after , In order to ensure that data will not be lost after power failure .
Pass above 5 Step by step , Maybe we would like to find out the following questions ：
- How often does the database call write, Write data to kernel buffer ？
- How long does the kernel write the data in the system buffer to the disk controller ？
- When does the disk controller write the cached data to the physical media ？
For the first question , Generally, the database level will be under full control . And on the second question , The operating system has its default policy , But we can also go through POSIX API Provided fsync A series of commands forces the operating system to write data from the kernel to the disk controller . For the third question , It's like the database is out of reach , But actually , In most cases, the disk cache is set to be turned off , Or just open for read cache , That is to say, write operations will not be cached , Write directly to disk .
The recommended approach is to turn on write caching only when your disk device has a spare battery .
Data corruption , It's just that data can't be recovered , We talked about how to ensure that the data is actually written to the disk , But writing to disk may not mean that the data will not be corrupted . For example, we may have two different write operations for one write request , When an accident happens , May cause a write operation to complete safely , But another time hasn't been done yet . If the data file structure of the database is not reasonably organized , It may lead to the situation that the data cannot be recovered at all .
There are usually three strategies for organizing data , To prevent data files from being damaged to irrecoverable conditions ：
- The first is the roughest treatment , It is not through the organization of data to ensure the recoverability of data . But by configuring data synchronization backup , After the data file is damaged, it can be recovered through data backup . actually MongoDB Do not open the operation log , By configuring Replica Sets That's what happened when .
- The other is to add an operation log based on the above , Remember the action of the operation every time , In this way, we can recover the data through the operation log . Because the operation log is written in the way of sequential appending , So there will be no operation log can not be recovered . This is similar to MongoDB When the operation log is turned on .
- The more safe way is not to modify the old data in the database , Just add in the way to complete the write operation , So the data itself is a log , In this way, the data can never be recovered . actually CouchDB It's a good example of this .
Two 、Redis Provides RDB Persistence and AOF Persistence
RDB The advantages of mechanism and its application
RDB Persistence refers to writing the data set snapshot in memory to disk within a specified time interval . It's also the default way to persist , This way is to write the data in memory to the binary file as a snapshot , The default filename is dump.rdb.
The snapshot persistence can be done automatically through configuration settings . We can configure redis stay n If it exceeds in seconds m individual key Automatically take a snapshot if it is modified , Here is the default snapshot save configuration
save 900 1 #900 If it exceeds in seconds 1 individual key Be modified , Then initiate snapshot saving save 300 10 #300 Second content if more than 10 individual key Be modified , Then initiate snapshot saving save 60 10000
RDB File saving process
- redis call fork, Now there are child processes and parent processes .
- The parent process continues to process client request , Subprocesses are responsible for writing memory contents to temporary files . because os The write time replication mechanism of （copy on write) The parent-child process will share the same physical page , When the parent process processes the write request os A copy of the page to be modified by the parent process , Instead of writing shared pages . So the number in the address space of the subprocess According to fork A snapshot of the entire database at all times .
- When the subprocess has finished writing the snapshot to the temporary file , Replace the original snapshot file with a temporary file , Then the subprocess exits .
client You can also use save perhaps bgsave Order notice redis Do a snapshot persistence .save The operation is to save the snapshot in the main thread , because redis Is to use a main thread to handle all client Request , This way it will block all client request . So... Is not recommended .
Another thing to note is , Each snapshot persistence is a complete write of memory data to disk once , Does not Incremental synchronization of dirty data only . If there's a lot of data , And there are many write operations , It's bound to cause a lot of disks io operation , Performance may be seriously affected .
- Once adopted , So your whole Redis The database will contain only one file , This is very convenient for backup . For example, you may not plan to 1 Days to file some data .
- Easy backup , We can easily put one by one RDB Move files to other storage media
- RDB Speed ratio when recovering large data sets AOF It's faster to recover .
- RDB Can be maximized Redis Performance of ： The parent process is saving RDB The only thing to do when you file is fork Make a sub process , Then this subprocess will handle all the subsequent saving work , The parent process does not need to execute any disks I/O operation .
- If you need to try to avoid losing data in the event of a server failure , that RDB Not for you. . although Redis Allows you to set different savepoints （save point） To control the preservation RDB File frequency , however , because RDB The file needs to save the state of the entire dataset , So it's not an easy operation . So you may at least 5 Minutes to save RDB file . under these circumstances , In the event of a breakdown stop , You could lose a few minutes of data .
- Every time you save RDB When ,Redis Both fork() Make a sub process , And it's up to the subprocesses to do the actual persistence work . When the data set is large , fork() It can be very time consuming , Cause the server to stop processing the client in a millisecond ; If the data set is very large , also CPU When time is very tight , So this kind of stop time may even be as long as a whole second . although AOF Rewriting also requires fork() , But no matter AOF How long is the execution interval of the rewrite , There will be no loss of data durability .
AOF File saving process
redis Will pass every written order received write Function appended to file ( The default is appendonly.aof).
When redis During restart, the contents of the whole database will be rebuilt in memory by executing the write command saved in the file again . Of course, because os Will be cached in the kernel write Changes made , So it may not be written to disk immediately . such aof The persistence of the method is also likely to lose some modifications . But we can tell... Through the configuration file redis We want to adopt fsync The function forces os Time to write to disk . There are three ways （ The default is ： Per second fsync once ）
appendonly yes // Enable aof Persistence mode # appendfsync always // Force write to disk every time write command is received , The slowest , But make sure it's completely persistent , It is not recommended to use appendfsync everysec // Force write to disk every second , A good compromise between performance and persistence , recommend # appendfsync no // Completely dependent on os, Best performance , Persistence is not guaranteed
aof The way it works also raises another question . Persistent files get bigger and bigger . For example, we call incr test command 100 Time , All files must be saved 100 Bar command , In fact, there are 99 All are superfluous . Because to restore the state of the database, a file is saved set test 100 That's enough .
To compress aof Persistent files for .redis Provides bgrewriteaof command . Received this order redis Data in memory will be stored in a similar way to snapshots Save to a temporary file by command , Finally replace the original file . The specific process is as follows
- redis call fork , Now there are two processes, father and son
- Subprocess according to database snapshot in memory , Write the command to the temporary file to rebuild the database state
- The parent process continues to process client request , In addition to writing the write command to the original aof In file . At the same time, cache the received write commands . This ensures that if the subprocess fails to rewrite, there will be no problem .
- After the subprocess writes the snapshot content to the temporary file in the command mode , The child process signals the parent process . Then the parent process writes the cached write command to the temporary file .
- Now the parent process can replace the old... With a temporary file aof file , And rename , Later, the written orders received began to go to the new aof Add... To the file .
Note that it's rewriting aof Operation of file , Not reading the old aof file , Instead, the database contents in the whole memory are rewritten with a new command aof file , It's a bit like a snapshot .
- Use AOF Persistence makes Redis Become very durable （much more durable）： You can set different fsync Strategy , Like none fsync , Once per second fsync , Or every time a write command is executed fsync . AOF The default policy for is per second fsync once , In this configuration ,Redis Still maintain good performance , And even in the event of a breakdown , And only lose one second of data at most （ fsync Will execute in the background thread , So the main thread can continue to work hard on command requests ）.
- AOF A file is a log file that only appends （append only log）, So right. AOF Writing files does not need to be done seek , Even if the log contains commands that are not written completely for some reason （ For example, the disk is full when writing , Write stoppage , wait ）, redis-check-aof Tools can also easily fix this problem . Redis Can be in AOF When the file size becomes too large , Automatically in the background AOF Rewrite ： The rewritten new AOF The file contains the minimum set of commands required to recover the current dataset . The whole rewrite operation is absolutely safe , because Redis Creating a new AOF In the process of documentation , Will continue to append the command to the existing AOF In the document , Even if there is a outage during the rewrite , The existing AOF Documents will not be lost . And once it's new AOF File creation complete ,Redis From the old AOF File switch to new AOF file , And start on the new AOF File to append .
- AOF The file holds all writes to the database in an orderly manner , These write operations to Redis The format of the protocol is saved , therefore AOF The contents of the document are very easy to read , Analyze the document （parse） It's easy too . export （export） AOF The documents are also very simple ： for instance , If you don't execute it carefully FLUSHALL command , But as long as AOF The file has not been rewritten , So just stop the server , remove AOF At the end of the document FLUSHALL command , And restart Redis , You can restore the dataset to FLUSHALL Status before execution .
- For the same dataset ,AOF The volume of the file is usually larger than RDB Volume of file .
- According to the fsync Strategy ,AOF May be slower than RDB . In general , Per second fsync Performance is still very high , Shut down fsync It can make AOF Speed and RDB As fast as , Even under high load . But when dealing with large write loads ,RDB More guaranteed maximum delay time （latency）.
- AOF This has happened in the past bug ： Because of individual orders , Lead to AOF When the file is reloaded , Unable to restore the dataset as it was when it was saved . （ for instance , Blocking order BRPOPLPUSH It has caused such bug .） The test suite adds tests for this situation ： They will automatically generate random 、 Complex datasets , And by reloading the data to make sure everything is OK . Although this kind of bug stay AOF Not common in documents , But by contrast , RDB This is almost impossible bug Of .
Generally speaking , If you want to achieve enough PostgreSQL Data security of , You should use both persistence functions at the same time . If you are very concerned about your data , But it can still withstand data loss within a few minutes , Then you can only use RDB Persistence .
author[Java program ape],Please bring the original link to reprint, thank you.
The sidebar is recommended
- [Tencent cloud 618 countdown!] Promotion strategy of the promotion activities
- Simple and flexible permission design?
- Making daily menu applet with micro build low code
- 99% of the students can't write good code because of this problem!
- Istio practical skills: using prism to construct multi version test services
- Kubernetes practical technique: setting kernel parameters for pod
- A scheme for crawlers to collect public opinion data
- Essential key steps in the construction of e-commerce live broadcast source code
- How do websites and we media tap user needs? Deeply expose the secrets behind the keywords!
- From theory to practice, decipher Alibaba's internal MySQL optimization scheme in simple terms
guess what you like
Continuous testing | key to efficient testing in Devops Era
It's settled! Bank retail credit risk control just does it!
Encapsulate the method of converting a picture file object to Base64
The pod is evicted due to insufficient disk space of tke node
Post processing - deep camera deformation effects
Tencent released credit risk control results safely: it has helped banks lend more than 100 billion yuan
Interesting erasure code
Five minutes to develop your own code generator
How to make secruecrt more productive
About me, a 19 line programmer
- Kubernetes practical skill: entering container netns
- [programming navigation] the practical code summarized by foreign great God, learned in 30 seconds!
- Design and implementation of high performance go log library zap
- Ghost, a synonym for blog system
- Engage in audio and video development? Several things I have to say about SRT live broadcast protocol
- IOMMU (VII) -vfio and mdev
- [highlights] summary of award-winning activities of Tencent cloud documents
- The programmer's graduation project is still bald after a year
- How to solve the problem that MBR does not support partitions over 2T, and lossless transfer to GPT
- Another prize! Tencent Youtu won the leading scientific and technological achievement award of the 2021 digital Expo
- Use txvideoeditor to add watermark and export video card at 99%? No successful failed callback?
- Reset the password, and the automatic login of the website saved by chrome Google browser is lost. What is the underlying reason?
- [log service CLS] Tencent cloud log service CLS accesses CDN
- [live broadcast of celebrities] elastic observability workshop
- Smart Policing: how to use video intelligent analysis technology to help urban policing visual comprehensive supervision and command system
- SMS SMS
- WPF from zero to 1 tutorial details, suitable for novices on the road
- Dingding, Feishu, and enterprise wechat: different business approaches
- [2022 national tournament simulation] BigBen -- determinant, Du Jiao sieve
- How to do research on plant endophytes? Special topic on Microbiology
- Mlife forum | microbiome and data mining
- 105. simple chat room 8: use socket to transfer pictures
- Parse NC format file and GRB format file dependent package edu ucar. API learning of netcdfall
- Reading notes of returning to hometown
- Generate the NC file of 4-D air pressure and temperature, and then read the code (provide the code)
- Codereview tool chain for micro medicine
- Babbitt | metauniverse daily must read: 618 scores have been announced. How much contribution has the digital collection made behind this satisfactory answer
- On the value foam of digital copyright works from the controversial nature of "Meng Hua Lu"
- Use the open source tool k8tz to gracefully set the kubernetes pod time zone
- How does Argo family bucket make Devops easier?
- A hero's note stirred up a thousand waves across 10 countries, and the first-line big factories sent people here- Gwei 2022 Singapore
- MySQL foreign key impact
- Common special characters in JS and TS
- The text to voice function is available online. You can experience the services of professional broadcasters. We sincerely invite you to try it out
- Who said that "programmers are useless without computers? The big brother around me disagrees! It's true
- what the fuck! I'm flattered. He actually wrote down the answers to the redis interview questions that big companies often ask!
- Concept + formula (excluding parameter estimation)
- [database] final review (planning Edition)
- Troubleshooting and optimization of files that cannot be globally searched by ordinary users in easydss video platform customization project
- How can ffmpeg streaming to the server save video as a file through easydss video platform?