APP is used by hundreds of millions of users. In order to deeply tap the needs of users, it is necessary to design a log collection system to collect the daily usage of users. Please use your knowledge to design such a log collection system.
It is assumed that the user will initiate a request every time he or she operates or clicks on APP.
Please give the system architecture design drawing, database storage mode, required machine quantity, and QPS that can be carried, and describe the difficulties and solutions in this system as much as possible in detail.
(Looking at several answers downstairs, I found that I missed an important point. The log does not need to be sent in real time. )
1. Receive the user’s request.
2. Store the request to the message queue
3. The machines in the background consume from the message queue and store in the database.
My opinion on whether users upload logs in real time and how much time is appropriate is: about 5-15 minutes.
It is necessary to consider the situation that the user quits for too long, for example, the user quits within 3 minutes. Then submit the log to the server before the user quits.
Hundreds of millions of users, let’s assume that 5 million users are online. Send logs every 5 minutes. Then 5 million users send 16,000 requests per second on average.
Step 1, definitely want to do load balancing. And this step is also very important. If you hang up, the log will not be collected at all.
16,000 requests, which may contain 1 million records or 5 million records. Look at the latitude of the recorded data.
That is, 1-10 million query insertions are made into the database in one second. This requires a message queue to buffer, otherwise the database will definitely hang up.
This is the basic structure. Most of the more users are just adding machines.