MongoDB+Redis Task Queue Performance Bottleneck

  mongodb, question

Problem background:
Recently, an important task system within the company was reconstructed. Since the original task system used MongoDB to save tasks, the client took them from MongoDB. As for why MongoDB was used, it is a historical problem. It is also because if array queries using MongoDB can reduce the number of tasks many times. Assuming such a situation, an md5 needs to do task processing for N situations. If MongoDB array is used, only one md5 needs to be taken as a task, which contains a list of tasks to be processed with a length of n (the entire task is processed only after n subtasks have been processed), thus the order of magnitude of the entire task system becomes 1/N of the original.

Details:
1. When MongoDB’s number of tasks increases, array query is quite slow, and when the number of tasks reaches 5K, it is intolerable.
2. task processing n subtasks corresponding to each md5 must be completed before being deleted from MongoDB
3. The task can be reset after timeout

The improvement plan is as follows:
Due to the coupling of the original code, MongoDB cannot be completely abandoned, so we decided to add a Redis cache. N subtasks corresponding to md5 are distributed to N Redis queues (molecular dismantling tasks). A separate process synchronizes tasks from MongoDB to Redis, and clients no longer fetch tasks from MongoDB. The advantage of this is that the original MongoDB array query is abandoned. The synchronization process takes tasks from MongoDB according to the priority offset (indexed) of the tasks, so the speed is faster than array query. In this way, the client takes subtasks from the n queues of Redis and returns the task results to the original MongoDB task record (subtasks are returned according to md5).

Problems encountered in the improvement process:
Since there will be an update operation when the client returns to MongoDB, if all N subtasks are completed, the task will be deleted from MongoDB. One such problem is that MongoDB’s performance is very low under the condition of high concurrent writing after testing. The task processing speed of the whole task system is up to 200/s(16 cores, 16G, CentOS, cores 2.6.32-358.6.3.el6.x86_64). The reason is roughly that MongoDB’s performance will drop sharply due to lock table operation under the condition of frequent writing.

Specific issues:
(Think out of the Box) Can you come up with a good solution that can save the task state (subtask state) at least faster than MongoDB?

After a preliminary consideration, it is only for reference:

  1. First of all, mention the index, believe that you should add the index.
  2. There is a problem to confirm, the latest version of mongodb is still at the Database level. I don’t know which version you are using, and it hasn’t reached the granularity of the lock table (Collection), so it is worse if the write concurrency is large, but the performance should not be as bad as you described. Don’t understand, suggest to consider the possibility of task library?
  3. Can we consider saving the sub-task status separately from the main task status? The status of subtasks can be put into redis, and the main task is only responsible for its own status, so that the update frequency of each main task is reduced to 1/N, which can greatly reduce the pressure on the main task table in mongodb.
  4. Can I consider the main task status of background asynchronous single-thread sequential synchronization mongodb after the subtasks have completed or timed out?