Yixin Open Source Microservice Task Scheduling Platform (SIA-TASK)

  Microservices, Open source framework

Background

Both Internet applications and enterprise applications are full of batch processing tasks. Some task scheduling systems are often needed to help developers solve problems. With the gradual evolution of the micro-service architecture, the single architecture gradually evolved into a distributed and micro-service architecture. Under this background, many original task scheduling platforms can no longer meet the needs of business systems. Therefore, some distributed task scheduling platforms have emerged. Each of these platforms has its own characteristics, but each has its own shortcomings, such as not supporting task scheduling, high coupling with business, and not supporting cross-platform issues. It does not meet the requirements of the new generation of micro-service architecture, so Yixin Company has developed a micro-service task scheduling platform (SIA-TASK).

SIA is the abbreviation of Simple is Awesome, the basic development platform of Yixin Company. SIA-TASK (Microservice Task Scheduling Platform) is one of the important products. SIA-TASK conforms to the current microservice architecture mode and has the characteristics of cross-platform, orchestration, high availability, non-intrusion, consistency, asynchronous parallel, dynamic expansion, real-time monitoring, etc.

Project introduction

SIA-TASK is an integrated solution for task scheduling. Metadata collection is carried out on the task, then visual arrangement of the task is carried out, finally task scheduling is carried out, and the whole process monitoring is carried out on the task, thus being simple and easy to use. It is completely non-invasive to the business, and can generate the expected task scheduling model through simple and flexible configuration.

SIA-TASK uses the design idea of microservices for reference to obtain task metadata distributed on each task executor and upload it to the task registration center. Task scheduling can be carried out in an online way, and the task clock can be dynamically modified. HTTP is adopted as the task scheduling protocol, JSON data format is uniformly used, and the scheduling center analyzes the clock, executes the task flow and notifies the task.

Key terms

  • Task: basic execution unit, an HTTP calling interface exposed by the executor;
  • Job: It is composed of one or more tasks with mutual logical relationship (serial/parallel), and is the smallest unit scheduled by the task scheduling center.
  • Plan: It consists of several jobs executed in sequence, each job has its own execution cycle, and the plan has no execution cycle;
  • Task Scheduler: Scheduling according to the execution cycle of each job, that is, HTTP request according to the logic of plan, job and task;
  • Task orchestration center (Config): orchestration center uses tasks to create plans and jobs;
  • Task executor: receiving HTTP request to execute business logic;
  • Hunter:Spring project extension package is responsible for capturing Tasks in the executor and uploading to the registration center. Businesses can rely on this component for task writing.

Characteristics of Microservice Task Scheduling Platform

  • Based on the annotation automatic capture task, add @OnlineTask annotation to the method exposed as HTTP service. The @OnlineTask will automatically capture the IP address, port, request path, request method, request parameter format and other information of the method to upload to the task registration center (zookeeper) and synchronously write into persistent storage. This method is the task.
  • Based on annotation non-intrusive multithreading control, a single task instance must be kept running on a single thread. The task scheduling framework automatically intercepts @OnlineTask annotations for sin gle thread running control, so that it will not be rescheduled when a task is running. Moreover, the whole control process has no perception of developers.
  • The scheduler adapts to the task allocation, and when there is failure or abnormality in the task execution process. According to the strategy customized by the task, the task can be awakened at multiple points to ensure uninterrupted execution of the task.
  • A highly flexible task arrangement mode, SIA-TASK’s design idea is to take tasks as atoms and combine multiple tasks according to the relation of execution to form a task. At the same time, the runtime is divided into a task scheduling center and a task scheduling center, so that the scheduling of jobs and the scheduling of jobs are separated and do not affect each other. When we need to adjust the process of the job, we only need to deal with it in the scheduling center. At the same time, the scheduling center supports the organization of tasks in serial, parallel, branch and other ways. When there are different task instances of the same task, multiple scheduling methods are also supported for processing.

Design of Microservice Task Scheduling Platform

SIA-TASK is mainly divided into five parts:

  • Task executor
  • Task scheduling center
  • Task scheduling center
  • Zookeeper
  • Persistent storage (Mysql)

SIA-TASK’s main operating logic:

  1. Capture the tasks in the task executor through comments and report them to the task registration center.
  2. The task scheduling center acquires data from the task registration center for scheduling and storing into persistent storage
  3. The task scheduling center obtains scheduling information from persistent storage
  4. The task scheduling center accesses the task executor according to scheduling logic

UI preview

The home page provides multi-dimensional monitoring.

  • Scheduler Information: Displays scheduler information (load capacity, early warning value) and job distribution.
  • Scheduling information: displays scheduling times triggered by the scheduling center, job and task multi-dimensional scheduling statistics.
  • Docking project statistics: make statistics on the systems using the project, the number of tasks, the number of tasks, etc.

Scheduling monitoring provides real-time monitoring and display of submitted jobs.

  • Real-time monitoring of job status: the project team is taken as the unit panel to show the job runtime status.
  • Real-time log association: the real-time log association can be displayed by coloring the status icon.

Task Management: Provides operations related to task metadata

  • Task metadata entry: tasks in manual mode can be entered here.
  • Task connectivity test: provide task connectivity function test.
  • Other operations of task metadata: modification and deletion.


Job Management: Provides job-related operations

  • Task arrangement: to arrange tasks.
  • Publishing jobs: job creation, modification, and publishing.
  • Cascade Settings: Provide job settings with time dependency.

Log management

  • Local log: the log interface is simple and quick to query. The log provides a 7-day dispatch log for quick inquiry.

Open source address

Author: head of Yixin development platform/senior architect Liang Xin