This series of articles is《Node.js Design Patterns Second Edition》The translation of the original text and the reading notes were serialized and updated in GitHub.Synchronized translation links.

Welcome to pay attention to my column, and subsequent blog posts will be synchronized in the column:

• Encounter’s Digging Column
• Programming Thinking of Zhihu’s Column Encounter
• Segmentfault front-end station

Asynchronous Control Flow Patterns with ES2015 and Beyond

In the previous chapter, we learned how to use callbacks to handle asynchronous code and how to solve asynchronous problems such as callback hell code. The callback isJavaScriptAndNode.jsHowever, other alternatives have emerged. These alternatives are more complex so that asynchronous code can be processed in a more convenient way.

In this chapter, we will explore some representative alternatives.PromiseAndGenerator. as well asasync awaitThis is an innovative grammar that can be used in higher versions ofJavaScriptWhich also serves asECMAScript 2017Part of the distribution.

We will see how these alternatives simplify the way asynchronous control flows are handled. Finally, we will compare all these methods to understand all the advantages and disadvantages of all these methods and be able to choose the one that is most suitable for us wisely.Node.jsMethods required by the project.

Promise

As we mentioned in the previous chapter,CPS styleNot the only way to write asynchronous code. In fact,JavaScriptEcosystems provide interesting alternatives to the traditional callback model. One of the most famous choices isPromiseEspecially now it isECMAScript 2015Part of, and now available atNode.jsAvailable in.

What is Promise?

PromiseIs an abstract object, we usually allow functions to return an object namedPromiseObject that represents the final result of an asynchronous operation. Normally, we say when asynchronous operation is not completed, we sayPromiseObject atpendingState, when the operation completed successfully, we saidPromiseObject atresolveState, when the operation error is terminated, we saidPromiseObject atrejectStatus. OncePromiseInresolveOrreject, we believe that the current asynchronous operation is over.

To receive the correct result or error capture of an asynchronous operation, we can use thePromiseThethenMethods:

promise.then([onFulfilled], [onRejected])

In the previous code,onFulfilled()Is a function that will eventually receivePromiseThe correct results, andonRejected()Is another function that receives the cause of the exception, if any. Both parameters are optional.

To understandPromiseHow to convert our code, let’s consider the following points:

asyncOperation(arg, (err, result) => {
  if (err) {
    // 错误处理
  }
  // 正常结果处理
});

PromiseLet’s take this typicalCPSThe code is converted into better structured and more elegant code, as follows:

asyncOperation(arg)
  .then(result => {
    // 错误处理
  }, err => {
    // 正常结果处理
  });

then()A key feature of the method is that it returns to another synchronouslyPromiseObject. IfonFulfilled()OronRejected()Any one of the functions returnsx, thenthen()The returned by the methodPromiseThe object will look like this:

• IfxIs a value, then thisPromiseObject will be handled correctly (resolve)x
• IfxIs aPromiseObject orthenable, it will be handled correctly (resolve)x
• IfxIs an exception, an exception (reject)x

Note: thenable is a Promise-like object with then method.

This feature enables us to build a chainPromise, allowing us to easily arrange and combine our asynchronous operations. In addition, if we do not specify oneonFulfilled()OronRejected()Handler, the correct result or exception capture will be automatically forwarded to thePromiseNext in the chainPromise. For example, this allows us to automatically propagate errors throughout the chain until they areonRejected()Handler capture. along withPromiseChain, the order of the task execution suddenly becomes much simpler:

asyncOperation(arg)
  .then(result1 => {
    // 返回另一个Promise
    return asyncOperation(arg2);
  })
  .then(result2 => {
    // 返回一个值
    return 'done';
  })
  .then(undefined, err => {
    // 捕获Promise链中的异常
  });

The following figure shows the chainPromiseHow to work:

PromiseAnother important feature of isonFulfilled()AndonRejected()Functions are called asynchronously, as in the example above, in the last onethenFunctionresolveA synchronousPromise, it is also synchronized. This mode is avoidedZalgo(seeChapter2-Node.js Essential Patterns), making our asynchronous code more consistent and robust.

If inonFulfilled()OronRejected()Exception thrown in handler (usingthrowStatement), thenthen()The returned by the methodPromiseWill be automaticallyreject, throw an exception asrejectWhy? This is relative toCPSIs a huge advantage, because it means there isPromise, the exception will propagate automatically throughout the chain, andthrowStatement can finally be used.

In the past, many different libraries were implementedPromise, most of the time they are incompatible, which means that it is impossible to use differentPromiseLibrarythenableChain propagation error.

JavaScriptThe community worked very hard to resolve this limitation, and these efforts led toPromises / A +Establishment of specifications. This specification describes in detailthenThe behavior of the method provides a mutually compatible basis, which makes thePromiseObjects can be compatible with each other, out of the box.

RelevantPromises / A +For detailed specification, please refer toPromises/A+official website.

Implementation of Promise/A+

InJavaScriptChina andNode.jsThere are several implementationsPromises / A +Standard library. The following are the most popular:

• Bluebird
• Q
• RSVP
• Vow
• When.js
• ES2015 promises

The real difference between them is inPromises / A +Additional functions provided above the standard. As we said above, the standard definesthen()Methods andPromiseThe behavior of the parsing process, but it does not specify other functions, such as how to create from asynchronous callback-based functionsPromise.

In our example, we will use theES2015ThePromiseBecausePromiseObject fromNode.js 4It can be used later without any library.

For reference, the following areES2015ThePromiseApis provided:

constructor(new Promise（function（resolve, reject）{}）): creates a newPromiseWhich is determined based on parameters that pass two types as functionsresolveOrreject. The parameters of the constructor are explained as follows:

• resolve(obj)：resolveOnePromiseWith a parameterobjIfobjIs a value that is the result of the successful asynchronous operation passed. IfobjIs aPromiseOr onethenable, it will be handled correctly.
• reject(err)：rejectOnePromiseWith a parametererr. It isErrorAn instance of the.

Static Method of Promise Object

• Promise.resolve(obj): A will be createdresolveThePromiseExample
• Promise.reject(err): A will be createdrejectThePromiseExample
• Promise.all(iterable): returns a newPromiseInstance, and in theiterableCentral research institute

YesPromiseStatus isrejectWhen, the returnedPromiseThe state of the instance is set torejectIfiterableAt least one of themPromiseStatus isrejectWhen, the returnedPromiseThe instance state is also set toreject, andrejectThe reason is the first to berejectThePromiseObject’srejectThe reason.

• Promise.race(iterable): returns onePromiseExample, wheniterableAny one of themPromiseBeresolveOr byrejectWhen, the returnedPromiseFor the same reasonresolveOrreject.

Promise instance method

• Promise.then(onFulfilled, onRejected): This isPromiseThe basic method of. Its behavior is similar to what we described earlier.Promises / A +Standards compatible.
• Promise.catch(onRejected)This is justPromise.then(undefined，onRejected)The grammar of sugar.

It is worth mentioning that some Promise implementations provide another mechanism to create a new Promise, called deferreds S. We will not describe it here because it is not part of the ES2015 standard, but if you want to know more, you can read the Q document (https://github.com/kriskowal/ …Or When.js documents (https://github.com/cujojs/whe ….

Promisifying a Node.js callback style function

InJavaScriptIn, not all asynchronous functions and libraries support out of the boxPromise. In most cases, we must convert a typical callback-based function into a returnPromiseThis process is also calledpromisification.

Fortunately,Node.jsThe callback convention used in allows us to create a reusable function by using thePromiseObject to simplify anyNode.jsStylishAPI. Let’s create one calledpromisify()And include it in theutilities.jsIn the module (so that later in ourWeb crawler applicationUse it in):

module.exports.promisify = function(callbackBasedApi) {
  return function promisified() {
    const args = [].slice.call(arguments);
    return new Promise((resolve, reject) => {
      args.push((err, result) => {
        if (err) {
          return reject(err);
        }
        if (arguments.length <= 2) {
          resolve(result);
        } else {
          resolve([].slice.call(arguments, 1));
        }
      });
      callbackBasedApi.apply(null, args);
    });
  }
};

The previous function returned another function namedpromisified()A function of that represents the given in the inputcallbackBasedApiThepromisifiedVersion. Here’s how it works:

1. promisified()Function usagePromiseThe constructor creates a newPromiseObject and immediately returns it to the caller.
2. After passing toPromiseIn the function of the constructor, we ensure that the is passed to thecallbackBasedApiThis is a special callback function. Since we know that callbacks are always the last to be called, we just need to attach the callback function to thepromisified()Function (args）。
3. In a special callback, if we receive an error, we immediatelyrejectThis ..Promise.
4. If no error is received, we use a value or an array value toresolveThis ..PromiseDepending on the number of results passed to the callback.
5. Finally, we just need to use the parameter list we built to callcallbackBasedApi.

Most Promise already provides an out-of-the-box interface to convert a Node.js-style API into an API that returns Promise. For example, q has Q.denodeify () and Q.nbind (), Bluebird has Promise.promisify (), and When.js has node.lift ().

Sequential execution

After some necessary theories, we are now ready to transfer ourWeb crawler applicationConvert to UsePromiseThe form of. Let’s start with version 2 and download the link to a Web page directly.

Inspider.jsIn the module, the first step is to load ourPromiseImplement (we’ll use it later) andPromisifyingThe callback-based functions we intend to use:

const utilities = require('./utilities');
const request = utilities.promisify(require('request'));
const mkdirp = utilities.promisify(require('mkdirp'));
const fs = require('fs');
const readFile = utilities.promisify(fs.readFile);
const writeFile = utilities.promisify(fs.writeFile);

Now, we begin to change ourdownloadFunctions:

function download(url, filename) {
  console.log(`Downloading ${url}`);
  let body;
  return request(url)
    .then(response => {
      body = response.body;
      return mkdirp(path.dirname(filename));
    })
    .then(() => writeFile(filename, body))
    .then(() => {
      console.log(`Downloaded and saved: ${url}`);
      return body;
    });
}

The most important thing to note here is that we alsoreadFile()ReturnedPromiseregistered
OneonRejected()Function, which is used to handle the situation that a web page has not been downloaded (or the file does not exist). Also, see how we use it.throwTo deliveronRejected()Error in function.

Now that we have changed ourspider()Function, we modify the way it is called as follows:

spider(process.argv[2], 1)
  .then(() => console.log('Download complete'))
  .catch(err => console.log(err));

Notice how we used it for the first time.PromiseGrammatical sugarcatchTo deal withspider()Any error condition of the function. If we look at all the code we have written so far, we will be pleasantly surprised to find that we do not contain any error propagation logic because we are forced to do such things when using callback functions. This is obviously a huge advantage because it greatly reduces the template files in our code and the chance of losing any asynchronous errors.

Now, complete the only thing we are missingWeb crawler applicationThe second edition ofspiderLinks()Function, we will implement it later.

Sequential iteration

So far,Web crawler applicationThe code base is mainly rightPromiseAn overview of what it is and how it is used shows how it is used.PromiseRealize the simplicity and elegance of the sequential execution process. However, the code we are considering now only involves the execution of a set of known asynchronous operations. Therefore, the missing part of completing our exploration of the sequential execution process is to see how we use it.PromiseTo achieve iteration. Similarly, the second edition of the web spiderspiderLinks()Function is also a good example.

Let’s add the missing piece:

function spiderLinks(currentUrl, body, nesting) {
  let promise = Promise.resolve();
  if (nesting === 0) {
    return promise;
  }
  const links = utilities.getPageLinks(currentUrl, body);
  links.forEach(link => {
    promise = promise.then(() => spider(link, nesting - 1));
  });
  return promise;
}

In order to asynchronously iterate through all links of a web page, we must dynamically create onePromiseThe iterative chain of.

1. First, we define an emptyPromise,resolveForundefined. This ..PromiseJust to doPromiseThe starting point of the iteration chain of.
2. Then, we call the previous one in the chain in a loopPromiseThethen()Method to obtain the newPromiseTo updatePromiseVariable. This is what we usePromiseThe asynchronous iterative mode of.

In this way, the end of the cycle,promiseThe variable will contain the last in the loopthen()ReturnedPromiseObject, so it only works whenPromiseAll in the iterative chain ofPromiseObject isresolveBefore they can beresolve.

Note: The then method was called at the end to resolve the Promise object

Through this, we have usedPromiseObject rewrites ourWeb crawler application. We should be able to run it now.

Sequential iteration mode

In order to summarize the parts executed in this order, let’s extract a pattern to traverse a group in turn.Promise：

let tasks = [ /* ... */ ]
let promise = Promise.resolve();
tasks.forEach(task => {
  promise = promise.then(() => {
    return task();
  });
});
promise.then(() => {
  // 所有任务都完成
});

Usereduce()Method to replaceforEach()Method, allowing us to write more concise code:

let tasks = [ /* ... */ ]
let promise = tasks.reduce((prev, task) => {
  return prev.then(() => {
    return task();
  });
}, Promise.resolve());

promise.then(() => {
  //All tasks completed
});

As usual, through simple adjustment of this mode, we can collect the results of all tasks into an array, and we can implement onemappingAlgorithms, or building afilterWait.

The above mode uses loops to dynamically establish a chain of Promise.

concurrent execution

The other is suitable for use.PromiseThe execution flow of is a parallel execution flow. In fact, what we need to do is to use the built-inPromise.all(). This method creates anotherPromiseObject, only all in the inputPromiseAllresolveTime and talentresolve. This is a parallel execution because in its parametersPromiseThere is no order of execution between objects.

To demonstrate this, let’s look at ourWeb crawler applicationThe third edition of the, which downloads all links in the page in parallel. Let’s use it againPromiseUpdatespiderLinks()Function to implement parallel processes:

function spiderLinks(currentUrl, body, nesting) {
  if (nesting === 0) {
    return Promise.resolve();
  }
  const links = utilities.getPageLinks(currentUrl, body);
  const promises = links.map(link => spider(link, nesting - 1));
  return Promise.all(promises);
}

The pattern here iselements.map()In the iteration, an array is generated to store all asynchronous tasks, which is convenient to start at the same time.spider()Task. This time, in the loop, we don’t wait for the previous download to complete, and then start a new download task: all the download tasks start one by one in the loop. After that, we usedPromise.all()Method, which returns a newPromiseObject when all thePromiseObjects areresolveWhen, thisPromiseThe object will beresolve. In other words, all the downloading tasks are completed, which is exactly what we want.

Restrict parallel execution

Unfortunately,ES2015ThePromise APIIt does not provide a native way to limit the number of concurrent tasks, but we can always rely on what we have learned about using commonJavaScriptTo limit concurrency. In fact, we areTaskQueueThe pattern implemented in the class can be easily adjusted to support the task of returning the promise. This can be easily modifiednext()Method to complete:

class TaskQueue {
  constructor(concurrency) {
    this.concurrency = concurrency;
    this.running = 0;
    this.queue = [];
  }

  pushTask(task) {
    this.queue.push(task);
    this.next();
  }

  next() {
    while (this.running < this.concurrency && this.queue.length) {
      const task = this.queue.shift();
      task().then(() => {
        this.running--;
        this.next();
      });
      this.running++;
    }
  }
}

Instead of using a callback function to handle tasks, we simply callPromiseThethen().

Let’s go backspider.jsModule and modify it to support our new version ofTaskQueueClass. First, we ensure that we define aTaskQueueNew instance of:

const TaskQueue = require('./taskQueue');
const downloadQueue = new TaskQueue(2);

Then, it’s oursspiderLinks()Function. The amendment here is also very simple:

function spiderLinks(currentUrl, body, nesting) {
  if (nesting === 0) {
    return Promise.resolve();
  }
  const links = utilities.getPageLinks(currentUrl, body);
  // 我们需要如下代码，用于创建Promise对象
  // 如果没有下列代码，当任务数量为0时，将永远不会resolve
  if (links.length === 0) {
    return Promise.resolve();
  }
  return new Promise((resolve, reject) => {
    let completed = 0;
    let errored = false;
    links.forEach(link => {
      let task = () => {
        return spider(link, nesting - 1)
          .then(() => {
            if (++completed === links.length) {
              resolve();
            }
          })
          .catch(() => {
            if (!errored) {
              errored = true;
              reject();
            }
          });
      };
      downloadQueue.pushTask(task);
    });
  });
}

There are several points worth our attention in the above code:

• First of all, we need to return to usePromiseThe new created by the constructorPromiseObject. As we will see, this enables us to manually complete all tasks in the queueresolveOursPromiseObject.
• Then, we should see how we define the task. What we have done is to put aonFulfilled()The call to the callback function is added to thespider()ReturnedPromiseObject, so we can calculate the number of download tasks completed. When the number of downloads completed is the same as the number of links in the current page, we know that the task has been processed, so we can call outsidePromiseTheresolve()Function.

The Promises/A+specification stipulates that the onFulfilled () and onRejected () callback functions of the then () method can only be called once (only onFulfilled () and onRejected ()) . The implementation of Promise interface ensures that Promise can be resolved or rejected only once even if we manually call resolve or reject many times.

Now, usePromiseTheWeb crawler applicationThe 4th edition of should be ready. We may once again notice how the download tasks run in parallel, and the number of concurrent tasks is limited to 2.

Expose callback function and Promise in public API

As we learned earlier,PromiseCan be used as a good substitute for callback functions. They make our code more readable and easier to understand. AlthoughPromiseIt brings many advantages, but it also requires developers to understand many concepts that are not easy to understand so as to use them correctly and skillfully. For this and other reasons, in some cases, compared withPromiseIn general, many developers prefer callback functions.

Now let’s imagine that we want to build a common library that performs asynchronous operations. What do we need to do? We have created a callback functionAPIOr is it an orientationPromiseTheAPI? Or both?

This is a problem faced by many well-known libraries, and there are at least two methods worth mentioning to enable us to provide a multifunctionalAPI.

Likerequest,redisAndmysqlThe first method used by such a library is to provide a simple callback function-basedAPIIf necessary, developers can choose to expose functions. Some of these libraries provide tool functions toPromiseAsynchronous callback, but developers still need to expose theAPITo be able to usePromiseObject.

The second method is more transparent. It also provides a callback-orientedAPIHowever, it makes callback parameters optional. Every time a callback is passed as a parameter, the function will run normally, executing the callback on completion or failure. When the callback is not passed, the function returns an immediatelyPromiseObject. This method effectively combines callback functions andPromiseSo that developers can choose which interface to use when calling, without having to do it in advance.PromiseHua. Many libraries, such asmongooseAndsequelize, all support this method.

Let’s look at a simple example. Suppose we want to implement a module that performs division asynchronously:

module.exports = function asyncDivision(dividend, divisor, cb) {
  return new Promise((resolve, reject) => { // [1]
    process.nextTick(() => {
      const result = dividend / divisor;
      if (isNaN(result) || !Number.isFinite(result)) {
        const error = new Error('Invalid operands');
        if (cb) {
          cb(error); // [2]
        }
        return reject(error);
      }
      if (cb) {
        cb(null, result); // [3]
      }
      resolve(result);
    });
  });
};

The code of this module is very simple, but there are some details worth emphasizing:

• First, return to usePromiseThe new promise created by the constructor of. We define all logic within the constructor parameter function.
• In the event of an error, werejectThis ..PromiseHowever, if the callback function is passed as a parameter when called, we also execute the callback to propagate the error.
• After the calculation results, weresolveTake thisPromiseHowever, if there is a callback function, we will also propagate the result to the callback function.

Let’s now look at how to use callback functions andPromiseTo use this module:

// 回调函数的方式
asyncDivision(10, 2, (error, result) => {
  if (error) {
    return console.error(error);
  }
  console.log(result);
});

// Promise化的调用方式
asyncDivision(22, 11)
  .then(result => console.log(result))
  .catch(error => console.error(error));

It should be clear that developers who are about to start using new modules similar to the ones described above will easily choose the style that best suits their needs without having to use it in the hope of doing so.PromiseWhen introducing externalpromisificationFunctions.

Generators

ES2015The specification introduces another mechanism that can be used to simplify, among other new functionsNode.jsAsynchronous control flow of application program. We are talking aboutGenerator, also known assemi-coroutines. They are generalizations of subprograms and can have different entry points. In a normal function, in fact we can only have one entry point, which corresponds to the call of the function itself.GeneratorSimilar to general functions, but can be paused (usingyieldStatement) and then continue execution at a later time. When implementing iterators,GeneratorThis is especially useful because we have discussed how to use iterators to implement important asynchronous control flow patterns, such as sequential execution and limiting parallel execution.

Generators foundation

Before we explore the use ofGeneratorBefore realizing asynchronous control flow, it is very important to learn some basic concepts. Let’s start with grammar. This can be done by appending after the function key*(asterisk) operator to declareGeneratorFunctions:

function* makeGenerator() {
  // body
}

InmakeGenerator()Function, we can use keywordsyieldPause execution and return the value passed to it by the caller:

function* makeGenerator() {
  yield 'Hello World';
  console.log('Re-entered');
}

In the previous code,GeneratorviayieldA stringHello WorldPauses execution of the current function. WhenGeneratorOn recovery, execution starts with the following statement:

console.log('Re-entered');

makeGenerator()The function is essentially a factory that returns a new when calledGeneratorObject:

const gen = makeGenerator();

The most important way to generate objects is tonext()Which is used for startup/recoveryGenerator, and returns an object in the following form:

{
  value: <yielded value>
  done: <true if the execution reached the end>
}

This object containsGenerator yieldThe value of the and an indicationGeneratorSymbol for whether execution has been completed.

A simple example

For demonstration purposesGenerator, let’s create a file calledfruitGenerator.jsNew module for:

function* fruitGenerator() {
  yield 'apple';
  yield 'orange';
  return 'watermelon';
}
const newFruitGenerator = fruitGenerator();
console.log(newFruitGenerator.next()); // [1]
console.log(newFruitGenerator.next()); // [2]
console.log(newFruitGenerator.next()); // [3]

The preceding code prints the following output:

{ value: 'apple', done: false }
{ value: 'orange', done: false }
{ value: 'watermelon', done: true }

We can explain the above phenomenon as follows:

• First callnewFruitGenerator.next()At that time,GeneratorThe function starts executing until the first is reachedyieldStatement, the command pausesGeneratorThe function executes and returns the valueappleReturn to caller.
• On the second callnewFruitGenerator.next()At that time,GeneratorFunction resumes execution from the secondyieldStatement, which in turn suspends execution, and at the same timeorangeReturn to caller.
• newFruitGenerator.next()The last call to resulted inGeneratorFunction is executed from its lastyieldRecovery, a return statement that terminatesGeneratorFunction, returnwatermelonIn the result objectdoneThe property is set totrue.

Generators as Iterators

In order to better understand whyGeneratorFunctions are very useful for implementing iterators. Let’s build an example. Before we calliteratorGenerator.jsIn the new module of, we write the following code:

function* iteratorGenerator(arr) {
  for (let i = 0; i < arr.length; i++) {
    yield arr[i];
  }
}
const iterator = iteratorGenerator(['apple', 'orange', 'watermelon']);
let currentItem = iterator.next();
while (!currentItem.done) {
  console.log(currentItem.value);
  currentItem = iterator.next();
}

This code should print the elements in the array as follows:

apple
orange
watermelon

In this example, every time we calliterator.next()We will all recoverGeneratorFunctionalforCycle throughyieldThe next item in the array runs another loop. This demonstrates how to maintain during function calls.GeneratorThe state of. When execution continues, the values of the loop and all variables are the same asGeneratorFunction execution is suspended in exactly the same state.

Pass values to Generators

Now we continue to studyGeneratorFirst, learn how to pass the value back toGeneratorFunction. This is actually very simple, all we need to do is tonext()The method provides a parameter and the value is used asGeneratorWithin functionyieldThe return value of the statement is provided.

To show this, let’s create a new simple module:

function* twoWayGenerator() {
  const what = yield null;
  console.log('Hello ' + what);
}
const twoWay = twoWayGenerator();
twoWay.next();
twoWay.next('world');

When executed, the previous code will be outputHello world. We make the following explanation:

• First callnext()In the method,GeneratorThe function reaches the firstyieldStatement, and then pause.
• Whennext('world')When called,GeneratorFunction from the last stop position, that is, the last timeyieldStatement point recovery, but this time we have a value passed toGeneratorFunction. This value will be assigned towhatVariable. The generator then executesconsole.log()Command and terminate.

In a similar way, we can forceGeneratorFunction throws an exception. This can be achieved by usingGeneratorFunctionalthrowMethod, as shown in the following example:

const twoWay = twoWayGenerator();
twoWay.next();
twoWay.throw(new Error());

In this last piece of code,twoWayGenerator()The function will set theyieldThe function throws an exception when it returns. It’s like starting fromGeneratorFunction throws an exception inside, which means that it can be used liketry ... catchBlock to capture and handle exceptions.

Generator Implements Asynchronous Control Flow

You must want to knowGeneratorFunction to help us handle asynchronous operations. We can do this by creating an acceptanceGeneratorFunction as a special function of parameters to demonstrate this, and allows us to inGeneratorThe function uses asynchronous code internally. This function should pay attention to recovery when asynchronous operation is completed.GeneratorFunction execution. We will call this functionasyncFlow()：

function asyncFlow(generatorFunction) {
  function callback(err) {
    if (err) {
      return generator.throw(err);
    }
    const results = [].slice.call(arguments, 1);
    generator.next(results.length > 1 ? results : results[0]);
  }
  const generator = generatorFunction(callback);
  generator.next();
}

Take one of the previous functionsGeneratorFunction as input and immediately call:

const generator = generatorFunction(callback);
generator.next();

generatorFunction()Accept a special callback function as a parameter whengenerator.throw()If an error is received, it will be returned immediately. In addition, by comparing the values received in the callback functionresultsReturn valueGeneratorFunction continuationGeneratorFunction execution:

if (err) {
  return generator.throw(err);
}
const results = [].slice.call(arguments, 1);
generator.next(results.length > 1 ? results : results[0]);

In order to illustrate the power of this simple auxiliary function, we created a program calledclone.jsThis module only creates its own clone. Paste what we just createdasyncFlow()Function, the core code is as follows:

const fs = require('fs');
const path = require('path');
asyncFlow(function*(callback) {
  const fileName = path.basename(__filename);
  const myself = yield fs.readFile(fileName, 'utf8', callback);
  yield fs.writeFile(`clone_of_${filename}`, myself, callback);
  console.log('Clone created');
});

Obviously, there isasyncFlow()With the help of the function, we can write asynchronous code in the same way as we write synchronous blocking functions. And the principle behind this result is very clear. Once the asynchronous operation ends, the callback function passed to each asynchronous function will continueGeneratorFunction execution. There is nothing complicated, but the result is really surprising.

There are two other changes in this technology, one isPromiseThe use of, the other isthunks.

Thunk used in the generator-based control flow is only a simple function, which partially applies all parameters of the original function except callback. The return value is another function that only accepts callbacks as arguments. For example, the thunkified version of fs.readFile () is as follows:

function readFileThunk(filename, options) {
  return function(callback) {
    fs.readFile(filename, options, callback);
  }
}

thunkAndPromiseAll allow us to create a without callbackGeneratorFunction is passed as a parameter, for example, using thethunkTheasyncFlow()The version is as follows:

function asyncFlowWithThunks(generatorFunction) {
  function callback(err) {
    if (err) {
      return generator.throw(err);
    }
    const results = [].slice.call(arguments, 1);
    const thunk = generator.next(results.length > 1 ? results : results[0]).value;
    thunk && thunk(callback);
  }
  const generator = generatorFunction();
  const thunk = generator.next().value;
  thunk && thunk(callback);
}

This technique is readinggenerator.next()The return value of containsthunk. The next step is to inject special callback function callsthunkIn itself. This allows us to write the following code:

asyncFlowWithThunk(function*() {
  const fileName = path.basename(__filename);
  const myself = yield readFileThunk(__filename, 'utf8');
  yield writeFileThunk(`clone_of_${fileName}`, myself);
  console.log("Clone created")
});

Gernator-based control flow using co

As you should have guessed,Node.jsThe ecosystem will useGeneratorFunction to provide some solutions for handling asynchronous control flow, for example,suspendIs one of the oldest supportPromise、thunksAndNode.jsA library of style callback functions and normal style callback functions. Also, most of what we analyzed earlierPromiseLibraries all provide tool functions that enableGeneratorAndPromiseCan be used together.

We choosecoAs an example of this chapter. It supports many types ofyieldables, some of which are:

• Thunks
• Promises
• Arrays(parallel execution)
• Objects(parallel execution)
• Generators(Entrusted)
• GeneratorFunction (delegate)

There are also many frameworks or libraries based oncoEcosystem, including the following:

• Web framework, the most popular iskoa
• Libraries that implement specific control flow patterns
• Packaging is popularAPICompatiblecoThe library of

We usecoTo realize ourGeneratorVersion ofWeb crawler application.

In order toNode.jsStyle functions are converted tothunks, we will use a method calledthunkifyThe library.

Sequential execution

Let’s revise itWeb crawler applicationAt the beginning of version 2 of theGeneratorFunction sumcoThe actual exploration of. The first thing we need to do is to load our dependency package and generate the functions we want to use.thunkifiedVersion. These will be inspider.jsThe beginning of the module:

const thunkify = require('thunkify');
const co = require('co');
const request = thunkify(require('request'));
const fs = require('fs');
const mkdirp = thunkify(require('mkdirp'));
const readFile = thunkify(fs.readFile);
const writeFile = thunkify(fs.writeFile);
const nextTick = thunkify(process.nextTick);

Looking at the above code, we can notice that it is similar to the previous section of this chapterpromisifyChemicalAPISome similarities in the code of. At this point, it is interesting that if we use ourpromisifiedVersion of the function to replacethunkifiedThe code will remain exactly the same, thanks tocoSupportthunkAndPromiseObject asyieldableObject. In fact, we can even use it in the same application if we want to.thunkAndPromiseEven in the sameGeneratorIn the function. In terms of flexibility, this is a huge advantage, because it enables us to use information based onGeneratorFunction to solve problems in our applications.

All right, now let’s startdownload()Function to aGeneratorFunctions:

function* download(url, filename) {
  console.log(`Downloading ${url}`);
  const response = yield request(url);
  const body = response[1];
  yield mkdirp(path.dirname(filename));
  yield writeFile(filename, body);
  console.log(`Downloaded and saved ${url}`);
  return body;
}

By usingGeneratorAndcoOurdownload()Functions become much simpler. When we need to do asynchronous operations, we use asynchronousGeneratorFunction asthunkTo translate the previous content intoGeneratorFunction and use theyieldClause.

Then we began to realize ourspider()Functions:

function* spider(url, nesting) {
  cost filename = utilities.urlToFilename(url);
  let body;
  try {
    body = yield readFile(filename, 'utf8');
  } catch (err) {
    if (err.code !== 'ENOENT') {
      throw err;
    }
    body = yield download(url, filename);
  }
  yield spiderLinks(url, body, nesting);
}

An interesting detail from the above code is that we can usetry...catchStatement block to handle exceptions. We can also usethrowTo spread the anomaly. Another detail is that weyieldOursdownload()Function, and this function is neither athunk, is not a.promisifiedFunction, just another oneGeneratorFunction. This is no problem, becausecoWe also support others.GeneratorsAs ayieldables.

Final conversionspiderLinks()In this function, we recursively download a link to a web page. Use in this functionGenerators, appear much simpler:

function* spiderLinks(currentUrl, body, nesting) {
  if (nesting === 0) {
    return nextTick();
  }
  const links = utilities.getPageLinks(currentUrl, body);
  for (let i = 0; i < links.length; i++) {
    yield spider(links[i], nesting - 1);
  }
}

Look at the above code. Although there is no pattern to show for sequential iteration.GeneratorAndcoWe have done a lot of things to help us write asynchronous code in a synchronous way.

Look at the most important part, the entrance to the program:

co(function*() {
  try {
    yield spider(process.argv[2], 1);
    console.log(`Download complete`);
  } catch (err) {
    console.log(err);
  }
});

This is the only place to callco(...)To package oneGenerator. In fact, once we do this,coIt automatically encapsulates what we pass toyieldAny of theGeneratorFunction, and this process is recursive, so the rest of the program is related to whether we usecoIs completely irrelevant, although iscoEncapsulated inside.

It should be operational by now.GeneratorFunction rewrittenWeb crawler applicationHere we go.

concurrent execution

Unfortunately, althoughGeneratorIt is convenient to execute in sequence, but it cannot be directly used to execute a group of tasks in parallel, at least not onlyyieldAndGenerator. Previously, the pattern we used in this case was simply based on a callback orPromiseFunction of, but uses theGeneratorFunction, everything will look simpler.

Fortunately, if parallel execution of concurrency is not limited,coIt can already be passedyieldOnePromiseObjects,thunk、GeneratorFunctions, even includingGeneratorFunction.

With this in mind, ourWeb crawler applicationThe third edition can be rewrittenspiderLinks()Function to make the following changes:

function* spiderLinks(currentUrl, body, nesting) {
  if (nesting === 0) {
    return nextTick();
  }
  const links = utilities.getPageLinks(currentUrl, body);
  const tasks = links.map(link => spider(link, nesting - 1));
  yield tasks;
}

But all the above functions do is get all the tasks, which are essentially passedGeneratorFunction to implement asynchronous, ifcoThethunkAn inner pair containsGeneratorThe array of functions uses theyield, these tasks will be executed in parallel. Outer layerGeneratorThe function waits untilyieldClause is executed in parallel before continuing.

Next we will look at how to use a callback function-based approach to solve the same parallel flow. We rewrite it this way.spiderLinks()Functions:

function spiderLinks(currentUrl, body, nesting) {
  if (nesting === 0) {
    return nextTick();
  }
  // 返回一个thunk
  return callback => {
    let completed = 0,
      hasErrors = false;
    const links = utilities.getPageLinks(currentUrl, body);
    if (links.length === 0) {
      return process.nextTick(callback);
    }

    function done(err, result) {
      if (err && !hasErrors) {
        hasErrors = true;
        return callback(err);
      }
      if (++completed === links.length && !hasErrors) {
        callback();
      }
    }
    for (let i = 0; i < links.length; i++) {
      co(spider(links[i], nesting - 1)).then(done);
    }
  }
}

We usecoparallel runningspider()Function, callingGeneratorThe function returned aPromiseObject. In this way, waitPromiseCalled after completiondone()Function. Usually, based onGeneratorAll libraries that control flow have this function, so you can always add aGeneratorConvert to a callback-based or callback-basedPromiseThe function of.

To open multiple download tasks in parallel, we only need to reuse the callback-based parallel execution mode defined earlier. We should also note that we willspiderLinks()Convert to athunk(not one anymoreGeneratorFunction). This allows us to have a callback function to call when all parallel tasks are completed.

What is mentioned above is to convert a Generator function into a thunk mode, so that it can support other callback-based or Promise-based control flow algorithms, and can write asynchronous code through synchronous blocking code style.

Restrict parallel execution

Now that we know how to handle asynchronous execution processes, it should be easy to plan ourWeb crawler applicationThe implementation of the fourth version of the, which imposes a limit on the number of concurrent download tasks. We have several options to do this. Some of these programmes are as follows:

• Using the callback-based previously implementedTaskQueueClass. All we need isthunkifyOursGeneratorFunction and the callback function it provides.
• Use based onPromiseTheTaskQueueClass and ensure that eachGeneratorFunctions are converted to a returnPromiseThe function of the object.
• Useasync,thunkifyThe tool functions we intend to use, in addition, we need to use theGeneratorFunctions are converted into callback-based modes so that they can be better used by this library.
• Use based oncoThe library in the ecosystem of, especially for this kind of scene library, such asco-limiter.
• Implement a custom algorithm based on producer-consumer model, which is similar toco-limiterThe internal implementation principle of is the same.

In order to learn, we chose the last scheme, even helping us to better understand a pattern that is often synchronized with synergetics (also with threads and processes).

Producer-consumer model

Our goal is to use queues to provide a fixed number ofworkersAs many concurrency levels as we want to set. In order to implement this algorithm, we will base on the definition earlier in this chapterTaskQueueClass rewrite:

class TaskQueue {
  constructor(concurrency) {
    this.concurrency = concurrency;
    this.running = 0;
    this.taskQueue = [];
    this.consumerQueue = [];
    this.spawnWorkers(concurrency);
  }
  pushTask(task) {
    if (this.consumerQueue.length !== 0) {
      this.consumerQueue.shift()(null, task);
    } else {
      this.taskQueue.push(task);
    }
  }
  spawnWorkers(concurrency) {
    const self = this;
    for (let i = 0; i < concurrency; i++) {
      co(function*() {
        while (true) {
          const task = yield self.nextTask();
          yield task;
        }
      });
    }
  }
  nextTask() {
    return callback => {
      if (this.taskQueue.length !== 0) {
        return callback(null, this.taskQueue.shift());
      }
      this.consumerQueue.push(callback);
    }
  }
}

Let’s analyze thisTaskQueueThe new implementation of the class. The first is in the constructor. Need to call oncethis.spawnWorkers()Because this is startupworkerThe method of.

OursworkerIt’s very simple, they just useco()Immediate execution of packagingGeneratorFunction, so eachGeneratorFunctions can be executed in parallel. Inside, eachworkerIs running in a dead loop (while(true){}) has been blocked (yield) until a new task is available in the queue (yield self.nextTask()Once a new task can be performed,yieldThis asynchronous task will not be completed until it is completed. You may want to know how we can limit parallel execution and leave the next task waiting in the queue. The answer isnextTask()In the method. Let’s take a closer look at the principle of this method:

nextTask() {
  return callback => {
    if (this.taskQueue.length !== 0) {
      return callback(null, this.taskQueue.shift());
    }
    this.consumerQueue.push(callback);
  }
}

Let’s look at what happens inside this function. This is the core of this pattern:

1. This method returns a value for thecoIs a legal oneyieldableThethunk.
2. As long astaskQueueThere is the next task in the instance generated by the class.thunkThe callback function of will be called immediately. When a callback function is called, it immediately unlocks oneworkerThe blocking state of,yieldThis task.
3. If there are no more tasks in the queue, the callback function itself will be placedconsumerQueueChina. Through this approach, we will have aworkerLeave idle (idle) mode. Once we have a new task to deal with, inconsumerQueueThe callback function in the queue will be called and wake us up immediately.workerPerform asynchronous processing.

Now, in order to understandconsumerQueueIdle in queueworkerHow to resume work, we need to analyzepushTask()Methods. If a callback function is currently available,pushTask()The method calls theconsumerQueueThe first callback function in the queue, which cancels theworkerThe lock of the. If no callback function is available, this means that allworkerThey are all in working status, only need to add a new task totaskQueueTask queue.

InTaskQueueClass,workerActing as a consumer and callingpushTask()The role of a function can be considered as a producer. This model shows us oneGeneratorFunctions can actually be similar to a thread or process. In fact, the problem between producers and consumers is the most common problem when studying inter-process communication and synchronization, but as we have already mentioned, it is also a common example for processes and threads.

Limit the concurrency of download tasks

Now that we have used itGeneratorFunctions and producer-consumer models implement a restricted parallel algorithm and have been implemented inWeb crawler applicationVersion 4 applies it to limit the number of concurrent downloads in. First, we load and initialize aTaskQueueObject:

const TaskQueue = require('./taskQueue');
const downloadQueue = new TaskQueue(2);

Then, modifyspiderLinks()Function. It is similar to the previous version that does not restrict concurrency, so we only show the modified part here, mainly by calling the new version.TaskQueueThe of the instance generated by the classpushTask()Method to restrict parallel execution:

function spiderLinks(currentUrl, body, nesting) {
  //...
  return (callback) => {
    //...
    function done(err, result) {
      //...
    }
    links.forEach(function(link) {
      downloadQueue.pushTask(function*() {
        yield spider(link, nesting - 1);
        done();
      });
    });
  }
}

In each task, we call immediately after the download is completeddone()Function, so we can calculate how many links have been downloaded and notify when the download is complete.thunkThe callback function of the.

Use Asyncaway New Syntax with Babel

Callback function,PromiseAndGeneratorFunctions are used for processingJavaScriptAndNode.jsHow to solve asynchronous problems. As we can see,GeneratorThe real meaning of is that it provides a way to pause the execution of a function and then wait for the previous task to complete before continuing. We can use this feature to write asynchronous code and let developers write asynchronous code in a synchronous blocking code style. The execution of the current function will not resume until the result of the asynchronous operation is returned.

But ..GeneratorFunctions are mostly used to handle iterators, however, iterators are somewhat cumbersome to use in asynchronous code. The code may be difficult to understand, resulting in poor readability and maintainability of the code.

But there will be a more concise grammar in the near future. In fact, this proposal will soon be introduced intoESMASCript 2017This specification definesasyncFunction syntax.

asyncThe function specification introduces two keywords (asyncAndawait) to the nativeJavaScriptLanguage, improve the way we write asynchronous code.

To understand the usage and advantages of this grammar, let’s look at a simple example:

const request = require('request');

function getPageHtml(url) {
  return new Promise(function(resolve, reject) {
    request(url, function(error, response, body) {
      resolve(body);
    });
  });
}
async function main() {
  const html = await getPageHtml('http://google.com');
  console.log(html);
}

main();
console.log('Loading...');

In the above code, there are two functions:getPageHtmlAndmain. The first function is to extract a givenURLOf a remote web page ofHTMLDocument code. It is worth noting that this function returns aPromiseObject.

The point ismainFunction, because it is used hereasyncAndawaitKey words. The first thing to notice is that the function must be based onasyncThe keyword is prefixed. This means that this function executes asynchronous code and allows it to be used within the function bodyawaitKey words.awaitKeyword ingetPageHtmlBefore calling, tellJavaScriptThe interpreter waits before proceeding with the next instruction.getPageHtmlReturnedPromiseThe result of the object. In this way,mainWhich part of the code inside the function is asynchronous, it will wait for the completion of the asynchronous code before continuing to perform subsequent operations, and will not block the normal execution of the rest of the program. In fact, the console prints stringsLoading ..., followed by Google’s home pageHTMLCode.

Is this method more readable and easier to understand? Unfortunately, this proposal has not yet been finalized. Even if this proposal is passed, we need to wait for the next version.
TheECMAScriptStandardize and integrate it intoNode.jsOnly after that can we use this new grammar. So what did we do today? Just waiting aimlessly? No, of course not! We can already use it in our codeasync awaitGrammar, as long as we use itBabel.

Installing and Running Babel

BabelIs aJavaScriptCompiler (or translator), can use syntax converter to convert the high version ofJavaScriptCode conversion to otherJavaScriptCode. Syntax converters allow us to write and use, for exampleES2015,ES2016,JSXAnd other new syntax to translate into later compatible codeJavaScriptOperating environment such as browser orNode.jsBoth can be usedBabel.

Use in projectsnpmInstallationBabel, the command is as follows:

npm install --save-dev babel-cli

We also need to install plug-ins to supportasync awaitInterpretation or translation of grammar:

npm install --save-dev babel-plugin-syntax-async-functions babel-plugin-tranform-async-to-generator

Now suppose we want to run our previous example (calledindex.js）。 We need to start with the following command:

node_modules/.bin/babel-node --plugins "syntax-async-functions,transform-async-to-generator" index.js

In this way, we use supportasync awaitTo dynamically convert source code.Node.jsThe actual running is backward compatible code stored in memory.

BabelIt can also be configured as a code building tool to save the translated or interpreted code into the local file system, which is convenient for us to deploy and run the generated code.

About how to install and configure Babel, you can go to the official website.https://babeljs.ioConsult relevant documents.

Comparison of Several Ways

Now, what should we do about itJavaScriptWe have a better understanding and summary of the asynchronous problem of. The advantages and disadvantages of several mechanisms are summarized in the following table:

It is worth mentioning that we have chosen to introduce only the most popular or widely used solutions for handling asynchronous control processes in this chapter, but for example, Fibers (https://npmjs.org/package/fibers) and Streamline (https://npmjs.org/pAckage/streamline) is also worth seeing.

Summary

In this chapter, we analyzed some methods to deal with asynchronous control flowPromise、GeneratorFunctions and the upcomingasync awaitGrammar.

We learned how to use these methods to write more concise and readable asynchronous code. We discussed some of the most important advantages and disadvantages of these methods and realized that even if they were very useful, it would take some time to master them. This is why these methods have not completely replaced callbacks that are still very useful in many cases. As a developer, one should analyze the actual situation and decide which solution to use. If you are building a common library that performs asynchronous operations, you should provide an easy-to-useAPIEven for developers who only want to use callbacks.

In the next chapter, we will explore another mechanism related to asynchronous code execution, which is also the wholeNode.jsAnother basic building block in the ecosystem:streams.