This series of articles is《Node.js Design Patterns Second Edition》The translation of the original text and the reading notes were serialized and updated in GitHub.Synchronized translation links.
Welcome to pay attention to my column, and subsequent blog posts will be synchronized in the column:
- Encounter’s Digging Column
- Programming Thinking of Zhihu’s Column Encounter
- Segmentfault front-end station
Asynchronous Control Flow Patterns with ES2015 and Beyond
In the previous chapter, we learned how to use callbacks to handle asynchronous code and how to solve asynchronous problems such as callback hell code. The callback isJavaScript
AndNode.js
However, other alternatives have emerged. These alternatives are more complex so that asynchronous code can be processed in a more convenient way.
In this chapter, we will explore some representative alternatives.Promise
AndGenerator
. as well asasync await
This is an innovative grammar that can be used in higher versions ofJavaScript
Which also serves asECMAScript 2017
Part of the distribution.
We will see how these alternatives simplify the way asynchronous control flows are handled. Finally, we will compare all these methods to understand all the advantages and disadvantages of all these methods and be able to choose the one that is most suitable for us wisely.Node.js
Methods required by the project.
Promise
As we mentioned in the previous chapter,CPS style
Not the only way to write asynchronous code. In fact,JavaScript
Ecosystems provide interesting alternatives to the traditional callback model. One of the most famous choices isPromise
Especially now it isECMAScript 2015
Part of, and now available atNode.js
Available in.
What is Promise?
Promise
Is an abstract object, we usually allow functions to return an object namedPromise
Object that represents the final result of an asynchronous operation. Normally, we say when asynchronous operation is not completed, we sayPromise
Object atpending
State, when the operation completed successfully, we saidPromise
Object atresolve
State, when the operation error is terminated, we saidPromise
Object atreject
Status. OncePromise
Inresolve
Orreject
, we believe that the current asynchronous operation is over.
To receive the correct result or error capture of an asynchronous operation, we can use thePromise
Thethen
Methods:
promise.then([onFulfilled], [onRejected])
In the previous code,onFulfilled()
Is a function that will eventually receivePromise
The correct results, andonRejected()
Is another function that receives the cause of the exception, if any. Both parameters are optional.
To understandPromise
How to convert our code, let’s consider the following points:
asyncOperation(arg, (err, result) => {
if (err) {
// 错误处理
}
// 正常结果处理
});
Promise
Let’s take this typicalCPS
The code is converted into better structured and more elegant code, as follows:
asyncOperation(arg)
.then(result => {
// 错误处理
}, err => {
// 正常结果处理
});
then()
A key feature of the method is that it returns to another synchronouslyPromise
Object. IfonFulfilled()
OronRejected()
Any one of the functions returnsx
, thenthen()
The returned by the methodPromise
The object will look like this:
- If
x
Is a value, then thisPromise
Object will be handled correctly (resolve
)x
- If
x
Is aPromise
Object orthenable
, it will be handled correctly (resolve
)x
- If
x
Is an exception, an exception (reject
)x
Note: thenable is a Promise-like object with then method.
This feature enables us to build a chainPromise
, allowing us to easily arrange and combine our asynchronous operations. In addition, if we do not specify oneonFulfilled()
OronRejected()
Handler, the correct result or exception capture will be automatically forwarded to thePromise
Next in the chainPromise
. For example, this allows us to automatically propagate errors throughout the chain until they areonRejected()
Handler capture. along withPromise
Chain, the order of the task execution suddenly becomes much simpler:
asyncOperation(arg)
.then(result1 => {
// 返回另一个Promise
return asyncOperation(arg2);
})
.then(result2 => {
// 返回一个值
return 'done';
})
.then(undefined, err => {
// 捕获Promise链中的异常
});
The following figure shows the chainPromise
How to work:
Promise
Another important feature of isonFulfilled()
AndonRejected()
Functions are called asynchronously, as in the example above, in the last onethen
Functionresolve
A synchronousPromise
, it is also synchronized. This mode is avoidedZalgo
(seeChapter2-Node.js Essential Patterns
), making our asynchronous code more consistent and robust.
If inonFulfilled()
OronRejected()
Exception thrown in handler (usingthrow
Statement), thenthen()
The returned by the methodPromise
Will be automaticallyreject
, throw an exception asreject
Why? This is relative toCPS
Is a huge advantage, because it means there isPromise
, the exception will propagate automatically throughout the chain, andthrow
Statement can finally be used.
In the past, many different libraries were implementedPromise
, most of the time they are incompatible, which means that it is impossible to use differentPromise
Librarythenable
Chain propagation error.
JavaScript
The community worked very hard to resolve this limitation, and these efforts led toPromises / A +
Establishment of specifications. This specification describes in detailthen
The behavior of the method provides a mutually compatible basis, which makes thePromise
Objects can be compatible with each other, out of the box.
RelevantPromises / A +
For detailed specification, please refer toPromises/A+official website.
Implementation of Promise/A+
InJavaScript
China andNode.js
There are several implementationsPromises / A +
Standard library. The following are the most popular:
The real difference between them is inPromises / A +
Additional functions provided above the standard. As we said above, the standard definesthen()
Methods andPromise
The behavior of the parsing process, but it does not specify other functions, such as how to create from asynchronous callback-based functionsPromise
.
In our example, we will use theES2015
ThePromise
BecausePromise
Object fromNode.js 4
It can be used later without any library.
For reference, the following areES2015
ThePromise
Apis provided:
constructor
(new Promise(function(resolve, reject){})
): creates a newPromise
Which is determined based on parameters that pass two types as functionsresolve
Orreject
. The parameters of the constructor are explained as follows:
-
resolve(obj)
:resolve
OnePromise
With a parameterobj
Ifobj
Is a value that is the result of the successful asynchronous operation passed. Ifobj
Is aPromise
Or onethenable
, it will be handled correctly. -
reject(err)
:reject
OnePromise
With a parametererr
. It isError
An instance of the.
Static Method of Promise Object
-
Promise.resolve(obj)
: A will be createdresolve
ThePromise
Example -
Promise.reject(err)
: A will be createdreject
ThePromise
Example -
Promise.all(iterable)
: returns a newPromise
Instance, and in theiterable
Central research institute
YesPromise
Status isreject
When, the returnedPromise
The state of the instance is set toreject
Ifiterable
At least one of themPromise
Status isreject
When, the returnedPromise
The instance state is also set toreject
, andreject
The reason is the first to bereject
ThePromise
Object’sreject
The reason.
-
Promise.race(iterable)
: returns onePromise
Example, wheniterable
Any one of themPromise
Beresolve
Or byreject
When, the returnedPromise
For the same reasonresolve
Orreject
.
Promise instance method
-
Promise.then(onFulfilled, onRejected)
: This isPromise
The basic method of. Its behavior is similar to what we described earlier.Promises / A +
Standards compatible. -
Promise.catch(onRejected)
This is justPromise.then(undefined,onRejected)
The grammar of sugar.
It is worth mentioning that some Promise implementations provide another mechanism to create a new Promise, called deferreds S. We will not describe it here because it is not part of the ES2015 standard, but if you want to know more, you can read the Q document (https://github.com/kriskowal/ …Or When.js documents (https://github.com/cujojs/whe ….
Promisifying a Node.js callback style function
InJavaScript
In, not all asynchronous functions and libraries support out of the boxPromise
. In most cases, we must convert a typical callback-based function into a returnPromise
This process is also calledpromisification
.
Fortunately,Node.js
The callback convention used in allows us to create a reusable function by using thePromise
Object to simplify anyNode.js
StylishAPI
. Let’s create one calledpromisify()
And include it in theutilities.js
In the module (so that later in ourWeb crawler application
Use it in):
module.exports.promisify = function(callbackBasedApi) {
return function promisified() {
const args = [].slice.call(arguments);
return new Promise((resolve, reject) => {
args.push((err, result) => {
if (err) {
return reject(err);
}
if (arguments.length <= 2) {
resolve(result);
} else {
resolve([].slice.call(arguments, 1));
}
});
callbackBasedApi.apply(null, args);
});
}
};
The previous function returned another function namedpromisified()
A function of that represents the given in the inputcallbackBasedApi
Thepromisified
Version. Here’s how it works:
-
promisified()
Function usagePromise
The constructor creates a newPromise
Object and immediately returns it to the caller. - After passing to
Promise
In the function of the constructor, we ensure that the is passed to thecallbackBasedApi
This is a special callback function. Since we know that callbacks are always the last to be called, we just need to attach the callback function to thepromisified()
Function (args
)。 - In a special callback, if we receive an error, we immediately
reject
This ..Promise
. - If no error is received, we use a value or an array value to
resolve
This ..Promise
Depending on the number of results passed to the callback. - Finally, we just need to use the parameter list we built to call
callbackBasedApi
.
Most Promise already provides an out-of-the-box interface to convert a Node.js-style API into an API that returns Promise. For example, q has Q.denodeify () and Q.nbind (), Bluebird has Promise.promisify (), and When.js has node.lift ().
Sequential execution
After some necessary theories, we are now ready to transfer ourWeb crawler application
Convert to UsePromise
The form of. Let’s start with version 2 and download the link to a Web page directly.
Inspider.js
In the module, the first step is to load ourPromise
Implement (we’ll use it later) andPromisifying
The callback-based functions we intend to use:
const utilities = require('./utilities');
const request = utilities.promisify(require('request'));
const mkdirp = utilities.promisify(require('mkdirp'));
const fs = require('fs');
const readFile = utilities.promisify(fs.readFile);
const writeFile = utilities.promisify(fs.writeFile);
Now, we begin to change ourdownload
Functions:
function download(url, filename) {
console.log(`Downloading ${url}`);
let body;
return request(url)
.then(response => {
body = response.body;
return mkdirp(path.dirname(filename));
})
.then(() => writeFile(filename, body))
.then(() => {
console.log(`Downloaded and saved: ${url}`);
return body;
});
}
The most important thing to note here is that we alsoreadFile()
ReturnedPromise
registered
OneonRejected()
Function, which is used to handle the situation that a web page has not been downloaded (or the file does not exist). Also, see how we use it.throw
To deliveronRejected()
Error in function.
Now that we have changed ourspider()
Function, we modify the way it is called as follows:
spider(process.argv[2], 1)
.then(() => console.log('Download complete'))
.catch(err => console.log(err));
Notice how we used it for the first time.Promise
Grammatical sugarcatch
To deal withspider()
Any error condition of the function. If we look at all the code we have written so far, we will be pleasantly surprised to find that we do not contain any error propagation logic because we are forced to do such things when using callback functions. This is obviously a huge advantage because it greatly reduces the template files in our code and the chance of losing any asynchronous errors.
Now, complete the only thing we are missingWeb crawler application
The second edition ofspiderLinks()
Function, we will implement it later.
Sequential iteration
So far,Web crawler application
The code base is mainly rightPromise
An overview of what it is and how it is used shows how it is used.Promise
Realize the simplicity and elegance of the sequential execution process. However, the code we are considering now only involves the execution of a set of known asynchronous operations. Therefore, the missing part of completing our exploration of the sequential execution process is to see how we use it.Promise
To achieve iteration. Similarly, the second edition of the web spiderspiderLinks()
Function is also a good example.
Let’s add the missing piece:
function spiderLinks(currentUrl, body, nesting) {
let promise = Promise.resolve();
if (nesting === 0) {
return promise;
}
const links = utilities.getPageLinks(currentUrl, body);
links.forEach(link => {
promise = promise.then(() => spider(link, nesting - 1));
});
return promise;
}
In order to asynchronously iterate through all links of a web page, we must dynamically create onePromise
The iterative chain of.
- First, we define an empty
Promise
,resolve
Forundefined
. This ..Promise
Just to doPromise
The starting point of the iteration chain of. - Then, we call the previous one in the chain in a loop
Promise
Thethen()
Method to obtain the newPromise
To updatePromise
Variable. This is what we usePromise
The asynchronous iterative mode of.
In this way, the end of the cycle,promise
The variable will contain the last in the loopthen()
ReturnedPromise
Object, so it only works whenPromise
All in the iterative chain ofPromise
Object isresolve
Before they can beresolve
.
Note: The then method was called at the end to resolve the Promise object
Through this, we have usedPromise
Object rewrites ourWeb crawler application
. We should be able to run it now.
Sequential iteration mode
In order to summarize the parts executed in this order, let’s extract a pattern to traverse a group in turn.Promise
:
let tasks = [ /* ... */ ]
let promise = Promise.resolve();
tasks.forEach(task => {
promise = promise.then(() => {
return task();
});
});
promise.then(() => {
// 所有任务都完成
});
Usereduce()
Method to replaceforEach()
Method, allowing us to write more concise code:
let tasks = [ /* ... */ ]
let promise = tasks.reduce((prev, task) => {
return prev.then(() => {
return task();
});
}, Promise.resolve());
promise.then(() => {
//All tasks completed
});
As usual, through simple adjustment of this mode, we can collect the results of all tasks into an array, and we can implement onemapping
Algorithms, or building afilter
Wait.
The above mode uses loops to dynamically establish a chain of Promise.
concurrent execution
The other is suitable for use.Promise
The execution flow of is a parallel execution flow. In fact, what we need to do is to use the built-inPromise.all()
. This method creates anotherPromise
Object, only all in the inputPromise
Allresolve
Time and talentresolve
. This is a parallel execution because in its parametersPromise
There is no order of execution between objects.
To demonstrate this, let’s look at ourWeb crawler application
The third edition of the, which downloads all links in the page in parallel. Let’s use it againPromise
UpdatespiderLinks()
Function to implement parallel processes:
function spiderLinks(currentUrl, body, nesting) {
if (nesting === 0) {
return Promise.resolve();
}
const links = utilities.getPageLinks(currentUrl, body);
const promises = links.map(link => spider(link, nesting - 1));
return Promise.all(promises);
}
The pattern here iselements.map()
In the iteration, an array is generated to store all asynchronous tasks, which is convenient to start at the same time.spider()
Task. This time, in the loop, we don’t wait for the previous download to complete, and then start a new download task: all the download tasks start one by one in the loop. After that, we usedPromise.all()
Method, which returns a newPromise
Object when all thePromise
Objects areresolve
When, thisPromise
The object will beresolve
. In other words, all the downloading tasks are completed, which is exactly what we want.
Restrict parallel execution
Unfortunately,ES2015
ThePromise API
It does not provide a native way to limit the number of concurrent tasks, but we can always rely on what we have learned about using commonJavaScript
To limit concurrency. In fact, we areTaskQueue
The pattern implemented in the class can be easily adjusted to support the task of returning the promise. This can be easily modifiednext()
Method to complete:
class TaskQueue {
constructor(concurrency) {
this.concurrency = concurrency;
this.running = 0;
this.queue = [];
}
pushTask(task) {
this.queue.push(task);
this.next();
}
next() {
while (this.running < this.concurrency && this.queue.length) {
const task = this.queue.shift();
task().then(() => {
this.running--;
this.next();
});
this.running++;
}
}
}
Instead of using a callback function to handle tasks, we simply callPromise
Thethen()
.
Let’s go backspider.js
Module and modify it to support our new version ofTaskQueue
Class. First, we ensure that we define aTaskQueue
New instance of:
const TaskQueue = require('./taskQueue');
const downloadQueue = new TaskQueue(2);
Then, it’s oursspiderLinks()
Function. The amendment here is also very simple:
function spiderLinks(currentUrl, body, nesting) {
if (nesting === 0) {
return Promise.resolve();
}
const links = utilities.getPageLinks(currentUrl, body);
// 我们需要如下代码,用于创建Promise对象
// 如果没有下列代码,当任务数量为0时,将永远不会resolve
if (links.length === 0) {
return Promise.resolve();
}
return new Promise((resolve, reject) => {
let completed = 0;
let errored = false;
links.forEach(link => {
let task = () => {
return spider(link, nesting - 1)
.then(() => {
if (++completed === links.length) {
resolve();
}
})
.catch(() => {
if (!errored) {
errored = true;
reject();
}
});
};
downloadQueue.pushTask(task);
});
});
}
There are several points worth our attention in the above code:
- First of all, we need to return to use
Promise
The new created by the constructorPromise
Object. As we will see, this enables us to manually complete all tasks in the queueresolve
OursPromise
Object. - Then, we should see how we define the task. What we have done is to put a
onFulfilled()
The call to the callback function is added to thespider()
ReturnedPromise
Object, so we can calculate the number of download tasks completed. When the number of downloads completed is the same as the number of links in the current page, we know that the task has been processed, so we can call outsidePromise
Theresolve()
Function.
The Promises/A+specification stipulates that the onFulfilled () and onRejected () callback functions of the then () method can only be called once (only onFulfilled () and onRejected ()) . The implementation of Promise interface ensures that Promise can be resolved or rejected only once even if we manually call resolve or reject many times.
Now, usePromise
TheWeb crawler application
The 4th edition of should be ready. We may once again notice how the download tasks run in parallel, and the number of concurrent tasks is limited to 2.
Expose callback function and Promise in public API
As we learned earlier,Promise
Can be used as a good substitute for callback functions. They make our code more readable and easier to understand. AlthoughPromise
It brings many advantages, but it also requires developers to understand many concepts that are not easy to understand so as to use them correctly and skillfully. For this and other reasons, in some cases, compared withPromise
In general, many developers prefer callback functions.
Now let’s imagine that we want to build a common library that performs asynchronous operations. What do we need to do? We have created a callback functionAPI
Or is it an orientationPromise
TheAPI
? Or both?
This is a problem faced by many well-known libraries, and there are at least two methods worth mentioning to enable us to provide a multifunctionalAPI
.
Likerequest
,redis
Andmysql
The first method used by such a library is to provide a simple callback function-basedAPI
If necessary, developers can choose to expose functions. Some of these libraries provide tool functions toPromise
Asynchronous callback, but developers still need to expose theAPI
To be able to usePromise
Object.
The second method is more transparent. It also provides a callback-orientedAPI
However, it makes callback parameters optional. Every time a callback is passed as a parameter, the function will run normally, executing the callback on completion or failure. When the callback is not passed, the function returns an immediatelyPromise
Object. This method effectively combines callback functions andPromise
So that developers can choose which interface to use when calling, without having to do it in advance.Promise
Hua. Many libraries, such asmongoose
Andsequelize
, all support this method.
Let’s look at a simple example. Suppose we want to implement a module that performs division asynchronously:
module.exports = function asyncDivision(dividend, divisor, cb) {
return new Promise((resolve, reject) => { // [1]
process.nextTick(() => {
const result = dividend / divisor;
if (isNaN(result) || !Number.isFinite(result)) {
const error = new Error('Invalid operands');
if (cb) {
cb(error); // [2]
}
return reject(error);
}
if (cb) {
cb(null, result); // [3]
}
resolve(result);
});
});
};
The code of this module is very simple, but there are some details worth emphasizing:
- First, return to use
Promise
The new promise created by the constructor of. We define all logic within the constructor parameter function. - In the event of an error, we
reject
This ..Promise
However, if the callback function is passed as a parameter when called, we also execute the callback to propagate the error. - After the calculation results, we
resolve
Take thisPromise
However, if there is a callback function, we will also propagate the result to the callback function.
Let’s now look at how to use callback functions andPromise
To use this module:
// 回调函数的方式
asyncDivision(10, 2, (error, result) => {
if (error) {
return console.error(error);
}
console.log(result);
});
// Promise化的调用方式
asyncDivision(22, 11)
.then(result => console.log(result))
.catch(error => console.error(error));
It should be clear that developers who are about to start using new modules similar to the ones described above will easily choose the style that best suits their needs without having to use it in the hope of doing so.Promise
When introducing externalpromisification
Functions.
Generators
ES2015
The specification introduces another mechanism that can be used to simplify, among other new functionsNode.js
Asynchronous control flow of application program. We are talking aboutGenerator
, also known assemi-coroutines
. They are generalizations of subprograms and can have different entry points. In a normal function, in fact we can only have one entry point, which corresponds to the call of the function itself.Generator
Similar to general functions, but can be paused (usingyield
Statement) and then continue execution at a later time. When implementing iterators,Generator
This is especially useful because we have discussed how to use iterators to implement important asynchronous control flow patterns, such as sequential execution and limiting parallel execution.
Generators foundation
Before we explore the use ofGenerator
Before realizing asynchronous control flow, it is very important to learn some basic concepts. Let’s start with grammar. This can be done by appending after the function key*
(asterisk) operator to declareGenerator
Functions:
function* makeGenerator() {
// body
}
InmakeGenerator()
Function, we can use keywordsyield
Pause execution and return the value passed to it by the caller:
function* makeGenerator() {
yield 'Hello World';
console.log('Re-entered');
}
In the previous code,Generator
viayield
A stringHello World
Pauses execution of the current function. WhenGenerator
On recovery, execution starts with the following statement:
console.log('Re-entered');
makeGenerator()
The function is essentially a factory that returns a new when calledGenerator
Object:
const gen = makeGenerator();
The most important way to generate objects is tonext()
Which is used for startup/recoveryGenerator
, and returns an object in the following form:
{
value: <yielded value>
done: <true if the execution reached the end>
}
This object containsGenerator
yield
The value of the and an indicationGenerator
Symbol for whether execution has been completed.
A simple example
For demonstration purposesGenerator
, let’s create a file calledfruitGenerator.js
New module for:
function* fruitGenerator() {
yield 'apple';
yield 'orange';
return 'watermelon';
}
const newFruitGenerator = fruitGenerator();
console.log(newFruitGenerator.next()); // [1]
console.log(newFruitGenerator.next()); // [2]
console.log(newFruitGenerator.next()); // [3]
The preceding code prints the following output:
{ value: 'apple', done: false }
{ value: 'orange', done: false }
{ value: 'watermelon', done: true }
We can explain the above phenomenon as follows:
- First call
newFruitGenerator.next()
At that time,Generator
The function starts executing until the first is reachedyield
Statement, the command pausesGenerator
The function executes and returns the valueapple
Return to caller. - On the second call
newFruitGenerator.next()
At that time,Generator
Function resumes execution from the secondyield
Statement, which in turn suspends execution, and at the same timeorange
Return to caller. -
newFruitGenerator.next()
The last call to resulted inGenerator
Function is executed from its lastyield
Recovery, a return statement that terminatesGenerator
Function, returnwatermelon
In the result objectdone
The property is set totrue
.
Generators as Iterators
In order to better understand whyGenerator
Functions are very useful for implementing iterators. Let’s build an example. Before we calliteratorGenerator.js
In the new module of, we write the following code:
function* iteratorGenerator(arr) {
for (let i = 0; i < arr.length; i++) {
yield arr[i];
}
}
const iterator = iteratorGenerator(['apple', 'orange', 'watermelon']);
let currentItem = iterator.next();
while (!currentItem.done) {
console.log(currentItem.value);
currentItem = iterator.next();
}
This code should print the elements in the array as follows:
apple
orange
watermelon
In this example, every time we calliterator.next()
We will all recoverGenerator
Functionalfor
Cycle throughyield
The next item in the array runs another loop. This demonstrates how to maintain during function calls.Generator
The state of. When execution continues, the values of the loop and all variables are the same asGenerator
Function execution is suspended in exactly the same state.
Pass values to Generators
Now we continue to studyGenerator
First, learn how to pass the value back toGenerator
Function. This is actually very simple, all we need to do is tonext()
The method provides a parameter and the value is used asGenerator
Within functionyield
The return value of the statement is provided.
To show this, let’s create a new simple module:
function* twoWayGenerator() {
const what = yield null;
console.log('Hello ' + what);
}
const twoWay = twoWayGenerator();
twoWay.next();
twoWay.next('world');
When executed, the previous code will be outputHello world
. We make the following explanation:
- First call
next()
In the method,Generator
The function reaches the firstyield
Statement, and then pause. - When
next('world')
When called,Generator
Function from the last stop position, that is, the last timeyield
Statement point recovery, but this time we have a value passed toGenerator
Function. This value will be assigned towhat
Variable. The generator then executesconsole.log()
Command and terminate.
In a similar way, we can forceGenerator
Function throws an exception. This can be achieved by usingGenerator
Functionalthrow
Method, as shown in the following example:
const twoWay = twoWayGenerator();
twoWay.next();
twoWay.throw(new Error());
In this last piece of code,twoWayGenerator()
The function will set theyield
The function throws an exception when it returns. It’s like starting fromGenerator
Function throws an exception inside, which means that it can be used liketry ... catch
Block to capture and handle exceptions.
Generator Implements Asynchronous Control Flow
You must want to knowGenerator
Function to help us handle asynchronous operations. We can do this by creating an acceptanceGenerator
Function as a special function of parameters to demonstrate this, and allows us to inGenerator
The function uses asynchronous code internally. This function should pay attention to recovery when asynchronous operation is completed.Generator
Function execution. We will call this functionasyncFlow()
:
function asyncFlow(generatorFunction) {
function callback(err) {
if (err) {
return generator.throw(err);
}
const results = [].slice.call(arguments, 1);
generator.next(results.length > 1 ? results : results[0]);
}
const generator = generatorFunction(callback);
generator.next();
}
Take one of the previous functionsGenerator
Function as input and immediately call:
const generator = generatorFunction(callback);
generator.next();
generatorFunction()
Accept a special callback function as a parameter whengenerator.throw()
If an error is received, it will be returned immediately. In addition, by comparing the values received in the callback functionresults
Return valueGenerator
Function continuationGenerator
Function execution:
if (err) {
return generator.throw(err);
}
const results = [].slice.call(arguments, 1);
generator.next(results.length > 1 ? results : results[0]);
In order to illustrate the power of this simple auxiliary function, we created a program calledclone.js
This module only creates its own clone. Paste what we just createdasyncFlow()
Function, the core code is as follows:
const fs = require('fs');
const path = require('path');
asyncFlow(function*(callback) {
const fileName = path.basename(__filename);
const myself = yield fs.readFile(fileName, 'utf8', callback);
yield fs.writeFile(`clone_of_${filename}`, myself, callback);
console.log('Clone created');
});
Obviously, there isasyncFlow()
With the help of the function, we can write asynchronous code in the same way as we write synchronous blocking functions. And the principle behind this result is very clear. Once the asynchronous operation ends, the callback function passed to each asynchronous function will continueGenerator
Function execution. There is nothing complicated, but the result is really surprising.
There are two other changes in this technology, one isPromise
The use of, the other isthunks
.
Thunk used in the generator-based control flow is only a simple function, which partially applies all parameters of the original function except callback. The return value is another function that only accepts callbacks as arguments. For example, the thunkified version of fs.readFile () is as follows:
function readFileThunk(filename, options) {
return function(callback) {
fs.readFile(filename, options, callback);
}
}
thunk
AndPromise
All allow us to create a without callbackGenerator
Function is passed as a parameter, for example, using thethunk
TheasyncFlow()
The version is as follows:
function asyncFlowWithThunks(generatorFunction) {
function callback(err) {
if (err) {
return generator.throw(err);
}
const results = [].slice.call(arguments, 1);
const thunk = generator.next(results.length > 1 ? results : results[0]).value;
thunk && thunk(callback);
}
const generator = generatorFunction();
const thunk = generator.next().value;
thunk && thunk(callback);
}
This technique is readinggenerator.next()
The return value of containsthunk
. The next step is to inject special callback function callsthunk
In itself. This allows us to write the following code:
asyncFlowWithThunk(function*() {
const fileName = path.basename(__filename);
const myself = yield readFileThunk(__filename, 'utf8');
yield writeFileThunk(`clone_of_${fileName}`, myself);
console.log("Clone created")
});
Gernator-based control flow using co
As you should have guessed,Node.js
The ecosystem will useGenerator
Function to provide some solutions for handling asynchronous control flow, for example,suspendIs one of the oldest supportPromise
、thunks
AndNode.js
A library of style callback functions and normal style callback functions. Also, most of what we analyzed earlierPromise
Libraries all provide tool functions that enableGenerator
AndPromise
Can be used together.
We choosecoAs an example of this chapter. It supports many types ofyieldables
, some of which are:
Thunks
Promises
-
Arrays
(parallel execution) -
Objects
(parallel execution) -
Generators
(Entrusted) -
Generator
Function (delegate)
There are also many frameworks or libraries based onco
Ecosystem, including the following:
-
Web framework
, the most popular iskoa - Libraries that implement specific control flow patterns
- Packaging is popular
API
Compatibleco
The library of
We useco
To realize ourGenerator
Version ofWeb crawler application
.
In order toNode.js
Style functions are converted tothunks
, we will use a method calledthunkifyThe library.
Sequential execution
Let’s revise itWeb crawler application
At the beginning of version 2 of theGenerator
Function sumco
The actual exploration of. The first thing we need to do is to load our dependency package and generate the functions we want to use.thunkified
Version. These will be inspider.js
The beginning of the module:
const thunkify = require('thunkify');
const co = require('co');
const request = thunkify(require('request'));
const fs = require('fs');
const mkdirp = thunkify(require('mkdirp'));
const readFile = thunkify(fs.readFile);
const writeFile = thunkify(fs.writeFile);
const nextTick = thunkify(process.nextTick);
Looking at the above code, we can notice that it is similar to the previous section of this chapterpromisify
ChemicalAPI
Some similarities in the code of. At this point, it is interesting that if we use ourpromisified
Version of the function to replacethunkified
The code will remain exactly the same, thanks toco
Supportthunk
AndPromise
Object asyieldable
Object. In fact, we can even use it in the same application if we want to.thunk
AndPromise
Even in the sameGenerator
In the function. In terms of flexibility, this is a huge advantage, because it enables us to use information based onGenerator
Function to solve problems in our applications.
All right, now let’s startdownload()
Function to aGenerator
Functions:
function* download(url, filename) {
console.log(`Downloading ${url}`);
const response = yield request(url);
const body = response[1];
yield mkdirp(path.dirname(filename));
yield writeFile(filename, body);
console.log(`Downloaded and saved ${url}`);
return body;
}
By usingGenerator
Andco
Ourdownload()
Functions become much simpler. When we need to do asynchronous operations, we use asynchronousGenerator
Function asthunk
To translate the previous content intoGenerator
Function and use theyield
Clause.
Then we began to realize ourspider()
Functions:
function* spider(url, nesting) {
cost filename = utilities.urlToFilename(url);
let body;
try {
body = yield readFile(filename, 'utf8');
} catch (err) {
if (err.code !== 'ENOENT') {
throw err;
}
body = yield download(url, filename);
}
yield spiderLinks(url, body, nesting);
}
An interesting detail from the above code is that we can usetry...catch
Statement block to handle exceptions. We can also usethrow
To spread the anomaly. Another detail is that weyield
Oursdownload()
Function, and this function is neither athunk
, is not a.promisified
Function, just another oneGenerator
Function. This is no problem, becauseco
We also support others.Generators
As ayieldables
.
Final conversionspiderLinks()
In this function, we recursively download a link to a web page. Use in this functionGenerators
, appear much simpler:
function* spiderLinks(currentUrl, body, nesting) {
if (nesting === 0) {
return nextTick();
}
const links = utilities.getPageLinks(currentUrl, body);
for (let i = 0; i < links.length; i++) {
yield spider(links[i], nesting - 1);
}
}
Look at the above code. Although there is no pattern to show for sequential iteration.Generator
Andco
We have done a lot of things to help us write asynchronous code in a synchronous way.
Look at the most important part, the entrance to the program:
co(function*() {
try {
yield spider(process.argv[2], 1);
console.log(`Download complete`);
} catch (err) {
console.log(err);
}
});
This is the only place to callco(...)
To package oneGenerator
. In fact, once we do this,co
It automatically encapsulates what we pass toyield
Any of theGenerator
Function, and this process is recursive, so the rest of the program is related to whether we useco
Is completely irrelevant, although isco
Encapsulated inside.
It should be operational by now.Generator
Function rewrittenWeb crawler application
Here we go.
concurrent execution
Unfortunately, althoughGenerator
It is convenient to execute in sequence, but it cannot be directly used to execute a group of tasks in parallel, at least not onlyyield
AndGenerator
. Previously, the pattern we used in this case was simply based on a callback orPromise
Function of, but uses theGenerator
Function, everything will look simpler.
Fortunately, if parallel execution of concurrency is not limited,co
It can already be passedyield
OnePromise
Objects,thunk
、Generator
Functions, even includingGenerator
Function.
With this in mind, ourWeb crawler application
The third edition can be rewrittenspiderLinks()
Function to make the following changes:
function* spiderLinks(currentUrl, body, nesting) {
if (nesting === 0) {
return nextTick();
}
const links = utilities.getPageLinks(currentUrl, body);
const tasks = links.map(link => spider(link, nesting - 1));
yield tasks;
}
But all the above functions do is get all the tasks, which are essentially passedGenerator
Function to implement asynchronous, ifco
Thethunk
An inner pair containsGenerator
The array of functions uses theyield
, these tasks will be executed in parallel. Outer layerGenerator
The function waits untilyield
Clause is executed in parallel before continuing.
Next we will look at how to use a callback function-based approach to solve the same parallel flow. We rewrite it this way.spiderLinks()
Functions:
function spiderLinks(currentUrl, body, nesting) {
if (nesting === 0) {
return nextTick();
}
// 返回一个thunk
return callback => {
let completed = 0,
hasErrors = false;
const links = utilities.getPageLinks(currentUrl, body);
if (links.length === 0) {
return process.nextTick(callback);
}
function done(err, result) {
if (err && !hasErrors) {
hasErrors = true;
return callback(err);
}
if (++completed === links.length && !hasErrors) {
callback();
}
}
for (let i = 0; i < links.length; i++) {
co(spider(links[i], nesting - 1)).then(done);
}
}
}
We useco
parallel runningspider()
Function, callingGenerator
The function returned aPromise
Object. In this way, waitPromise
Called after completiondone()
Function. Usually, based onGenerator
All libraries that control flow have this function, so you can always add aGenerator
Convert to a callback-based or callback-basedPromise
The function of.
To open multiple download tasks in parallel, we only need to reuse the callback-based parallel execution mode defined earlier. We should also note that we willspiderLinks()
Convert to athunk
(not one anymoreGenerator
Function). This allows us to have a callback function to call when all parallel tasks are completed.
What is mentioned above is to convert a Generator function into a thunk mode, so that it can support other callback-based or Promise-based control flow algorithms, and can write asynchronous code through synchronous blocking code style.
Restrict parallel execution
Now that we know how to handle asynchronous execution processes, it should be easy to plan ourWeb crawler application
The implementation of the fourth version of the, which imposes a limit on the number of concurrent download tasks. We have several options to do this. Some of these programmes are as follows:
- Using the callback-based previously implemented
TaskQueue
Class. All we need isthunkify
OursGenerator
Function and the callback function it provides. - Use based on
Promise
TheTaskQueue
Class and ensure that eachGenerator
Functions are converted to a returnPromise
The function of the object. - Use
async
,thunkify
The tool functions we intend to use, in addition, we need to use theGenerator
Functions are converted into callback-based modes so that they can be better used by this library. - Use based on
co
The library in the ecosystem of, especially for this kind of scene library, such asco-limiter. - Implement a custom algorithm based on producer-consumer model, which is similar to
co-limiter
The internal implementation principle of is the same.
In order to learn, we chose the last scheme, even helping us to better understand a pattern that is often synchronized with synergetics (also with threads and processes).
Producer-consumer model
Our goal is to use queues to provide a fixed number ofworkers
As many concurrency levels as we want to set. In order to implement this algorithm, we will base on the definition earlier in this chapterTaskQueue
Class rewrite:
class TaskQueue {
constructor(concurrency) {
this.concurrency = concurrency;
this.running = 0;
this.taskQueue = [];
this.consumerQueue = [];
this.spawnWorkers(concurrency);
}
pushTask(task) {
if (this.consumerQueue.length !== 0) {
this.consumerQueue.shift()(null, task);
} else {
this.taskQueue.push(task);
}
}
spawnWorkers(concurrency) {
const self = this;
for (let i = 0; i < concurrency; i++) {
co(function*() {
while (true) {
const task = yield self.nextTask();
yield task;
}
});
}
}
nextTask() {
return callback => {
if (this.taskQueue.length !== 0) {
return callback(null, this.taskQueue.shift());
}
this.consumerQueue.push(callback);
}
}
}
Let’s analyze thisTaskQueue
The new implementation of the class. The first is in the constructor. Need to call oncethis.spawnWorkers()
Because this is startupworker
The method of.
Oursworker
It’s very simple, they just useco()
Immediate execution of packagingGenerator
Function, so eachGenerator
Functions can be executed in parallel. Inside, eachworker
Is running in a dead loop (while(true){}
) has been blocked (yield
) until a new task is available in the queue (yield self.nextTask()
Once a new task can be performed,yield
This asynchronous task will not be completed until it is completed. You may want to know how we can limit parallel execution and leave the next task waiting in the queue. The answer isnextTask()
In the method. Let’s take a closer look at the principle of this method:
nextTask() {
return callback => {
if (this.taskQueue.length !== 0) {
return callback(null, this.taskQueue.shift());
}
this.consumerQueue.push(callback);
}
}
Let’s look at what happens inside this function. This is the core of this pattern:
- This method returns a value for the
co
Is a legal oneyieldable
Thethunk
. - As long as
taskQueue
There is the next task in the instance generated by the class.thunk
The callback function of will be called immediately. When a callback function is called, it immediately unlocks oneworker
The blocking state of,yield
This task. - If there are no more tasks in the queue, the callback function itself will be placed
consumerQueue
China. Through this approach, we will have aworker
Leave idle (idle
) mode. Once we have a new task to deal with, inconsumerQueue
The callback function in the queue will be called and wake us up immediately.worker
Perform asynchronous processing.
Now, in order to understandconsumerQueue
Idle in queueworker
How to resume work, we need to analyzepushTask()
Methods. If a callback function is currently available,pushTask()
The method calls theconsumerQueue
The first callback function in the queue, which cancels theworker
The lock of the. If no callback function is available, this means that allworker
They are all in working status, only need to add a new task totaskQueue
Task queue.
InTaskQueue
Class,worker
Acting as a consumer and callingpushTask()
The role of a function can be considered as a producer. This model shows us oneGenerator
Functions can actually be similar to a thread or process. In fact, the problem between producers and consumers is the most common problem when studying inter-process communication and synchronization, but as we have already mentioned, it is also a common example for processes and threads.
Limit the concurrency of download tasks
Now that we have used itGenerator
Functions and producer-consumer models implement a restricted parallel algorithm and have been implemented inWeb crawler application
Version 4 applies it to limit the number of concurrent downloads in. First, we load and initialize aTaskQueue
Object:
const TaskQueue = require('./taskQueue');
const downloadQueue = new TaskQueue(2);
Then, modifyspiderLinks()
Function. It is similar to the previous version that does not restrict concurrency, so we only show the modified part here, mainly by calling the new version.TaskQueue
The of the instance generated by the classpushTask()
Method to restrict parallel execution:
function spiderLinks(currentUrl, body, nesting) {
//...
return (callback) => {
//...
function done(err, result) {
//...
}
links.forEach(function(link) {
downloadQueue.pushTask(function*() {
yield spider(link, nesting - 1);
done();
});
});
}
}
In each task, we call immediately after the download is completeddone()
Function, so we can calculate how many links have been downloaded and notify when the download is complete.thunk
The callback function of the.
Use Asyncaway New Syntax with Babel
Callback function,Promise
AndGenerator
Functions are used for processingJavaScript
AndNode.js
How to solve asynchronous problems. As we can see,Generator
The real meaning of is that it provides a way to pause the execution of a function and then wait for the previous task to complete before continuing. We can use this feature to write asynchronous code and let developers write asynchronous code in a synchronous blocking code style. The execution of the current function will not resume until the result of the asynchronous operation is returned.
But ..Generator
Functions are mostly used to handle iterators, however, iterators are somewhat cumbersome to use in asynchronous code. The code may be difficult to understand, resulting in poor readability and maintainability of the code.
But there will be a more concise grammar in the near future. In fact, this proposal will soon be introduced intoESMASCript 2017
This specification definesasync
Function syntax.
async
The function specification introduces two keywords (async
Andawait
) to the nativeJavaScript
Language, improve the way we write asynchronous code.
To understand the usage and advantages of this grammar, let’s look at a simple example:
const request = require('request');
function getPageHtml(url) {
return new Promise(function(resolve, reject) {
request(url, function(error, response, body) {
resolve(body);
});
});
}
async function main() {
const html = await getPageHtml('http://google.com');
console.log(html);
}
main();
console.log('Loading...');
In the above code, there are two functions:getPageHtml
Andmain
. The first function is to extract a givenURL
Of a remote web page ofHTML
Document code. It is worth noting that this function returns aPromise
Object.
The point ismain
Function, because it is used hereasync
Andawait
Key words. The first thing to notice is that the function must be based onasync
The keyword is prefixed. This means that this function executes asynchronous code and allows it to be used within the function bodyawait
Key words.await
Keyword ingetPageHtml
Before calling, tellJavaScript
The interpreter waits before proceeding with the next instruction.getPageHtml
ReturnedPromise
The result of the object. In this way,main
Which part of the code inside the function is asynchronous, it will wait for the completion of the asynchronous code before continuing to perform subsequent operations, and will not block the normal execution of the rest of the program. In fact, the console prints stringsLoading ...
, followed by Google’s home pageHTML
Code.
Is this method more readable and easier to understand? Unfortunately, this proposal has not yet been finalized. Even if this proposal is passed, we need to wait for the next version.
TheECMAScript
Standardize and integrate it intoNode.js
Only after that can we use this new grammar. So what did we do today? Just waiting aimlessly? No, of course not! We can already use it in our codeasync await
Grammar, as long as we use itBabel
.
Installing and Running Babel
Babel
Is aJavaScript
Compiler (or translator), can use syntax converter to convert the high version ofJavaScript
Code conversion to otherJavaScript
Code. Syntax converters allow us to write and use, for exampleES2015
,ES2016
,JSX
And other new syntax to translate into later compatible codeJavaScript
Operating environment such as browser orNode.js
Both can be usedBabel
.
Use in projectsnpm
InstallationBabel
, the command is as follows:
npm install --save-dev babel-cli
We also need to install plug-ins to supportasync await
Interpretation or translation of grammar:
npm install --save-dev babel-plugin-syntax-async-functions babel-plugin-tranform-async-to-generator
Now suppose we want to run our previous example (calledindex.js
)。 We need to start with the following command:
node_modules/.bin/babel-node --plugins "syntax-async-functions,transform-async-to-generator" index.js
In this way, we use supportasync await
To dynamically convert source code.Node.js
The actual running is backward compatible code stored in memory.
Babel
It can also be configured as a code building tool to save the translated or interpreted code into the local file system, which is convenient for us to deploy and run the generated code.
About how to install and configure Babel, you can go to the official website.https://babeljs.ioConsult relevant documents.
Comparison of Several Ways
Now, what should we do about itJavaScript
We have a better understanding and summary of the asynchronous problem of. The advantages and disadvantages of several mechanisms are summarized in the following table:
It is worth mentioning that we have chosen to introduce only the most popular or widely used solutions for handling asynchronous control processes in this chapter, but for example, Fibers (https://npmjs.org/package/fibers) and Streamline (https://npmjs.org/pAckage/streamline) is also worth seeing.
Summary
In this chapter, we analyzed some methods to deal with asynchronous control flowPromise
、Generator
Functions and the upcomingasync await
Grammar.
We learned how to use these methods to write more concise and readable asynchronous code. We discussed some of the most important advantages and disadvantages of these methods and realized that even if they were very useful, it would take some time to master them. This is why these methods have not completely replaced callbacks that are still very useful in many cases. As a developer, one should analyze the actual situation and decide which solution to use. If you are building a common library that performs asynchronous operations, you should provide an easy-to-useAPI
Even for developers who only want to use callbacks.
In the next chapter, we will explore another mechanism related to asynchronous code execution, which is also the wholeNode.js
Another basic building block in the ecosystem:streams
.