This series of articles is《Node.js Design Patterns Second Edition》The translation of the original text and the reading notes were serialized and updated in GitHub.Synchronized translation links.

Welcome to pay attention to my column, and subsequent blog posts will be synchronized in the column:

• Encounter’s Digging Column
• Programming Thinking of Zhihu’s Column Encounter
• Segmentfault front-end station

Node.js Essential Patterns

ForNode.jsAsynchrony is its most prominent feature, but for some other languages, such asPHP, asynchronous code is not often processed.

In synchronous programming, we are used to imagining the execution of code as a continuous calculation step from top to bottom. Each operation is blocked, which means that the next operation can be executed only after the execution of one operation is completed. This method is beneficial to our understanding and debugging.

However, in asynchronous programming, we can perform some operations such as reading files or executing network requests in the background. When we call the asynchronous operation method, even if the current or previous operation has not been completed, the following subsequent operations will continue to be executed. The operations executed in the background will be completed at any time, and the application will react in the correct way when the asynchronous call is completed.

Although this non-blocking method has better performance than blocking method, it is really difficult for programmers to understand, and asynchronous sequence may become difficult to operate when processing advanced applications with more complex asynchronous control flow.

Node.jsA series of tools and design patterns are provided so that we can best handle asynchronous code. It is very important to know how to use them to write applications that have performance and are easy to understand and debug.

In this chapter, we will see the two most important asynchronous modes: callbacks and event publishers.

Callback mode

As mentioned in the previous chapter, the callback isReactor modeThehandlerThe callback was originallyNode.jsOne of the unique programming styles. Callback functions are functions that propagate the results of asynchronous operations after they are completed. They are always used to replace the return instructions of synchronous operations. AndJavaScriptIt happens to be the best language for callback. InJavaScriptIn, a function is a first-class citizen. We can pass a function variable as a parameter, call it in another function, and store the result of the call in a certain data structure. Another ideal structure to implement callbacks is closures. Using closures, we can preserve the context in which the function was created, so that whenever a callback is called, the context in which asynchronous operations are requested is maintained.

In this section, we analyze callback-based programming ideas and patterns, rather than patterns of return instructions for synchronous operations.

CPS

InJavaScriptIn, the callback function is passed as a parameter to another function and called when the operation is completed. In functional programming, this method of transferring results is calledCPS. This is a general concept and not only for asynchronous operations. In fact, it just passes the result as a parameter to another function (callback function), and then calls the callback function in the body logic to get the operation result, instead of directly returning it to the caller.

CPS synchronization

For a clearer understandingCPS, let’s look at this simple synchronization function:

function add(a, b) {
  return a + b;
}

The above example has become a direct programming style. In fact, there is nothing special about it. It is the use ofreturnStatement passes the result directly to the caller. It represents the most common way to return results in synchronous programming. Of the above functionsCPSWrite it as follows:

function add(a, b, callback) {
  callback(a + b);
}

add()The function is synchronizedCPSFunction,CPSThe function only gets it when it is calledadd()The following code is the calling method for the execution result of the function:

console.log('before');
add(1, 2, result => console.log('Result: ' + result));
console.log('after');

Sinceadd()Is synchronized, then the above code prints the following results:

before
Result: 3
after

Asynchronous CPS

Then let’s think about the following example, hereadd()The function is asynchronous:

function additionAsync(a, b, callback) {
 setTimeout(() => callback(a + b), 100);
}

In the above code, we usesetTimeout()Simulates the call of an asynchronous callback function. Now, we calladditionalAsyncAnd see the specific output results.

console.log('before');
additionAsync(1, 2, result => console.log('Result: ' + result));
console.log('after');

The above code will have the following output:

before
after
Result: 3

Because ..setTimeout()Is an asynchronous operation, so it will not wait for the callback to be executed, but will return immediately and give control to.addAsync()And then return it to its caller.Node.jsThis attribute in is critical because whenever asynchronous requests are generated, control is given to the event loop, allowing new events from the queue to be processed.

The following picture showsNode.jsIn the event cycle process:

When the asynchronous operation is completed, the execution right will be given to the callback function where the asynchronous operation starts. Execution will start fromEvent cycleAt first, so it will have a new stack. ForJavaScriptThis is its advantage. It is precisely because closures preserve their context that callbacks can be executed normally even at different points in time and at different locations.

The synchronization function is blocked until it completes the operation. The asynchronous function returns immediately, and the result is passed to the handler (a callback in our example) in a later loop of the event loop.

Non-CPS style callback mode

In some cases, we may think that callback CPS is written asynchronously, but it is not. For example, the following code,ArrayObject’smap()Methods:

const result = [1, 5, 7].map(element => element - 1);
console.log(result); // [0, 4, 6]

In the above example, the callback is only used to iterate over the elements of the array, not to pass the result of the operation. In fact, in this example, the callback method is used to return synchronously instead of passing the result. Is it a callback that passes the result of an operationAPIThe document has clear instructions.

Synchronous or asynchronous?

We have seen that the order of code execution will fundamentally change due to synchronous or asynchronous execution. This has a significant impact on the process, correctness and efficiency of the entire application. The following is an analysis of the two modes and their defects. Generally speaking, it is necessary to avoid confusion that is difficult to detect and expand due to inconsistent execution sequence. The following is an asynchronous instance with traps:

A Problem Function

One of the most dangerous situations is that synchronous execution should be asynchronous execution under certain conditions.API. Take the following code as an example:

const fs = require('fs');
const cache = {};

function inconsistentRead(filename, callback) {
  if (cache[filename]) {
    // 如果缓存命中，则同步执行回调
    callback(cache[filename]);
  } else {
    // 未命中，则执行异步非阻塞的I/O操作
    fs.readFile(filename, 'utf8', (err, data) => {
      cache[filename] = data;
      callback(data);
    });
  }
}

The above functions use cache to store the results of different file read operations. However, remember, this is just an example, it lacks error handling, and its cache logic itself is not optimal (e.g. there is no cache retirement policy). In addition, the above function is very dangerous, because if the cache is not set, its behavior is asynchronous untilfs.readFile()Until the function returns the result, it will not execute synchronously, and the cache will not trigger, but will go to the asynchronous callback call.

Liberationzalgo

AboutzalgoIn fact, it refers to the uncertainty of synchronous or asynchronous behavior, which almost always leads to very difficult tracking.bug.

Now, let’s look at how to use a function whose order is unpredictable. It can even easily interrupt an application. Look at the following code:

function createFileReader(filename) {
  const listeners = [];
  inconsistentRead(filename, value => {
    listeners.forEach(listener => listener(value));
  });
  return {
    onDataReady: listener => listeners.push(listener)
  };
}

When the above function is called, it creates a new object that acts as an event publisher, allowing us to set up multiple event listeners for file read operations. When the read operation is completed and the data is available, all listeners will be called immediately. The previous function uses the previously definedinconsistentRead()Function to implement this function. We are now trying to callcreateFileReader()Functions:

const reader1 = createFileReader('data.txt');
reader1.onDataReady(data => {
 console.log('First call data: ' + data);
 // 之后再次通过fs读取同一个文件
 const reader2 = createFileReader('data.txt');
 reader2.onDataReady(data => {
   console.log('Second call data: ' + data);
 });
});

After the output is like this:

First call data: some data

Let’s analyze why the second callback was not called:

In creatingreader1At that time,inconsistentRead()The function is executed asynchronously, and no cached results are available, so we have time to register the event listener. After the read operation is completed, it will be called in the next event cycle.

Then, it is created in the loop of the event loopreader2, where the cache for the requested file already exists. In this case, the internal callinconsistentRead()Will be synchronized. Therefore, its callback will be called immediately, which means thatreader2All listeners of will also be called synchronously. However, in creatingreader2After that, we started registering listeners, so they will never be called.

inconsistentRead()The behavior of the callback function is unpredictable because it depends on many factors, such as the frequency of calls, the file name passed as a parameter, and the time it takes to load the file.

In practical applications, for example, the errors we just saw may be very complex and difficult to identify and copy in real applications. Imagine, inweb serverUsing similar functions in, you can have multiple concurrent requests; Imagine that these requests are pending without any obvious reason and without any logs being recorded. This is definitely annoyingbug.

npmThe founder of the and formerNode.jsProject leaderIsaac Z. SchlueterIn one of his blog articles, he compared the use of this unpredictable function to releaseZalgo. If you are not familiar with itZalgo. You can have a lookOriginal post by Isaac Z. Schlueter.

Use synchronization apis

From the above aboutzalgoIn the example of, we know that,APIIts nature must be clearly defined: synchronous or asynchronous?

We are suitable.fixAboveinconsistentRead()Function generatedbugThe way to do this is to make it block execution completely synchronously. And this is entirely possible becauseNode.jsFor most basicI/OThe operation provides a set ofAPI. For example, we can usefs.readFileSync()Function to replace its asynchronous counterpart. The code is now as follows:

const fs = require('fs');
const cache = {};

function consistentReadSync(filename) {
 if (cache[filename]) {
   return cache[filename];
 } else {
   cache[filename] = fs.readFileSync(filename, 'utf8');
   return cache[filename];
 }
}

We can see that the entire function is transformed into a synchronous blocking call mode. If a function is synchronous, it will not beCPSThe style of. In fact, we can say that usingCPSTo achieve a synchronousAPIIt has always been a best practice, which will eliminate any confusion in its nature and will be more effective from a performance point of view.

Please remember, willAPIFromCPSChange to the style returned by direct call, or from asynchronous to synchronous. For example, in our example, we must completely change ourcreateFileReader()To synchronize and adapt it to always work.

In addition, synchronization is usedAPIRather than asynchronousAPI, pay special attention to the following matters needing attention:

• SynchronizationAPINot applicable to all application scenarios.
• SynchronizationAPILoop blocking events and place concurrent requests in blocking state. It will destroyJavaScriptEven the performance of the entire application is degraded due to the concurrency model of. We will see the impact on our applications later in this book.

In ourinconsistentRead()Function, because each file name is called only once, synchronous blocking calls will have little impact on the application, and cache values will be used for all subsequent calls. If the number of our static files is limited, then useconsistentReadSync()It will not have a great impact on our event cycle. If we have a large number of files that need to be read once and have a high performance requirement, we do not recommend using theNode.jsUse synchronization inI/O. However, in some cases, synchronizationI/OMay be the simplest and most effective solution. Therefore, we must correctly evaluate the specific application scenario to select the most appropriate scheme. The above example actually shows that synchronous blocking is used in actual applicationsAPILoading configuration files is very meaningful.

Therefore, remember to consider using synchronous blocking only if it does not affect the concurrency of the application.I/O.

delayed time processing

Another kindfixAboveinconsistentRead()Function generatedbugThe way to do this is to make it only asynchronous. The solution here is to call synchronously in the next event cycle, instead of running immediately in the same event cycle, so that it is actually asynchronous. InNode.js, you can use theprocess.nextTick()Which delays the execution of the function until the next pass event loop. Its function is very simple, it takes a callback as a parameter and pushes it to the top of the event queue, in any unprocessedI/OBefore the event, and immediately return. Once the event loop runs again, the callback is called immediately.

So look at the following code, we can better use this technology to deal withinconsistentRead()Asynchronous sequence of:

const fs = require('fs');
const cache = {};

function consistentReadAsync(filename, callback) {
  if (cache[filename]) {
    // 下一次事件循环立即调用
    process.nextTick(() => callback(cache[filename]));
  } else {
    // 异步I/O操作
    fs.readFile(filename, 'utf8', (err, data) => {
      cache[filename] = data;
      callback(data);
    });
  }
}

Now, the above function guarantees to call its callback function asynchronously under any circumstances, thus solving the above problem.bug.

The other for delaying code executionAPIYessetImmediate(). Although their functions look very similar, their actual meanings are quite different.process.nextTick()The callback function of theI/OCalled before the operation, whereas forsetImmediate()Will be in otherI/OCalled after the operation. Due toprocess.nextTick()In otherI/OPreviously called, so in some cases it may result inI/OEnter an indefinite wait, such as a recursive callprocess.nextTick()But forsetImmediate()This will not happen. When we analyze using deferred calls to run synchronization later in this bookCPUWhen binding tasks, we will learn more about the differences between the two apis.

We promise that through the use ofprocess.nextTick()Call its callback function asynchronously.

Js callback style

ForNode.jsIn general,CPSStylishAPIAnd callback functions follow a special set of conventions. These agreements are not only applicable toNode.jsCoreAPIIt is also meaningful for most user-level modules and applications that follow them. Therefore, we understand these styles and make sure we design asynchrony when needed.APIIt is of utmost importance to abide by the regulations.

Callback is always the last argument

In all coresNode.jsIn the method, the standard convention is that when a function accepts a callback in its input, it must be passed as the last argument. We use the followingNode.jsCoreAPIFor example:

fs.readFile(filename, [options], callback);

As can be seen from the previous example, even in the presence of optional parameters, callbacks are always placed at the last position. The reason is that in the case of callback definitions, function calls are more readable.

Error handling always comes first

InCPSIn, the error is passed in a callback function in a form different from the correct result. InNode.jsChina,CPSAny error generated by a style callback function is always passed as the first parameter of the callback, and any actual result is passed from the second parameter. If the operation is successful and there are no errors, the first parameter will benullOrundefined. Look at the following code:

fs.readFile('foo.txt', 'utf8', (err, data) => {
  if (err)
    handleError(err);
  else
    processData(data);
});

The above example is the best way to detect errors. If we do not detect errors, we may find it difficult to find and debug thebugBut another problem to consider is that mistakes are alwaysErrorType, which means that simple strings or numbers should not be passed as error objectstry catchCode block capture).

Error propagation

As for the writing method of synchronous blocking, our mistakes are all throughthrowStatement, even if the error jumps in the error stack, we can still capture the error context well.

But forCPSStyle of asynchronous calls, by passing the error to the next callback in the error stack to complete, the following is a typical example:

const fs = require('fs');

function readJSON(filename, callback) {
  fs.readFile(filename, 'utf8', (err, data) => {
    let parsed;
    if (err)
    // 如果有错误产生则退出当前调用
      return callback(err);
    try {
      // 解析文件中的数据
      parsed = JSON.parse(data);
    } catch (err) {
      // 捕获解析中的错误，如果有错误产生，则进行错误处理
      return callback(err);
    }
    // 没有错误，调用回调
    callback(null, parsed);
  });
};

The details we notice from the above example are how to handle the exception correctly when we want tocallbackPass parameters. In addition, when errors occur, we use thereturnStatement, immediately exit the current function call, avoid the following related execution.

Unreachable exception

From the abovereadJSON()Function to avoid throwing any exceptions to thefs.readFile()In the callback function of theJSON.parse()Place one aroundtry catchCode block. Once an error occurs in an asynchronous callback, an exception will be thrown and the loop will jump to the event without propagating the error to the next callback function.

InNode.jsIn, this is an unrecoverable state, and the application will close and print the error to standard output. In order to prove this point, we try to define from the previousreadJSON()Delete from functiontry catchCode block:

const fs = require('fs');

function readJSONThrows(filename, callback) {
  fs.readFile(filename, 'utf8', (err, data) => {
    if (err) {
      return callback(err);
    }
    // 假设parse的执行没有错误
    callback(null, JSON.parse(data));
  });
};

In the above code, we have no way to captureJSON.parseIf we try to pass a nonstandardJSONFile format, will throw the following error:

SyntaxError: Unexpected token d
at Object.parse (native)
at [...]
at fs.js:266:14
at Object.oncomplete (fs.js:107:15)

Now, if we look at the previous error stack trace, we will see it fromfsThe module starts somewhere, just locally.APIFinished reading file and returned tofs.readFile()Function, through the event loop. This information is clearly displayed to us, and the exception is passed from our callback to the stack, then directly into the event loop, and finally captured and thrown into the console.
This also means usingtry catchCode block packing pairreadJSONThrows()The call to will not work because the block is on a different stack than the call callback. The following code shows the opposite of what we just described:

try {
  readJSONThrows('nonJSON.txt', function(err, result) {
    // ... 
  });
} catch (err) {
  console.log('This will not catch the JSON parsing exception');
}

previouscatchStatements will never be receivedJSONParses the exception because it will return to the stack that threw the exception. We just saw the stack end in an event loop instead of triggering asynchronous operations.
As mentioned earlier, the application stopped at the moment when the exception reached the event loop, however, we still had the opportunity to perform some cleanup or logging before the application was terminated. In fact, when this happens,Node.jsA message nameduncaughtExceptionSpecial events of. The following code shows a sample use case:

process.on('uncaughtException', (err) => {
  console.error('This will catch at last the ' +
    'JSON parsing exception: ' + err.message);
  // Terminates the application with 1 (error) as exit code:
  // without the following line, the application would continue
  process.exit(1);
});

What is important is that the uncaught exceptions will leave the application in a state where consistency cannot be guaranteed, which may lead to unforeseen problems. For example, it may be incompleteI/ORequests to run or shut down may become inconsistent. This is why it is always recommended, especially in production environments, to write the above code for error logging after receiving an unhandled exception.

Module system and related modes

Modules are not only the basis for building large-scale applications, but their main mechanism is to encapsulate internal implementations, methods and variables through interfaces. In this section, we will introduceNode.jsThe module system of and its most common usage mode.

About modules

JavaScriptOne of the main problems of is that there is no namespace. Programs running in the global scope will pollute the global namespace and cause conflicts among related variables, data and method names. The technology to solve this problem is called module mode. Look at the following code:

const module = (() => {
  const privateFoo = () => {
    // ...
  };
  const privateBar = [];
  const exported = {
    publicFoo: () => {
      // ...
    },
    publicBar: () => {
      // ...
    }
  };
  return exported;
})();
console.log(module);

This mode uses self-executing anonymous functions to implement the module and only exports the parts that are intended to be called publicly. In the above code, the module variable contains only the exportedAPIAnd the rest of the module content is actually not accessible from outside. We will see later that the idea behind this model is used asNode.jsThe foundation of the modular system.

Js module related explanation

CommonJSIt is an attempt to regulateJavaScriptThe organization of the ecosystem, they put forwardCommonJS module specification.Node.jsBased on this specification, its module system is built and some customized extensions are added. In order to describe its working principle, we can explain the module mode through such an example that each module runs under a private namespace, so that each variable defined in the module will not pollute the global namespace.

Custom module system

In order to explain the remoteness of the module system, let’s build a similar module system from scratch. The following code creates an impersonationNode.jsOriginalrequire()The function of a function.

Let’s first create a function that loads the contents of the module and wrap it in a private namespace:

function loadModule(filename, module, require) {
  const wrappedSrc = `(function(module, exports, require) {
         ${fs.readFileSync(filename, 'utf8')}
       })(module, module.exports, require);`;
  eval(wrappedSrc);
}

The source code of the module is wrapped in a function, just like executing anonymous functions. The difference here is that we pass some inherent variables to the module, specificallymodule,exportsAndrequire. Note that the parameters of the export module aremodule.exportsAndexportsWe will discuss it later.

Please remember, this is just an example, don’t do it in real projects. such aseval()OrVm moduleIt may lead to some security problems, and others may exploit vulnerabilities to carry out injection attacks. We should use it very carefully or even avoid using it completely.eval.

Let’s now look at how the module’s interfaces, variables, etc. arerequire()Function to introduce:

const require = (moduleName) => {
  console.log(`Require invoked for module: ${moduleName}`);
  const id = require.resolve(moduleName);
  // 是否命中缓存
  if (require.cache[id]) {
    return require.cache[id].exports;
  }
  // 定义module
  const module = {
    exports: {},
    id: id
  };
  // 新模块引入，存入缓存
  require.cache[id] = module;
  // 加载模块
  loadModule(id, module, require);
  // 返回导出的变量
  return module.exports;
};
require.cache = {};
require.resolve = (moduleName) => {
  /* 通过模块名作为参数resolve一个完整的模块 */
};

The above function simulates the native for loading modulesNode.jsTherequire()The behavior of a function. Of course, this is just onedemo, it does not accurately and completely reflectrequire()Function, but for better understandingNode.jsInternal implementation of the module system, defining modules and loading modules. The functions of our self-made module system are as follows:

• The module name is passed in as a parameter. The first thing we do is to find the complete path of the module. We call itid.require.resolve()It is specially responsible for this function. It implements related functions through a specific parsing algorithm (to be discussed later).
• If the module has already been loaded, it should exist in the cache. In this case, we immediately return to the modules in the cache.
• If the module has not been loaded, we will load the module for the first time. Creates a module object that contains a that is initialized with an empty object literalexportsProperty. This attribute will be used by the module’s code to derive the module’s commonAPI.
• Cache the first loaded module object.
• The module source code is read from its file, and the code is imported, as described earlier. We passedrequire()The function provides the module object we just created to the module. By operation or replacementmodule.exportsObject to export its public apis.
• Finally, it will represent the commonality of the modulesAPIThemodule.exportsIs returned to the caller.

As we can see,Node.jsThe principle of the module system is not as profound as imagined, it is just that we create and import and export module source code through a series of operations.

Define a module

By viewing our customizationsrequire()Function, now that we know how to define a module. Let’s look at the following example:

// 加载另一个模块
const dependency = require('./anotherModule');
// 模块内的私有函数
function log() {
  console.log(`Well done ${dependency.username}`);
}
// 通过导出API实现共有方法
module.exports.run = () => {
  log();
};

It should be noted that everything in the module is private unless it is assigned tomodule.exportsVariable. Then, when usingrequire()When the module is loaded, the contents of this variable are cached and returned.

Define global variables

Even if all variables and functions declared in the module are defined within their local scope, global variables can still be defined. In fact, the module system discloses a system calledglobalThe special variable of. Everything assigned to this variable will be defined in the global environment.

Note: It is not good to pollute the global namespace, and the advantages of the module system are not fully utilized. Therefore, global variables should only be used if they really need to be used.

Exports module.exports

For many are not yet familiar withNode.jsFor the developers of, they are most likely to confuse isexportsAndmodule.exportsTo export publicAPIThe difference between. VariableexportIt’s just rightmodule.exportsReference to the initial value of; As we have seen,exportsIn essence, it is only a simple object before the module is loaded.

This means that we can only attach new attributes to objects referenced by exported variables, as shown in the following code:

exports.hello = () => {
  console.log('Hello');
}

Give againexportsAssignment has no effect because it does not change.module.exportsThe content of the, it just changed the variable itself. Therefore, the following code is wrong:

exports = () => {
  console.log('Hello');
}

If we want to export something other than objects, such as functions, we can givemodule.exportsReassign:

module.exports = () => {
  console.log('Hello');
}

The require function is synchronized

Another important detail is what we wrote above.require()Function is synchronous, it uses a relatively simple way to return the module contents, and does not need callback function. Therefore, formodule.exportsAlso synchronized, for example, the following code is incorrect:

setTimeout(() => {
  module.exports = function() {
    // ...
  };
}, 100);

Exporting modules in this way will have an important impact on how we define modules, because it limits the way we define and use modules synchronously. This is actually why the coreNode.jsThe library provides synchronizationAPIInstead of asynchronyAPIOne of the most important reasons.

If we need to define a module that requires asynchronous operation for initialization, we can also define and export the module that requires our asynchronous initialization at any time. However, we cannot guarantee that asynchronous modules are defined in this way.require()After that, it can be used immediately. In Chapter 9, we will analyze this problem in detail and propose some models to optimize and solve this problem.

In fact, in the early daysNode.jsThere was once an asynchronous version ofrequire()However, due to its effect on initialization time and asynchronyI/OThe performance of the has a huge impact, soon thisAPIIt was deleted.

Resolve algorithm

Rely on hellThe dependence of software on software packages of different versions is described.Node.jsThis problem is solved by loading different versions of modules, depending on the loading location of the modules. But all bynpmTo complete, the relevant algorithm is calledResolve algorithmTo be usedrequire()In the function.

Now let’s give a quick overview of this algorithm. As described below,resolve()The function returns a module name (moduleName) as input and returns the full path of the module. The path is then used to load its code and can also uniquely identify the module.Resolve algorithmThere are three kinds of rules:

• File module: ifmoduleNameIn order to/At the beginning, then it has been considered as the absolute path of the module. If by./At the beginning, thenmoduleNameConsidered a relative path, it is derived from the use ofrequireThe location of the module of the.
• Core Module: IfmoduleNameNot to/Or./At the beginning, the algorithm will first try to search in the core Node.js module.
• Module package: if no match is foundmoduleNameThe core module of the, then search in the current directorynode_modulesIf it is not foundnode_modules, the search will continue to the upper directory.node_modulesUntil it reaches the root directory of the file system.

For files and package modules, individual files and directories can also be matched tomoduleName. In particular, the algorithm will attempt to match the following:

• <moduleName>.js
• <moduleName>/index.js
• In<moduleName>/package.jsonThemainThe file or directory declared under the value

Specific documents of resolve algorithm

node_modulesThe catalog is actuallynpmPlace where each package is installed and related dependencies are stored. This means that based on the algorithm we just described, each package has its own private dependency. For example, look at the following directory structure:

myApp
├── foo.js
└── node_modules
    ├── depA
    │   └── index.js
    └── depB
        │
        ├── bar.js
        ├── node_modules
        ├── depA
        │    └── index.js
        └── depC
             ├── foobar.js
             └── node_modules
                 └── depA
                     └── index.js

In the previous example,myApp,depBAnddepCAll depend ondepA; However, they all have their own version of private dependency! According to the rules of the analytic algorithm, use therequire('depA')Different files will be loaded according to the required modules, as follows:

• In/myApp/foo.jsCalled in therequire('depA')Will load/myApp/node_modules/depA/index.js
• In/myApp/node_modules/depB/bar.jsCalled in therequire('depA')Will load/myApp/node_modules/depB/node_modules/depA/index.js
• In/myApp/node_modules/depC/foobar.jsCalled in therequire('depA')Will load/myApp/node_modules/depC/node_modules/depA/index.js

Resolve algorithmYesNode.jsThe core part of dependency management, its existence makes even if the application has hundreds of packages, there will be no conflict and version incompatibility.

When we callrequire()The analytic algorithm is transparent to us. However, you can still use therequire.resolve()Directly used by any module.

Module cache

Each module will only be loaded when it is first introduced, and any time thereafterrequire()Calls are taken from previously cached versions. By looking at our previously written customrequire()Function, you can see that caching is crucial for performance improvement, and it also has some other advantages, as follows:

• Making it possible to reuse module dependencies
• To some extent, it is guaranteed that the same instance will always be returned when the same module is required from a given package, thus avoiding conflicts.

Module cache passrequire.cacheVariable view, so you can access it directly if you want. An example in practical application is through deletionrequire.cacheVariable to invalidate a cached module, which is very useful during testing, but can be very dangerous under normal circumstances.

Circular dependence

Many people think that circular dependence isNode.jsInternal design problems, but they can really happen in real projects, so we at least know how toNode.jsMake circular dependency effective. Let’s look at our customrequire()Function, we can immediately see its working principle and matters needing attention.

Look at the following two modules:

• modulea.js

exports.loaded = false;
const b = require('./b');
module.exports = {
  bWasLoaded: b.loaded,
  loaded: true
};

• moduleb.js

exports.loaded = false;
const a = require('./a');
module.exports = {
  aWasLoaded: a.loaded,
  loaded: true
};

Then we were inmain.jsWrite the following code in:

const a = require('./a');
const b = require('./b');
console.log(a);
console.log(b);

Executing the above code prints the following results:

{
  bWasLoaded: true,
  loaded: true
}
{
  aWasLoaded: false,
  loaded: true
}

This result shows the processing sequence of cyclic dependency. Althougha.jsAndb.jsBoth modules are fully initialized when the master module needs them, but when the slave moduleb.jsWhen loading,a.jsThe module is incomplete. In particular, this state of affairs will continue untilb.jsThe moment the load was completed. This kind of situation we should pay attention to, especially to confirm that we are inmain.jsThe order required for the two modules in.

This is due to the modulea.jsYou will receive an incomplete version of theb.js. We now understand that if we lose control of which module to load first, and if the project is large enough, this may easily lead to circular dependency.

Documents on Circular References

In short, in order to prevent the dead cycle of module loading,Node.jsAfter the module is loaded for the first time, its results will be cached, and the results will be directly retrieved from the cache the next time it is loaded again. Therefore, in this circular dependency situation, there will be no dead loop, but the module will not be exported as expected due to caching (exportFor detailed case analysis, see below).

The official website gives the simplest case that the three modules are not cyclic dependencies. In fact, the two modules can clearly express this situation. According to the idea of recursion, the simplest case is solved, and this kind of problem of arbitrary size is half solved (the other half needs to find out how the solution of the problem will change as the problem scale increases).

JavaScriptAs an explanatory language, the above printout clearly shows the trajectory of the program. In this example,a.jsFirst of allrequireTheb.jsProgram entryb.jsInb.jsIn the first line againrequireThea.js.

As mentioned earlier, in order to avoid infinite loop module dependence, inNode.jsruna.jsAfter that, it will be cached, but it should be noted that at this time the cache is only an unfinished one.a.js(an unfinished copy of the a.js）。 So inb.jsInrequireThea.jsWhen, the result is only an unfinished cachea.js, specifically, it does not explicitly export the specific content (a.jsEnd). sob.jsOutput fromaIs an empty object.

After that,b.jsAfter successful implementation, return toa.jsTherequireStatement, continue execution to complete.

Module definition mode

The most common function of a module system is definition, besides its own mechanism for handling dependency relationships.API. For definitionsAPI, the main need to consider the balance between private and public functions. Its purpose is to maximize the internal realization and exposure of information hiding.APIAvailability while balancing these with scalability and code reuse.

In this section, we will analyze some of the problems inNode.jsThe most popular mode for defining modules in; Each module ensures transparency, extensibility and code reuse of private variables.

Named export

Expose the publicAPIThe most basic method of is to use named export, which includes assigning all the values we want to expose to theexport(ormodule.exportsThe property of the object referenced by the. In this way, the generated export object will become a container or namespace for a set of related functions.

Look at the following code, is the implementation of this pattern:

//file logger.js
exports.info = (message) => {
  console.log('info: ' + message);
};
exports.verbose = (message) => {
  console.log('verbose: ' + message);
};

The exported function is then used as an attribute of the module into which it is introduced, as shown in the following code:

// file main.js
const logger = require('./logger');
logger.info('This is an informational message');
logger.verbose('This is a verbose message');

MostNode.jsModules use this definition.

CommonJSThe specification only allows useexportsVariable to the publicpublicMembers. Therefore, the named export mode is unique toCommonJSSpecification compatible mode. Usemodule.exportsYesNode.jsProvides an extension to support a broader module definition pattern.

Function derivation

One of the most popular module definition patterns includes integrating the entiremodule.exportsVariables are reassigned to a function. Its main advantage is that it only exposes one function, providing a clear entry point for the module, making it easier to understand and use, and it also well demonstrates the principle of single responsibility. This method of defining modules is also known in the community asSubstack mode, view this mode in the following example:

// file logger.js
module.exports = (message) => {
  console.log(`info: ${message}`);
};

This mode can also use the exported function as other publicAPIThe namespace of the. This is a very powerful combination because it still gives the module a separate entry point (exportsThe main function of). This approach also allows us to expose other functions with secondary or higher-level use cases. The following code shows how to use the exported functions as namespaces to extend the modules we previously defined:

module.exports.verbose = (message) => {
  console.log(`verbose: ${message}`);
};

This code demonstrates how to call the module we just defined:

// file main.js
const logger = require('./logger');
logger('This is an informational message');
logger.verbose('This is a verbose message');

Although it may be a limitation to export only one function, it is actually a perfect way to focus on a single function, which represents the most important function of this module, and at the same time makes the attributes of internal private variables more transparent, but only exposes the attributes of the exported function itself.

Node.jsThe modularity of encourages us to follow the principle of single responsibility (SRP): Each module should be responsible for a single function, which should be completely encapsulated by the module to ensure reusability.

Attention, what is said hereSubstack mode, which is to expose the main functions of the module by exporting only one function. Use the exported function as a namespace to export other secondary functions.

Constructor (Class) Export

The module that derives the constructor is a special case of the module that derives the function. The difference is that with this new pattern, we allow users to create new instances using constructors, but we can also extend its prototype and create new classes (inheritance). The following is an example of this mode:

// file logger.js
function Logger(name) {
  this.name = name;
}
Logger.prototype.log = function(message) {
  console.log(`[${this.name}] ${message}`);
};
Logger.prototype.info = function(message) {
  this.log(`info: ${message}`);
};
Logger.prototype.verbose = function(message) {
  this.log(`verbose: ${message}`);
};
module.exports = Logger;

We use the above modules in the following ways:

// file main.js
const Logger = require('./logger');
const dbLogger = new Logger('DB');
dbLogger.info('This is an informational message');
const accessLogger = new Logger('ACCESS');
accessLogger.verbose('This is a verbose message');

viaES2015TheclassKeyword syntax can also implement the same pattern:

class Logger {
  constructor(name) {
    this.name = name;
  }
  log(message) {
    console.log(`[${this.name}] ${message}`);
  }
  info(message) {
    this.log(`info: ${message}`);
  }
  verbose(message) {
    this.log(`verbose: ${message}`);
  }
}
module.exports = Logger;

In view ofES2015The class of is only the syntax sugar of the prototype, and the use of this module will be exactly the same as its prototype and constructor based scheme.

The export constructor or class is still the single entry point for the module, but is different from theSubstack modeIn comparison, it exposes more internal structure of the module. However, on the other hand, when we want to expand the function of this module, we can be more convenient.

Variants of this mode include those that do not usenewThe call to the. This little trick lets us use our module as a factory. Look at the following code:

function Logger(name) {
  if (!(this instanceof Logger)) {
    return new Logger(name);
  }
  this.name = name;
};

In fact, this is very simple: we checkthisDoes it exist and is itLoggerAn example of. If any of these conditions arefalse, it meansLogger()Function is not usednew, and then continue to correctly create the new instance and return it to the caller. This technology allows us to also use modules as factories:

// file logger.js
const Logger = require('./logger');
const dbLogger = Logger('DB');
accessLogger.verbose('This is a verbose message');

ES2015Thenew.targetGrammar fromNode.js 6At first, it provides a more concise method to realize the above functions. The utilization disclosednew.targetProperty, which is available in all functionsMeta attributeIf you use thenewKeyword calls a function, the calculation result at runtime istrue.
We can rewrite the factory using this syntax:

function Logger(name) {
  if (!new.target) {
    return new LoggerConstructor(name);
  }
  this.name = name;
}

This code has exactly the same function as the previous code, so we can sayES2015Thenew.targetGrammatical sugar makes the code more readable and natural.

Instance export

We can userequire()To easily define stateful instances with states created from constructors or factories, which can be shared among different modules. The following code shows an example of this mode:

//file logger.js
function Logger(name) {
  this.count = 0;
  this.name = name;
}
Logger.prototype.log = function(message) {
  this.count++;
  console.log('[' + this.name + '] ' + message);
};
module.exports = new Logger('DEFAULT');

This newly defined module can be used as follows:

// file main.js
const logger = require('./logger');
logger.log('This is an informational message');

Because modules are cached, each needs toLoggerModules of a module actually always retrieve the same instance of the object, sharing its state. This pattern is much like creating a singleton. However, it does not guarantee the uniqueness of the entire application instance, because it occurs in the traditional singleton pattern. When analyzing the parsing algorithm, we have actually seen that a module may be installed in the dependency tree of the application many times. This results in multiple instances of the same logical module, all running in the same instance.Node.jsIn the context of the application. In chapter 7, we will analyze and export stateful instances and some alternative patterns.

The expansion of the model we just described includesexportsThe constructor used to create the instance and the instance itself. This allows the user to create new instances of the same object or extend them if necessary. To achieve this, we only need to assign a new attribute to the instance, as shown in the following code:

module.exports.Logger = Logger;

We can then use the exported constructor to create other instances of the class:

const customLogger = new logger.Logger('CUSTOM');
customLogger.log('This is an informational message');

From the perspective of code availability, this is similar to using the exported function as a namespace. The module exports a default instance of an object, which is the function we use most of the time, while more advanced functions (such as the function of creating new instances or extending objects) can still be used with less exposed attributes.

Modify other modules or global scopes

A module can even export anything, which may seem a bit inappropriate; However, we should not forget that a module can modify the global scope and any objects therein, including other modules in the cache. Please note that these are usually considered bad practices, but because this mode may be useful and safe in some cases (e.g. testing), it is worth understanding and understanding that this feature can be used sometimes. We say that one module can modify other modules or objects in the global scope. It usually refers to temporary changes that modify existing objects at runtime to change or extend their behavior or applications.

The following example shows how we add a new function to another module:

// file patcher.js
// ./logger is another module
require('./logger').customMessage = () => console.log('This is a new functionality');

Write the following code:

// file main.js
require('./patcher');
const logger = require('./logger');
logger.customMessage();

In the above code, must first be introducedpatcherPrograms can only be usedloggerModules.

The above wording is very dangerous. The main consideration is that modules with modified global namespaces or other modules are operations with side effects. In other words, it will affect the state of entities outside its scope, which may lead to unpredictable consequences, especially when multiple modules interact with the same entity. Imagine that two different modules try to set the same global variable or modify the same attribute of the same module, and the effect may be unpredictable (which module wins? However, the most important thing is that it will affect the entire application.

observer mode

Node.jsAnother important and basic mode in is the observer mode. AndReactor mode, callback mode and module, observer mode isNode.jsOne of the foundations is also the use of manyNode.jsThe foundation of core modules and user-defined modules.

The observe mode is rightNode.jsThe ideal solution for data response of is also a perfect complement to callback. We give the following definitions:

A publisher defines an object that can notify a group of observers (or listeners) when its state changes.

The main difference from callback mode is that the subject can actually notify multiple observers, whereas traditionalCPSStyle callbacks usually only propagate the main body’s results to a listener.

EventEmitter class

In traditional object-oriented programming, the observer pattern requires interfaces, specific classes, and hierarchical structures. InNode.jsIn fact, it has become much simpler. Observer mode is already built into the core module and can be accessed throughEventEmitterClass.EventEmitterClass allows us to register one or more functions as listeners. When a specific event type is triggered, its callback will be called to notify its listeners. The following images intuitively explain this concept:

EventEmitterIs a class (prototype) derived from the event core module. The following code shows how to get a reference to it:

const EventEmitter = require('events').EventEmitter;
const eeInstance = new EventEmitter();

EventEmitterThe basic method of is as follows:

• on(event，listener): This method allows you to specify the type of eventString typeRegister a new listener (a function)
• once(event, listener): This method registers a new listener and is then deleted after the event is first published
• emit(event, [arg1], [...]): This method generates a new event and provides additional parameters to pass to the listener
• removeListener(event, listener): This method deletes the listener for the specified event type

All of the above methods will returnEventEmitterInstance to allow linking. Listener functionfunction([arg1], [...])So it only accepts the parameters provided when the event is issued. In the listener, this refers toEventEmitterGenerates an instance of the event.
We can see that a listener and a traditionalNode.jsCallbacks are very different; In particular, the first parameter is noterrorWhich is passed to when calledemit()Any data of.

Creating and using EventEmitter

Let’s look at how we use it in practice.EventEmitter. The easiest way is to create a new instance and use it immediately. The following code shows how to use theEventEmitterRealize the function of notifying subscribers in real time:

const EventEmitter = require('events').EventEmitter;
const fs = require('fs');

function findPattern(files, regex) {
  const emitter = new EventEmitter();
  files.forEach(function(file) {
    fs.readFile(file, 'utf8', (err, content) => {
      if (err)
        return emitter.emit('error', err);
      emitter.emit('fileread', file);
      let match;
      if (match = content.match(regex))
        match.forEach(elem => emitter.emit('found', file, elem));
    });
  });
  return emitter;
}

From the previous functionEventEmitterHandle the three events that will occur:

• filereadEvent: Triggered when a file is read
• foundEvent: Triggered when the file content is successfully matched by regular matching
• errorEvent: Triggered when there is an error reading the file

Look belowfindPattern()How is the function triggered:

findPattern(['fileA.txt', 'fileB.json'], /hello \w+/g)
  .on('fileread', file => console.log(file + ' was read'))
  .on('found', (file, match) => console.log('Matched "' + match + '" in file ' + file))
  .on('error', err => console.log('Error emitted: ' + err.message));

In the previous example, we areEventParttern()FunctionEventEmitterOne listener is registered for each event type generated.

Error propagation

If the event is sent asynchronously,EventEmitterYou cannot throw an exception when an exception occurs, and the exception will be lost in the event loop. On the contrary, it isemitIs to issue a special event called an error,Error objectPass through parameters. This is exactly what we defined earlier.findPattern()What is being done in the function.

For error events, it is always best practice to register listeners becauseNode.jsIt will be handled in a special way, and if no associated listener is found, it will automatically throw an exception and exit the program.

Make any object visible

Sometimes, directly throughEventEmitterClass to create a new observable object is not enough because nativeEventEmitterClass does not provide us with the ability to expand the actual use of scenarios. We can expandEventEmitterClass makes a common object observable.

To demonstrate this pattern, we try to implement it in an object.findPattern()The function functions as shown in the following code:

const EventEmitter = require('events').EventEmitter;
const fs = require('fs');
class FindPattern extends EventEmitter {
  constructor(regex) {
    super();
    this.regex = regex;
    this.files = [];
  }
  addFile(file) {
    this.files.push(file);
    return this;
  }
  find() {
    this.files.forEach(file => {
      fs.readFile(file, 'utf8', (err, content) => {
        if (err) {
          return this.emit('error', err);
        }
        this.emit('fileread', file);
        let match = null;
        if (match = content.match(this.regex)) {
          match.forEach(elem => this.emit('found', file, elem));
        }
      });
    });
    return this;
  }
}

We definedFindPatternThe core module is used in the classutilProvidedinherits()Function to expandEventEmitter. In this way, it becomes an observable class that conforms to our actual application scenarios. The following is an example of its usage:

const findPatternObject = new FindPattern(/hello \w+/);
findPatternObject
  .addFile('fileA.txt')
  .addFile('fileB.json')
  .find()
  .on('found', (file, match) => console.log(`Matched "${match}"
       in file ${file}`))
  .on('error', err => console.log(`Error emitted ${err.message}`));

Now, through inheritanceEventEmitterThe function of, we can see nowFindPatternIn addition to being observable, the object has a complete set of methods.
This is inNode.jsIn the ecosystem is a very common model, for example, the coreHTTPmodularServerThe object defines thelisten(),close(),setTimeout()And internally it also inherits fromEventEmitterFunction to allow it to respond to requests related to events when it receives a new request, establishes a new connection, or the server shuts down.

extendEventEmitterOther examples of objects for areNode.jsFlow. We will analyze it in more detail in chapter 5.Node.jsThe flow of.

Synchronous and asynchronous events

Similar to callback mode, events also support synchronous or asynchronous sending. It is essential that we should never be in the same placeEventEmitterThe two methods are mixed in, but it is very important to consider synchronization or asynchrony when publishing the same event type, so as to avoid the inconsistency between synchronization and asynchronyzalgo.

The main difference between publishing synchronous and asynchronous events is the way observers register. When an event is released asynchronously, even whenEventEmitterAfter initialization, the program also registers a new observer, because it must ensure that this event is not triggered before the next cycle of the event cycle. As abovefindPattern()Function. It represents the majorityNode.jsCommon methods used in asynchronous modules.

On the contrary, synchronous publishing events require thatEventEmitterThe function must register the observer before starting to issue any event. Look at the following example:

const EventEmitter = require('events').EventEmitter;
class SyncEmit extends EventEmitter {
  constructor() {
    super();
    this.emit('ready');
  }
}
const syncEmit = new SyncEmit();
syncEmit.on('ready', () => console.log('Object is ready to be  used'));

IfreadyIf the event is issued asynchronously, the above code will run normally. However, since the event is issued synchronously and the listener is registered only after sending the event, the result does not call the listener, and the code will not print to the console.

Due to different application scenarios, it is sometimes used in a synchronous manner.EventEmitterFunctions are meaningful. Therefore, we must clearly highlight ourEventEmitterTo avoid unnecessary errors and exceptions.

Comparison of Event Mechanism and Callback Mechanism

When defining asynchronyAPIWhen, the common difficulty is to check whether to useEventEmitterOr accept only callback functions. The general distinction rule is this: when a result must be returned asynchronously, a callback function should be used, and when the result is needed and its way is uncertain, an event mechanism should be used to respond.

However, there is a lot of confusion because the two are so close and it is possible that both methods can achieve the same application scenario. Take the following code as an example:

function helloEvents() {
  const eventEmitter = new EventEmitter();
  setTimeout(() => eventEmitter.emit('hello', 'hello world'), 100);
  return eventEmitter;
}

function helloCallback(callback) {
  setTimeout(() => callback('hello world'), 100);
}

helloEvents()AndhelloCallback()It can be considered as equivalent in its function. The first uses event mechanism to implement, while the second uses callback to notify the caller and passes the event as a parameter. But what really distinguishes them is their executability, semantics and the amount of code to be implemented or used. Although we cannot give a set of definite rules to choose a style, of course we can provide some tips to help you make a decision.

Compared to the first example, observer mode, callback functions have some limitations in supporting different types of events. But in fact, we can still distinguish multiple events by passing the event type as a parameter of the callback or by accepting multiple callbacks. However, doing so cannot be considered an elegant one.API. In this case,EventEmitterIt can provide better interface and simpler code.

EventEmitterAnother better application scenario is when the same event is triggered multiple times or no event is triggered. In fact, no matter whether the operation is successful or not, a callback is expected to be called only once. However, there is a special case where we may not know at which time the event is triggered. In this case,EventEmitterIs the first choice.

Finally, use the callback’sAPINotify only specific callbacks, but useEventEmitterFunction allows multiple listeners to receive notifications.

The callback mechanism is used in combination with the event mechanism.

There are also cases where event mechanisms and callbacks can be combined. This pattern is particularly useful when we derive asynchronous functions.Node-glob moduleIs an example of this module.

glob(pattern, [options], callback)

The function takes a file name matching pattern as the first parameter, and the latter two parameters are a set of options and a callback function respectively. For the file list matching the specified file name matching pattern, the relevant callback function will be called. At the same time, the function returnsEventEmitterWhich shows the state of the current process. For example, it can be published in real time when the file name is successfully matched.matchEvent, which can be published in real time when all the file lists are matched.endEvent, or when the process is manually abortedabortEvents. Look at the following code:

const glob = require('glob');
glob('data/*.txt', (error, files) => console.log(`All files found: ${JSON.stringify(files)}`))
  .on('match', match => console.log(`Match found: ${match}`));

Summary

In this chapter, we first understand the difference between synchronous and asynchronous. Then, we discussed how to use callback mechanism and callback mechanism to deal with some basic asynchronous schemes. We have also learned the main differences between the two models and when they are more suitable for solving specific problems than the other model. We are only the first step towards a more advanced asynchronous mode.

In the next chapter, we will introduce more complex scenarios and learn how to use callback mechanism and event mechanism to deal with advanced asynchronous control problems.