Call Chain Series 4: Call Chain Context Transfer

  Protocol

In the previous series of articles on call chain, we have already introduced the call chain in detail. I believe everyone has a basic understanding of the call chain technology.

In fact, in the process of drawing the call chain, the transfer of the context of the call chain is very worthy of attention. Each node generates a new context after acquiring the upper context and passes it back. In the process of transfer, once the context is lost or an exception occurs, the data of the call chain will be lost or even broken.

This article mainly describes some implementation details in the context transfer process of call chain in UAV.

Preface

In the implementation of the call chain, there are mainly the following ways to transfer the context of the call chain:

  • Context transfer from before request processing to after request processing;
  • Context transfer between client calls;
  • Context transfer when calling between services.

In these three cases, the information passed and the problems encountered in the context transfer process will be different.

In the context transfer process before and after request processing, the information to be transferred generally includes traceID, spanID, request start time and some request parameters. The related code may face the problem of asynchronous thread delivery due to asynchronous execution.

In client-side calls and inter-service calls, the context information to be transferred generally only includes traceID and spanID. However, the context transfer between client calls may encounter the problem of cross-thread pool transfer, while the inter-service calls may encounter the problem of cross-application transfer.

Therefore, we divide the context transfer we talked about today into the following four scenarios for analysis:

1. Pass in the same thread

2. Cross-thread pool transfer

3. Asynchronous Thread Delivery

4. Transfer across Applications

In order to better illustrate these four scenarios, we assume the following business call procedures exist:

1.png

Assuming that a request first enters service A, a JDBC request is initiated in the service code of service A and a data source is accessed; Then, through httpClient (synchronous, asynchronous), an http access is initiated and the corresponding result is returned.

The number indicates that the point where the call chain context information is obtained. Transfer of call chain context is involved between most adjacent points.

For example, from 2 to 3 is the context transfer before and after the request, from 3 to 4 is the context transfer between two client calls, and from 4 to 5 is the context transfer between services. Next, we will explain the context transfer process between various points under different scenarios.

1. Context transfer within the same thread

This kind of scene is common and the simplest.

Assuming that all the above simulation processes are synchronous operations, and no thread pool (database connection pool does not affect) and asynchronous operations are involved in the business code, then the relevant codes of the call chain in service a will all be executed in the same thread.

Speaking of which, everyone will think that using ThreadLocal can solve the problem. Using ThreadLocal can indeed solve the problem of parameter sharing and passing in the same thread. In UAV, the context transfer between two client calls usually uses ThreadLocal directly (in fact, it is not the original ThreadLocal, which will be described later). The transfer process is as follows:

2.jpg

However, most of the time, asynchronous or submitting thread pool operations are often involved in business code. At this time, using ThreadLocal alone cannot meet the corresponding requirements. Next, let’s discuss the context passing problem with thread pool operations and asynchronous requests.

2. Context Transfer Across Thread Pools

For the first time, let’s look at the issue of context transfer across thread pools.

Assuming that the current thread is only responsible for submitting JDBC operations to the thread pool when performing JDBC operations in the above business scenario, the problem of cross-thread pool will be encountered when context information is transferred from 1 to 2. At this time, it is impossible to transfer context information using ThreadLocal.

Of course, some students may say “InheritableThreadLocal.” However, there is no parent-child relationship between the submitting thread and the thread pool thread itself, so InheritableThreadLocal cannot complete the context transfer across the thread pool.

In order to solve this problem, we used Ali’s open source ThreadLocal component across thread pools: transmittable-thread-localhttps://github.com/alibaba/tr ….

This component can enhance the function of ThreadLocal to realize the transfer across thread pools. The following is an example of TTL in github:

TransmittableThreadLocal<String> parent =newTransmittableThreadLocal<String>();

parent.set("value-set-in-parent");

 

Runnable task =new Task("1");

// 额外的处理,生成修饰了的对象ttlRunnable

Runnable ttlRunnable = TtlRunnable.get(task);

executorService.submit(ttlRunnable);

// Task中可以读取,值是"value-set-in-parent"

String value = parent.get();




As you can see, in order for TTL to work, you need to replace runnable in the service code with TtlRunnable. In order to achieve zero intrusion into business code, we added a ClassFileTransformer for some eexecutiors such as ThreadPoolExecutor with the help of javaagent mechanism, and packaged Runnable and Callable submitted to thread pool into corresponding TtlRunnable and TtlCallable, thus realizing context transfer across thread pool without modifying business code.

In addition, because TTL has all the characteristics of ThreadLocal, ThreadLocal used in context transfer of UAV is TTL.

3. context transfer in asynchronous thread

After looking at the cross-thread pool operation above, let’s look at the asynchronous thread problem again.

Let’s assume that in the above simulation scenario, we sent an asynchronous Http request using asynchronous HttpClient. Due to asynchronous operation, the 4-point code and the 7-point code (where the 7-point context is the context acquisition scenario before and after the request obtained from the 4-point context) will actually be executed in different threads, resulting in the 7-point inability to acquire the context data put into ThreadLocal at 4-point, thus causing data loss in the call chain.

In order to solve this problem, we use both byte code rewriting and dynamic proxy technology in UAV. The key lies in the selection of the target hijacking function, which needs to be able to obtain the callback object of the asynchronous thread.

Taking asynchronous HttpClient as an example, the following describes the transfer process of asynchronous thread context in UAV.

In the asynchronous HttpClient, we hijacked the execute () method of the InternalHttpAsyncClient class, which is declared as follows:

3.png

In general, asynchronous use is to pass in a callback interface object and implement corresponding asynchronous logic in callback. Or use the get () method of the returned Future interface object to implement an asynchronous to synchronous operation.

In order to obtain the context of the call chain in the corresponding place, we first generate the context information of the call chain before the method is executed by rewriting the bytecode. Then the FutureCallback interface is dynamically proxied, and the generated context information is transferred to the proxy object and replaced by the original callback object.

In this way, when the asynchronous request returns to call the callback interface, we actually get our proxy object, and the context transfer process in the asynchronous thread is completed. The specific process is as follows:

4.jpg

In order to support the asynchronous to synchronous operation through the get () method, we also do a dynamic proxy for the returned Future interface to complete the context transfer.

4. Cross-application context transfer

Having said the process of context transfer within applications, let’s look at the issue of context transfer across applications.

Cross-application scenarios are also quite common. In this scenario, the idea of context transfer is to deserialize the context information according to a certain protocol and then put it into the requested transmission message. Hijacked requests in downstream services and obtained the information to complete the context transfer. In the whole processing process, there is no impact on application message analysis.

Common transport protocols include HTTP protocol, Dubbo’s RPC protocol, RocketMQ’s MQ protocol, etc. These protocols will generally contain structures similar to header information to represent additional information for this request.

We can just take advantage of this and transfer the context information to the downstream service to complete the context transfer.

In the following, we will still use asynchronous HttpClient to introduce the UAV’s transfer process across application contexts.

As we said before, in asynchronous HttpClient, we hijacked the execute () method. In this method, we can get the HttpAsyncRequestProducer interface object as follows:

5.png

Through the generateRequest () method, we can get the request object that this request will send, and use the setHeader () method of request to transfer the context information of the call chain into the Header and downstream.

The context here is generally relatively simple, basically consisting of traceID and spanID strings, and the transmission cost is not high.

As for how to resolve this context in downstream services, in fact, it was mentioned in the previous call chain series that the request object corresponding to the request was hijacked in the server side by means of the middleware enhancement framework (MOF) of UAV, and then directly obtained from its header information.

6.jpg

Other protocols such as RPC or MQ are completed in this way in UAV, but the specific API and hijacking point are different.

For example, the RpcContext is used in the Dubbo remote call process, while RocketMQ is put into the UserProperty of msg. Interested students can visit the UAVStack (https://github.com/uavorg/uav ….

Summary

After understanding the transfer process of these contexts, people can realize more powerful functions based on the call chain. In UAV, the function of call chain and log correlation is realized by hijacking the relevant codes in the log input part, obtaining the context of call chain, and then outputting traceID to the business log.

You can also try to get the context of the call chain in your own business code to get through the business data and call chain data, which is convenient for data statistics and problem investigation.

Author: Zhu WenjiangYixin Institute of Technology