Bring in gRPC: distributed link tracking gRPC-Opentracing-Zipkin

  golang, grpc, List sorting, php, zipkin

Bring in gRPC: distributed link tracking gRPC+Opentracing+Zipkin

Original address:Bring in gRPC: distributed link tracking gRPC+Opentracing+Zipkin
Project address: …


In the actual application, you have done so much Server side and written n RPC methods. Want to see the indicators of the method, but nowhere to start?

This article will build one through gRPC+Opentracing+ZipkinDistributed Link Tracking SystemTo realize the link, performance and other indicators of the whole system


What is it?

OpenTracing enables developers to easily add (or replace) the implementation of tracking systems by providing platform-independent and vendor-independent API.

But OpenTracing is not the standard. Because CNCF is not the official standard organization, but its goal is to create more standard API and tools for distributed tracking.

Noun interpretation


A trace represents the execution of a transaction or process in a (distributed) system.


A span represents a single unit of work completed in a distributed system. It also contains “references” to other spans, which allows multiple SPANs to be combined into a complete Trace.

Each span encapsulates the following according to the OpenTracing specification:

  • Operation name
  • Start time and end time
  • key:value span Tags
  • key:value span Logs
  • SpanContext


Span tags (span tags) can be understood as user-defined span comments. It is convenient to query, filter and understand tracking data.


Span logs (Span logs) can record the log information of spannett timing or events. It is mainly used to capture log information of a specific Span and other debugging or information output of the application itself.


SpanContext represents the state that is passed to child span across process boundaries. It is often used when creating a context in a trace diagram

Baggage Items

Baggage Items can be understood as a collection of additional data transferred during trace global operation.

A case


The following can be seen in the figure:

  • Context of execution time
  • Hierarchical Relationship between Services
  • Serial or parallel call chain between services

Combined with the above information, we can find out where the pain points of the system are at once through the context, performance and other index information of the call chain of the whole system in the actual scene.



What is it?

Zipkin is a distributed tracking system. Its function is to collect the timing data needed to solve the delay problem in the microservice architecture. It manages the collection and search of these data

Zipkin’s design is based onGoogle DapperPapers.


docker run -d -p 9411:9411 openzipkin/zipkin

For other installation methods, see: …


Accesshttp:// that Zipkin is functioning properly.


gRPC + Opentracing + Zipkin

In the previous section, we made the following preparations:

  • Understand what Opentracing is.
  • Building Zipkin to Provide Functions of Distributed Tracking System

Next, gRPC interfaces Zipkin through the Opentracing standard API, and then views the data through Zipkin.

directory structure

New simple_zipkin_client, simple_zipkin_server directory, directory structure is as follows:

├── client
│   ├── ...
│   ├── simple_zipkin_client
├── conf
├── pkg
├── proto
├── server
│   ├── ...
│   ├── simple_zipkin_server
└── vendor


$ go get -u
$ go get -u



package main

import (

    zipkin ""

    pb ""

type SearchService struct{}

func (s *SearchService) Search(ctx context.Context, r *pb.SearchRequest) (*pb.SearchResponse, error) {
    return &pb.SearchResponse{Response: r.GetRequest() + " Server"}, nil

const (
    PORT = "9005"

    SERVICE_NAME              = "simple_zipkin_server"

func main() {
    collector, err := zipkin.NewHTTPCollector(ZIPKIN_HTTP_ENDPOINT)
    if err != nil {
        log.Fatalf("zipkin.NewHTTPCollector err: %v", err)

    recorder := zipkin.NewRecorder(collector, true, ZIPKIN_RECORDER_HOST_PORT, SERVICE_NAME)

    tracer, err := zipkin.NewTracer(
        recorder, zipkin.ClientServerSameSpan(false),
    if err != nil {
        log.Fatalf("zipkin.NewTracer err: %v", err)

    tlsServer := gtls.Server{
        CaFile:   "../../conf/ca.pem",
        CertFile: "../../conf/server/server.pem",
        KeyFile:  "../../conf/server/server.key",
    c, err := tlsServer.GetCredentialsByCA()
    if err != nil {
        log.Fatalf("GetTLSCredentialsByCA err: %v", err)

    opts := []grpc.ServerOption{
            otgrpc.OpenTracingServerInterceptor(tracer, otgrpc.LogPayloads()),
  • Zipkin.NewHTTPCollector: Create a Zipkin HTTP backend collector
  • Zipkin.NewRecorder: create a recorder based on the zipkin collector.
  • Zipkin.NewTracer: create an OpenTracing tracker (compatible with Zipkin Tracer)
  • Otgcrpc.opentracingclientnterceptor: returns grpc.UnaryServerInterceptor, except that the interceptor will look for OpenTracing SpanContext in gRPC Metadata. If found, it is the child node of the Span Context of the service.
  • Otgrpc.LogPayloads: sets and returns Option. The function is to let OpenTracing record the payload of the application in both directions.

In general, it is to initialize Zipkin, which in turn includes collector, recorder and tracker. Then the interceptor is used to realize bidirectional reading and management of SpanContext and Payload on the Server side.


func main() {
    // the same as zipkin server
    // ...
    conn, err := grpc.Dial(":"+PORT, grpc.WithTransportCredentials(c),
            otgrpc.OpenTracingClientInterceptor(tracer, otgrpc.LogPayloads()),
  • Otgrpc.opentracingclientnterceptor: return grpc.UnaryClientInterceptor. The core functions of the interceptor are as follows:

(1)OpenTracing SpanContext injection gRPC Metadata

(2) check the context relation in the context. if there is a parent Span, create a ChildOf reference to get a child Span.

In other aspects, it is consistent with the Server side, first initialize Zipkin, then add the interceptor specially needed by the Client side. You can finish the basic work


Start Server.go and execute Client.go examinehttp:// diagram of:



Complex point



Come on, practice it yourself


Under the multi-service architecture, serial, parallel and service package service is a very common situation. It is often difficult to find out where the problem is (the cost is too high) with the conventional scheme. And this kind of situation isDistributed tracking systemThe opportunity to make a big difference

I hope you can understand its concept and build and apply a tracking system through the introduction and study in this chapter.


This series of sample codes

Series catalog