For-loop and json.Unmarshal performance analysis summary

  Analysis, Back end, golang, performance, php

Original address:For-loop and json.Unmarshal performance analysis summary

Preface

In the project, it is often encountered the data processing scene of cyclic exchange assignment, especially RPC. The data exchange format needs to be converted to Protobuf, and assignment is inevitable. There are generally the following methods:

  • for
  • for range
  • json.Marshal/Unmarshal

At this time, we are facing “difficulty in choosing”. Which is better? I also want to have less code and worry about whether the performance will be affected. …

In order to clear up this confusion, the following three usage scenarios will be written separately. Let’s take a look at their performance and see who is better.

Function code

...
type Person struct {
    Name   string `json:"name"`
    Age    int    `json:"age"`
    Avatar string `json:"avatar"`
    Type   string `json:"type"`
}

type AgainPerson struct {
    Name   string `json:"name"`
    Age    int    `json:"age"`
    Avatar string `json:"avatar"`
    Type   string `json:"type"`
}

const MAX = 10000

func InitPerson() []Person {
    var persons []Person
    for i := 0; i < MAX; i++ {
        persons = append(persons, Person{
            Name:   "EDDYCJY",
            Age:    i,
            Avatar: "https://github.com/EDDYCJY",
            Type:   "Person",
        })
    }

    return persons
}

func ForStruct(p []Person, count int) {
    for i := 0; i < count; i++ {
        _, _ = i, p[i]
    }
}

func ForRangeStruct(p []Person) {
    for i, v := range p {
        _, _ = i, v
    }
}

func JsonToStruct(data []byte, againPerson []AgainPerson) ([]AgainPerson, error) {
    err := json.Unmarshal(data, &againPerson)
    return againPerson, err
}

func JsonIteratorToStruct(data []byte, againPerson []AgainPerson) ([]AgainPerson, error) {
    var jsonIter = jsoniter.ConfigCompatibleWithStandardLibrary
    err := jsonIter.Unmarshal(data, &againPerson)
    return againPerson, err
}

Test code

...
func BenchmarkForStruct(b *testing.B) {
    person := InitPerson()
    count := len(person)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        ForStruct(person, count)
    }
}

func BenchmarkForRangeStruct(b *testing.B) {
    person := InitPerson()

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        ForRangeStruct(person)
    }
}

func BenchmarkJsonToStruct(b *testing.B) {
    var (
        person = InitPerson()
        againPersons []AgainPerson
    )
    data, err := json.Marshal(person)
    if err != nil {
        b.Fatalf("json.Marshal err: %v", err)
    }

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        JsonToStruct(data, againPersons)
    }
}

func BenchmarkJsonIteratorToStruct(b *testing.B) {
    var (
        person = InitPerson()
        againPersons []AgainPerson
    )
    data, err := json.Marshal(person)
    if err != nil {
        b.Fatalf("json.Marshal err: %v", err)
    }

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        JsonIteratorToStruct(data, againPersons)
    }
}

test result

BenchmarkForStruct-4                    500000          3289 ns/op           0 B/op           0 allocs/op
BenchmarkForRangeStruct-4               200000          9178 ns/op           0 B/op           0 allocs/op
BenchmarkJsonToStruct-4                    100      19173117 ns/op     2618509 B/op       40036 allocs/op
BenchmarkJsonIteratorToStruct-4            300       4116491 ns/op     3694017 B/op       30047 allocs/op

Judging from the test results, the performance ranking is for < for range < JSON-iterator < encoding/JSON. Next, let’s look at what causes this ranking.

Performance comparison

image

for-loop

In the test results,for rangeCompared in performanceforPoor. Why is this? We can see herefor rangeTheRealization, pseudo implementation is as follows:

for_temp := range
len_temp := len(for_temp)
for index_temp = 0; index_temp < len_temp; index_temp++ {
    value_temp = for_temp[index_temp]
    index = index_temp
    value = value_temp
    original body
}

By analyzing the pseudo-implementation, we can know thatfor rangeCompared withforI have done the following more

Expression

RangeClause = [ ExpressionList "=" | IdentifierList ":=" ] "range" Expression .

The range expression is evaluated before the loop starts, and the final range value is obtained by doing more “solving” the expression.

Copy

...
value_temp = for_temp[index_temp]
index = index_temp
value = value_temp
...

As can be seen from the pseudo-implementation,for rangeAlways useCopy of valueTo generate loop variables. Generally speaking, the circulation variables will be redistributed each time they are circulated.

Summary

Through the above analysis, we can know its ratioforThe slow reason isfor rangeThere are additional performance overhead, mainlyThe act of copying valuesResulting in performance degradation. This is the reason why it is slow.

So in factfor rangeWe can use_AndT[i]Can also reach andforAbout the same performance. But it may not befor rangeThe design of the original intention

json.Marshal/Unmarshal

encoding/json

Json interconversion is the slowest of the three schemes, why?

As is known to all, officialencoding/jsonThe standard library is realized through a large number of reflections. So “slow” is also inevitable. See the following codes:

...
func newTypeEncoder(t reflect.Type, allowAddr bool) encoderFunc {
    ...
    switch t.Kind() {
    case reflect.Bool:
        return boolEncoder
    case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
        return intEncoder
    case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
        return uintEncoder
    case reflect.Float32:
        return float32Encoder
    case reflect.Float64:
        return float64Encoder
    case reflect.String:
        return stringEncoder
    case reflect.Interface:
        return interfaceEncoder
    case reflect.Struct:
        return newStructEncoder(t)
    case reflect.Map:
        return newMapEncoder(t)
    case reflect.Slice:
        return newSliceEncoder(t)
    case reflect.Array:
        return newArrayEncoder(t)
    case reflect.Ptr:
        return newPtrEncoder(t)
    default:
        return unsupportedTypeEncoder
    }
}

Since the official standard stock is at a certain “problem”, is there any other solution? At present, in the community, there are mostly two types of programs. As follows:

  • Precompiling the generated code (determining the type in advance) can solve the performance overhead caused by runtime reflection. The disadvantage is that a pre-generation step is added
  • Optimize serialization logic to maximize performance

In the following experiment, we will use the library of the second scheme to test and see if there is any change. In addition, it is recommended that you know the following items:

json-iterator/go

At present, json-iterator/go is commonly used in the community, and we used it in the test code.

Its usage is 100% compatible with the standard library and its performance has been greatly improved. Let’s take a rough look at how we did it together, as follows:

reflect2

utilizemodern-go/reflect2Reduce runtime scheduling overhead

...
type StructDescriptor struct {
    Type   reflect2.Type
    Fields []*Binding
}

...
type Binding struct {
    levels    []int
    Field     reflect2.StructField
    FromNames []string
    ToNames   []string
    Encoder   ValEncoder
    Decoder   ValDecoder
}

type Extension interface {
    UpdateStructDescriptor(structDescriptor *StructDescriptor)
    CreateMapKeyDecoder(typ reflect2.Type) ValDecoder
    CreateMapKeyEncoder(typ reflect2.Type) ValEncoder
    CreateDecoder(typ reflect2.Type) ValDecoder
    CreateEncoder(typ reflect2.Type) ValEncoder
    DecorateDecoder(typ reflect2.Type, decoder ValDecoder) ValDecoder
    DecorateEncoder(typ reflect2.Type, encoder ValEncoder) ValEncoder
}
struct Encoder/Decoder Cache

When the Type is struct, only Name and type need to be reflected once, and struct Encoder and Decoder will be cached.

var typeDecoders = map[string]ValDecoder{}
var fieldDecoders = map[string]ValDecoder{}
var typeEncoders = map[string]ValEncoder{}
var fieldEncoders = map[string]ValEncoder{}
var extensions = []Extension{}

....

fieldNames := calcFieldNames(field.Name(), tagParts[0], tag)
fieldCacheKey := fmt.Sprintf("%s/%s", typ.String(), field.Name())
decoder := fieldDecoders[fieldCacheKey]
if decoder == nil {
    decoder = decoderOfType(ctx.append(field.Name()), field.Type())
}
encoder := fieldEncoders[fieldCacheKey]
if encoder == nil {
    encoder = encoderOfType(ctx.append(field.Name()), field.Type())
}
文本解析优化

Summary

Compared with the official standard library, the third-party libraryjson-iterator/goDo better at runtime. This is the reason why it is fast.

There is a point to note, after Go1.10mapThere is not much difference in performance between the type and the standard library. However, for examplestructType, etc. still have greater performance improvement

Summary

In this article, we first carried out performance tests and then analyzed different schemes to find out why the speed was slow. Then in the end, when choosing the scheme, you can choose according to different application scenarios:

  • There are higher requirements for performance cost: selectionforWith minimal overhead
  • Moderate to Moderate: Selectionfor range, large objects with caution
  • Small quantity, small occupation, controllable quantity: optionaljson.Marshal/UnmarshalIt is also possible to adopt the plan of. thatDuplicate codeLess, but most expensive

In most scenes, it doesn’t make much difference which one to use. But as an engineer, you should know its advantages and disadvantages. These are different schemes.Summary of analysis, hoping to help you:)