Optimizing Configuration Structure and Realizing Picture Upload

Project address:https://github.com/EDDYCJY/go …

If it is helpful to you, welcome to Star.

Preface

One day, the product manager suddenly told you that the article list was not beautiful enough without the cover picture. ）&￥! &）#&￥! Add one, a few minutes

You open your program, analyze a wave and write a list:

• Optimize the configuration structure (because there are more and more configuration items)
• The File separated from the original logging is convenient for public use (it is not appropriate to keep one copy for logging and one copy for upload respectively)
• Realize the interface for uploading pictures (file format and size shall be limited)
• Modify the article interface (need to support the cover address parameter)
• Add the database field of blog_article
• File server

Well, you find that if you want to be better, you need to adjust the structure of some applications, because there are more and more functions, and the original design should keep pace with the pace.

That is, at the right time, optimize in time

Optimize configuration structure

I. explanation

In the previous chapters, the configuration items were stored by directly reading the KEY. however, in this requirement, the configuration items of pictures need to be added, which is somewhat redundant overall.

We adopt the following solutions:

• Map structure: use MapTo to set configuration parameters
• Configuration management: all configuration items are managed into setting

Mapping Structure (Example)

In go-ini, the structure can be mapped by MapTo, for example:

type Server struct {
    RunMode string
    HttpPort int
    ReadTimeout time.Duration
    WriteTimeout time.Duration
}

var ServerSetting = &Server{}

func main() {
    Cfg, err := ini.Load("conf/app.ini")
    if err != nil {
        log.Fatalf("Fail to parse 'conf/app.ini': %v", err)
    }
    
    err = Cfg.Section("server").MapTo(ServerSetting)
    if err != nil {
        log.Fatalf("Cfg.MapTo ServerSetting err: %v", err)
    }
}

In this code, you can notice that ServerSetting takes the address. why must MapTo include the address?

// MapTo maps section to given struct.
func (s *Section) MapTo(v interface{}) error {
    typ := reflect.TypeOf(v)
    val := reflect.ValueOf(v)
    if typ.Kind() == reflect.Ptr {
        typ = typ.Elem()
        val = val.Elem()
    } else {
        return errors.New("cannot map to non-pointer struct")
    }

    return s.mapTo(val, false)
}

In MapTotyp.Kind() == reflect.PtrConstraints must use pointer, otherwise it will returncannot map to non-pointer structA mistake. This is a superficial reason.

More to explore, can be considered to befield.SetThe reason, when executedval := reflect.ValueOf(v)The function passes thevCopy createdval, butvalThe change of cannot change the originalvIf you want tovalThe change of can be applied tov, it must be passedvAddress of

Obviously go-ini also includes the function of modifying the original value. what do you think is the reason?

Configure unified management

In the previous version, the configurations of models and file were parsed in their own files, while the others were in setting.go, so we need to take over them in setting.

You may think, copy and paste the configuration items of both directly into init of setting.go, and it will be finished in one fell swoop, causing so much trouble?

However, you are thinking that there are multiple init functions in the previous code, and the execution sequence is problematic, which cannot meet our requirements. you can try it.

(Here is a basic knowledge point)

In Go, when there are multiple init functions, the execution order is:

• Init function under the same package: determines the execution order according to the compilation order of the source files (by default, it is sorted by file name)
• Init function under different packages: determining the order according to the dependency relation of package import

So to avoid multiple init,As far as possible by the program to control the initialization sequence

II. Implementation

Modify configuration file

Open conf/app.ini to change the configuration file to the name of the big hump. in addition, we added 5 configuration items for uploading pictures and 4 configuration items for file log

[app]
PageSize = 10
JwtSecret = 233

RuntimeRootPath = runtime/

ImagePrefixUrl = http://127.0.0.1:8000
ImageSavePath = upload/images/
# MB
ImageMaxSize = 5
ImageAllowExts = .jpg,.jpeg,.png

LogSavePath = logs/
LogSaveName = log
LogFileExt = log
TimeFormat = 20060102

[server]
#debug or release
RunMode = debug
HttpPort = 8000
ReadTimeout = 60
WriteTimeout = 60

[database]
Type = mysql
User = root
Password = rootroot
Host = 127.0.0.1:3306
Name = blog
TablePrefix = blog_

Optimizing Configuration Reading and Setting Initialization Sequence

第一步

The configuration scattered in other files will be deleted.Unified processing in settingas well asModify init function to Setup method

Open the pkg/setting/setting.go file and modify it as follows:

package setting

import (
    "log"
    "time"

    "github.com/go-ini/ini"
)

type App struct {
    JwtSecret string
    PageSize int
    RuntimeRootPath string

    ImagePrefixUrl string
    ImageSavePath string
    ImageMaxSize int
    ImageAllowExts []string

    LogSavePath string
    LogSaveName string
    LogFileExt string
    TimeFormat string
}

var AppSetting = &App{}

type Server struct {
    RunMode string
    HttpPort int
    ReadTimeout time.Duration
    WriteTimeout time.Duration
}

var ServerSetting = &Server{}

type Database struct {
    Type string
    User string
    Password string
    Host string
    Name string
    TablePrefix string
}

var DatabaseSetting = &Database{}

func Setup() {
    Cfg, err := ini.Load("conf/app.ini")
    if err != nil {
        log.Fatalf("Fail to parse 'conf/app.ini': %v", err)
    }

    err = Cfg.Section("app").MapTo(AppSetting)
    if err != nil {
        log.Fatalf("Cfg.MapTo AppSetting err: %v", err)
    }

    AppSetting.ImageMaxSize = AppSetting.ImageMaxSize * 1024 * 1024

    err = Cfg.Section("server").MapTo(ServerSetting)
    if err != nil {
        log.Fatalf("Cfg.MapTo ServerSetting err: %v", err)
    }

    ServerSetting.ReadTimeout = ServerSetting.ReadTimeout * time.Second
    ServerSetting.WriteTimeout = ServerSetting.ReadTimeout * time.Second

    err = Cfg.Section("database").MapTo(DatabaseSetting)
    if err != nil {
        log.Fatalf("Cfg.MapTo DatabaseSetting err: %v", err)
    }
}

Here, we have done the following:

• Write structures (App, Server, Database) consistent with configuration items
• Use MapTo to map configuration items to structures
• Reassign some configuration items that need special settings

What you need to do:

• willmodels.go、setting.go、pkg/logging/log.goThe init function of is modified to the Setup method
• willmodels/models.goThe DB configuration item read independently is deleted and changed to read setting uniformly.
• willpkg/logging/fileThe independent LOG configuration item is deleted and changed to read setting uniformly.

I hope you can do it yourself. If you have any questions, you can turn right.Project address

第二步

In this step, we will set up the initialization process, open the main.go file and modify the contents:

func main() {
    setting.Setup()
    models.Setup()
    logging.Setup()

    endless.DefaultReadTimeOut = setting.ServerSetting.ReadTimeout
    endless.DefaultWriteTimeOut = setting.ServerSetting.WriteTimeout
    endless.DefaultMaxHeaderBytes = 1 << 20
    endPoint := fmt.Sprintf(":%d", setting.ServerSetting.HttpPort)

    server := endless.NewServer(endPoint, routers.InitRouter())
    server.BeforeBegin = func(add string) {
        log.Printf("Actual pid is %d", syscall.Getpid())
    }

    err := server.ListenAndServe()
    if err != nil {
        log.Printf("Server err: %v", err)
    }
}

After modification, the initialization function of multiple modules was successfully put into the startup process (the sequence can also be controlled)

验证

At this point, the configuration optimization for this requirement is finished and you need to perform it.go run main.goVerify that your function is normal

By the way, let’s leave a basic question for everyone to think about.

ServerSetting.ReadTimeout = ServerSetting.ReadTimeout * time.Second
ServerSetting.WriteTimeout = ServerSetting.ReadTimeout * time.Second

If these two lines in the setting.go file are deleted, what will be the problem and why?

Pull away from File

In previous versions, inlogging/file.goSome methods of os are used in. We found that this part can be reused in uploading pictures through early planning.

First step

Create a new file/file.go under the pkg directory, and write the following file contents:

package file

import (
    "os"
    "path"
    "mime/multipart"
    "io/ioutil"
)

func GetSize(f multipart.File) (int, error) {
    content, err := ioutil.ReadAll(f)

    return len(content), err
}

func GetExt(fileName string) string {
    return path.Ext(fileName)
}

func CheckExist(src string) bool {
    _, err := os.Stat(src)

    return os.IsNotExist(err)
}

func CheckPermission(src string) bool {
    _, err := os.Stat(src)

    return os.IsPermission(err)
}

func IsNotExistMkDir(src string) error {
    if exist := CheckExist(src); exist == false {
        if err := MkDir(src); err != nil {
            return err
        }
    }

    return nil
}

func MkDir(src string) error {
    err := os.MkdirAll(src, os.ModePerm)
    if err != nil {
        return err
    }

    return nil
}

func Open(name string, flag int, perm os.FileMode) (*os.File, error) {
    f, err := os.OpenFile(name, flag, perm)
    if err != nil {
        return nil, err
    }

    return f, nil
}

Here we have packaged a total of seven methods

• GetSize: get file size
• GetExt: get file suffix
• CheckExist: check whether the file exists
• CheckPermission: check file permissions
• IsNotExistMkDir: create a new folder if it does not exist
• MkDir: new folder
• Open: open file

We used it heremime/multipartPackage, which mainly implements multipart parsing of MIME, is mainly applicable toHTTPAnd multipart bodies generated by common browsers

What is multipart?rfc2388Take a look at the multipart/form-data of

Second step

In the first step, we have repackaged file by one layer. In this step, we have modified the original logging package method.

1. open the pkg/logging/file.go file and modify the contents of the file:

package logging

import (
    "fmt"
    "os"
    "time"

    "github.com/EDDYCJY/go-gin-example/pkg/setting"
    "github.com/EDDYCJY/go-gin-example/pkg/file"
)

func getLogFilePath() string {
    return fmt.Sprintf("%s%s", setting.AppSetting.RuntimeRootPath, setting.AppSetting.LogSavePath)
}

func getLogFileName() string {
    return fmt.Sprintf("%s%s.%s",
        setting.AppSetting.LogSaveName,
        time.Now().Format(setting.AppSetting.TimeFormat),
        setting.AppSetting.LogFileExt,
    )
}

func openLogFile(fileName, filePath string) (*os.File, error) {
    dir, err := os.Getwd()
    if err != nil {
        return nil, fmt.Errorf("os.Getwd err: %v", err)
    }

    src := dir + "/" + filePath
    perm := file.CheckPermission(src)
    if perm == true {
        return nil, fmt.Errorf("file.CheckPermission Permission denied src: %s", src)
    }

    err = file.IsNotExistMkDir(src)
    if err != nil {
        return nil, fmt.Errorf("file.IsNotExistMkDir src: %s, err: %v", src, err)
    }

    f, err := file.Open(src + fileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        return nil, fmt.Errorf("Fail to OpenFile :%v", err)
    }

    return f, nil
}

We changed all references to methods in the file/file.go package.

2. open pkg/logging/log.go file and modify the file contents:

package logging

...

func Setup() {
    var err error
    filePath := getLogFilePath()
    fileName := getLogFileName()
    F, err = openLogFile(fileName, filePath)
    if err != nil {
        log.Fatalln(err)
    }

    logger = log.New(F, DefaultPrefix, log.LstdFlags)
}

...

Since the formal parameters of the original method have changed, the openLogFile also needs to be adjusted.

The interface for uploading pictures is realized.

In this section, we begin to realize some methods and functions related to the last picture.

First you need to add a field to blog_articlecover_image_url, in the formatVarchar(255) DEFAULT '' COMMENT' cover picture address'

Step zero

Generally, the uploaded picture name will not be directly exposed, so we use MD5 to achieve this effect

Create a new md5.go in util directory and write the file contents:

package util

import (
    "crypto/md5"
    "encoding/hex"
)

func EncodeMD5(value string) string {
    m := md5.New()
    m.Write([]byte(value))

    return hex.EncodeToString(m.Sum(nil))
}

First step

We have already packaged the underlying method previously, and this step is actually the processing logic for packaging image.

Create a new upload/image.go file under the pkg directory and write the contents of the file:

package upload

import (
    "os"
    "path"
    "log"
    "fmt"
    "strings"
    "mime/multipart"

    "github.com/EDDYCJY/go-gin-example/pkg/file"
    "github.com/EDDYCJY/go-gin-example/pkg/setting"
    "github.com/EDDYCJY/go-gin-example/pkg/logging"
    "github.com/EDDYCJY/go-gin-example/pkg/util"
)

func GetImageFullUrl(name string) string {
    return setting.AppSetting.ImagePrefixUrl + "/" + GetImagePath() + name
}

func GetImageName(name string) string {
    ext := path.Ext(name)
    fileName := strings.TrimSuffix(name, ext)
    fileName = util.EncodeMD5(fileName)

    return fileName + ext
}

func GetImagePath() string {
    return setting.AppSetting.ImageSavePath
}

func GetImageFullPath() string {
    return setting.AppSetting.RuntimeRootPath + GetImagePath()
}

func CheckImageExt(fileName string) bool {
    ext := file.GetExt(fileName)
    for _, allowExt := range setting.AppSetting.ImageAllowExts {
        if strings.ToUpper(allowExt) == strings.ToUpper(ext) {
            return true
        }
    }

    return false
}

func CheckImageSize(f multipart.File) bool {
    size, err := file.GetSize(f)
    if err != nil {
        log.Println(err)
        logging.Warn(err)
        return false
    }

    return size <= setting.AppSetting.ImageMaxSize
}

func CheckImage(src string) error {
    dir, err := os.Getwd()
    if err != nil {
        return fmt.Errorf("os.Getwd err: %v", err)
    }

    err = file.IsNotExistMkDir(dir + "/" + src)
    if err != nil {
        return fmt.Errorf("file.IsNotExistMkDir err: %v", err)
    }

    perm := file.CheckPermission(src)
    if perm == true {
        return fmt.Errorf("file.CheckPermission Permission denied src: %s", src)
    }

    return nil
}

Here we have implemented 7 methods, as follows:

• GetImageFullUrl: get the full picture access URL
• GetImageName: get the picture name
• GetImagePath: get the picture path
• GetImageFullPath: get the full path of the picture
• CheckImageExt: check the picture suffix
• CheckImageSize: check the image size
• CheckImage: check picture

This is basically a secondary encapsulation of the underlying code. In order to handle some picture-specific logic more flexibly and to facilitate modification, the underlying code is not directly exposed to the outside world.

Second step

This step will write the business logic for uploading pictures, create a new upload.go file under the routers/api directory, and write the contents of the file:

package api

import (
    "net/http"

    "github.com/gin-gonic/gin"

    "github.com/EDDYCJY/go-gin-example/pkg/e"
    "github.com/EDDYCJY/go-gin-example/pkg/logging"
    "github.com/EDDYCJY/go-gin-example/pkg/upload"
)

func UploadImage(c *gin.Context) {
    code := e.SUCCESS
    data := make(map[string]string)

    file, image, err := c.Request.FormFile("image")
    if err != nil {
        logging.Warn(err)
        code = e.ERROR
        c.JSON(http.StatusOK, gin.H{
            "code": code,
            "msg":  e.GetMsg(code),
            "data": data,
        })
    }

    if image == nil {
        code = e.INVALID_PARAMS
    } else {
        imageName := upload.GetImageName(image.Filename)
        fullPath := upload.GetImageFullPath()
        savePath := upload.GetImagePath()

        src := fullPath + imageName
        if ! upload.CheckImageExt(imageName) || ! upload.CheckImageSize(file) {
            code = e.ERROR_UPLOAD_CHECK_IMAGE_FORMAT
        } else {
            err := upload.CheckImage(fullPath)
            if err != nil {
                logging.Warn(err)
                code = e.ERROR_UPLOAD_CHECK_IMAGE_FAIL
            } else if err := c.SaveUploadedFile(image, src); err != nil {
                logging.Warn(err)
                code = e.ERROR_UPLOAD_SAVE_IMAGE_FAIL
            } else {
                data["image_url"] = upload.GetImageFullUrl(imageName)
                data["image_save_url"] = savePath + imageName
            }
        }
    }

    c.JSON(http.StatusOK, gin.H{
        "code": code,
        "msg":  e.GetMsg(code),
        "data": data,
    })
}

Error codes involved (to be added at pkg/e/code.go, msg.go):

// 保存图片失败
ERROR_UPLOAD_SAVE_IMAGE_FAIL = 30001
// 检查图片失败
ERROR_UPLOAD_CHECK_IMAGE_FAIL = 30002
// 校验图片错误，图片格式或大小有问题
ERROR_UPLOAD_CHECK_IMAGE_FORMAT = 30003

In this large section of business logic, we have done the following:

• C.Request.FormFile: get the uploaded picture (return the first file of the provided form key)
• CheckImageExt, CheckImageSize check picture size, check picture suffix
• CheckImage: check the required (permissions, folders) for uploading pictures.
• SaveUploadedFile: save picture

In general, it is the application process saved by “enter the parameter-> check-“

Third step

Open the routers/router.go file to add routesr.POST("/upload", api.UploadImage)For example:

func InitRouter() *gin.Engine {
    r := gin.New()
    ...
    r.GET("/auth", api.GetAuth)
    r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))
    r.POST("/upload", api.UploadImage)

    apiv1 := r.Group("/api/v1")
    apiv1.Use(jwt.JWT())
    {
        ...
    }

    return r
}

Verification

Finally, we request the interface to upload pictures and test the functions written.

Check whether the directory contains files (pay attention to permissions)

$ pwd
$GOPATH/src/github.com/EDDYCJY/go-gin-example/runtime/upload/images

$ ll
... 96a3be3cf272e017046d1b2674a52bd3.jpg
... c39fa784216313cf2faa7c98739fc367.jpeg

Here we have returned a total of 2 parameters, one is the complete access URL and the other is the save path

File server

After completing the previous section, we also need to enable the front end to access the picture, which is generally as follows:

• CDN
• http.FileSystem

In the case of companies, CDN or self-built distributed file systems are in the majority and do not need too much attention. In practice, it must be built locally, Go itself has a good support for this, and Gin is another layer of encapsulation, only need to add a line of code in the route

r.StaticFS

Open the routers/router.go file to add routesr.StaticFS("/upload/images", http.Dir(upload.GetImageFullPath()))For example:

func InitRouter() *gin.Engine {
    ...
    r.StaticFS("/upload/images", http.Dir(upload.GetImageFullPath()))

    r.GET("/auth", api.GetAuth)
    r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))
    r.POST("/upload", api.UploadImage)
    ...
}

What did it do

When accessing $HOST/upload/images, files under $ gopath/src/github.com/eddycjy/go-gin-example/runtime/upload/images will be read.

What does this line of code do? Let’s look at the prototype method

// StaticFS works just like `Static()` but a custom `http.FileSystem` can be used instead.
// Gin by default user: gin.Dir()
func (group *RouterGroup) StaticFS(relativePath string, fs http.FileSystem) IRoutes {
    if strings.Contains(relativePath, ":") || strings.Contains(relativePath, "*") {
        panic("URL parameters can not be used when serving a static folder")
    }
    handler := group.createStaticHandler(relativePath, fs)
    urlPattern := path.Join(relativePath, "/*filepath")

    // Register GET and HEAD handlers
    group.GET(urlPattern, handler)
    group.HEAD(urlPattern, handler)
    return group.returnObj()
}

First, the use of * and: symbols is prohibited in the exposed URLcreateStaticHandlerAfter creating the static file service, the actual final call is stillfileServer.ServeHTTPAnd some processing logic

func (group *RouterGroup) createStaticHandler(relativePath string, fs http.FileSystem) HandlerFunc {
    absolutePath := group.calculateAbsolutePath(relativePath)
    fileServer := http.StripPrefix(absolutePath, http.FileServer(fs))
    _, nolisting := fs.(*onlyfilesFS)
    return func(c *Context) {
        if nolisting {
            c.Writer.WriteHeader(404)
        }
        fileServer.ServeHTTP(c.Writer, c.Request)
    }
}

http.StripPrefix

We can watch outfileServer := http.StripPrefix(absolutePath, http.FileServer(fs))This statement is very common in static file services. What is its function?

http.StripPrefixThe main function is to remove the given prefix from the path of the request URL and finally return oneHandler

Usually http.FileServer is used in combination with http.StripPrefix, otherwise when you run:

http.Handle("/upload/images", http.FileServer(http.Dir("upload/images")))

You will not be able to correctly access the file directory because/upload/imagesAlso included in the URL path, must use:

http.Handle("/upload/images", http.StripPrefix("upload/images", http.FileServer(http.Dir("upload/images"))))

/*filepath

As you can see belowurlPattern := path.Join(relativePath, "/*filepath"),/*filepathWho are you, what are you doing here, are you the product of Gin?

The processing logic of the route can be known through semantics, while Gin’s route is based on httprouter, and the following information can be obtained through consulting documents

Pattern: /src/*filepath

 /src/                     match
 /src/somefile.go          match
 /src/subdir/somefile.go   match

*filepathAll file paths will be matched and*filepathMust be at the end of the Pattern

Verification

Re-executiongo run main.goTo access the picture address just obtained in the upload interface and check whether http.FileSystem is normal

Modify article interface

Next, you are required to modify the two interfaces of routers/api/v1/article.go: AddArticle and EditArticle

• Add and update the article interface: support to enter cover_image_url
• Add and update article interface: add non-empty and longest length check for cover_image_url

As mentioned in the previous article, if you have any questions, please refer to the code of the project.

Summary

In this chapter, we simply analyzed the following requirements and made a small plan and implementation for the application.

After completing the function points and optimization in the list, it is also a common scene in actual projects. I hope you can taste them carefully and learn from them deeply.

References

This series of sample codes

• go-gin-example

This series of catalogues

• Serial One Golang Introduction and Environmental Installation
• Serialized 2 to build Blog API’s (1)
• Serial 3 to build Blog API’s (2)
• Serial 4 to build Blog API’s (3)
• Serial 5 Use JWT for Identity Verification
• Serial 6 Write a Simple File Log
• Serial Seven Golang Gracefully Restart HTTP Service
• Serial 8 Add Swagger to It
• Serial 9 Deploying Golang Applications to Docker
• Serial Ten Customized GORM Callbacks
• Serial Eleven Cron Scheduled Tasks
• Serial 12 Optimizing Configuration Structure and Realizing Picture Upload
• Serialization 13 Optimize Your Application Structure and Implement Redis Cache
• Serial 14 Realize Export and Import into Excel
• Serial 15 Generate Two-dimensional Code and Merge Posters
• Serial 16 Draw Text on Pictures
• Serial 17 Deploying Go Applications with Nginx
• Cross-compilation of safari Golang
• Please get started with Makefile