Optimizing Configuration Structure and Realizing Picture Upload
Project address:https://github.com/EDDYCJY/go …
If it is helpful to you, welcome to Star.
Preface
One day, the product manager suddenly told you that the article list was not beautiful enough without the cover picture. )&¥! &)#&¥! Add one, a few minutes
You open your program, analyze a wave and write a list:
- Optimize the configuration structure (because there are more and more configuration items)
- The File separated from the original logging is convenient for public use (it is not appropriate to keep one copy for logging and one copy for upload respectively)
- Realize the interface for uploading pictures (file format and size shall be limited)
- Modify the article interface (need to support the cover address parameter)
- Add the database field of blog_article
- File server
Well, you find that if you want to be better, you need to adjust the structure of some applications, because there are more and more functions, and the original design should keep pace with the pace.
That is, at the right time, optimize in time
Optimize configuration structure
I. explanation
In the previous chapters, the configuration items were stored by directly reading the KEY. however, in this requirement, the configuration items of pictures need to be added, which is somewhat redundant overall.
We adopt the following solutions:
- Map structure: use MapTo to set configuration parameters
- Configuration management: all configuration items are managed into setting
Mapping Structure (Example)
In go-ini, the structure can be mapped by MapTo, for example:
type Server struct {
RunMode string
HttpPort int
ReadTimeout time.Duration
WriteTimeout time.Duration
}
var ServerSetting = &Server{}
func main() {
Cfg, err := ini.Load("conf/app.ini")
if err != nil {
log.Fatalf("Fail to parse 'conf/app.ini': %v", err)
}
err = Cfg.Section("server").MapTo(ServerSetting)
if err != nil {
log.Fatalf("Cfg.MapTo ServerSetting err: %v", err)
}
}
In this code, you can notice that ServerSetting takes the address. why must MapTo include the address?
// MapTo maps section to given struct.
func (s *Section) MapTo(v interface{}) error {
typ := reflect.TypeOf(v)
val := reflect.ValueOf(v)
if typ.Kind() == reflect.Ptr {
typ = typ.Elem()
val = val.Elem()
} else {
return errors.New("cannot map to non-pointer struct")
}
return s.mapTo(val, false)
}
In MapTotyp.Kind() == reflect.Ptr
Constraints must use pointer, otherwise it will returncannot map to non-pointer struct
A mistake. This is a superficial reason.
More to explore, can be considered to befield.Set
The reason, when executedval := reflect.ValueOf(v)
The function passes thev
Copy createdval
, butval
The change of cannot change the originalv
If you want toval
The change of can be applied tov
, it must be passedv
Address of
Obviously go-ini also includes the function of modifying the original value. what do you think is the reason?
Configure unified management
In the previous version, the configurations of models and file were parsed in their own files, while the others were in setting.go, so we need to take over them in setting.
You may think, copy and paste the configuration items of both directly into init of setting.go, and it will be finished in one fell swoop, causing so much trouble?
However, you are thinking that there are multiple init functions in the previous code, and the execution sequence is problematic, which cannot meet our requirements. you can try it.
(Here is a basic knowledge point)
In Go, when there are multiple init functions, the execution order is:
- Init function under the same package: determines the execution order according to the compilation order of the source files (by default, it is sorted by file name)
- Init function under different packages: determining the order according to the dependency relation of package import
So to avoid multiple init,As far as possible by the program to control the initialization sequence
II. Implementation
Modify configuration file
Open conf/app.ini to change the configuration file to the name of the big hump. in addition, we added 5 configuration items for uploading pictures and 4 configuration items for file log
[app]
PageSize = 10
JwtSecret = 233
RuntimeRootPath = runtime/
ImagePrefixUrl = http://127.0.0.1:8000
ImageSavePath = upload/images/
# MB
ImageMaxSize = 5
ImageAllowExts = .jpg,.jpeg,.png
LogSavePath = logs/
LogSaveName = log
LogFileExt = log
TimeFormat = 20060102
[server]
#debug or release
RunMode = debug
HttpPort = 8000
ReadTimeout = 60
WriteTimeout = 60
[database]
Type = mysql
User = root
Password = rootroot
Host = 127.0.0.1:3306
Name = blog
TablePrefix = blog_
Optimizing Configuration Reading and Setting Initialization Sequence
第一步
The configuration scattered in other files will be deleted.Unified processing in settingas well asModify init function to Setup method
Open the pkg/setting/setting.go file and modify it as follows:
package setting
import (
"log"
"time"
"github.com/go-ini/ini"
)
type App struct {
JwtSecret string
PageSize int
RuntimeRootPath string
ImagePrefixUrl string
ImageSavePath string
ImageMaxSize int
ImageAllowExts []string
LogSavePath string
LogSaveName string
LogFileExt string
TimeFormat string
}
var AppSetting = &App{}
type Server struct {
RunMode string
HttpPort int
ReadTimeout time.Duration
WriteTimeout time.Duration
}
var ServerSetting = &Server{}
type Database struct {
Type string
User string
Password string
Host string
Name string
TablePrefix string
}
var DatabaseSetting = &Database{}
func Setup() {
Cfg, err := ini.Load("conf/app.ini")
if err != nil {
log.Fatalf("Fail to parse 'conf/app.ini': %v", err)
}
err = Cfg.Section("app").MapTo(AppSetting)
if err != nil {
log.Fatalf("Cfg.MapTo AppSetting err: %v", err)
}
AppSetting.ImageMaxSize = AppSetting.ImageMaxSize * 1024 * 1024
err = Cfg.Section("server").MapTo(ServerSetting)
if err != nil {
log.Fatalf("Cfg.MapTo ServerSetting err: %v", err)
}
ServerSetting.ReadTimeout = ServerSetting.ReadTimeout * time.Second
ServerSetting.WriteTimeout = ServerSetting.ReadTimeout * time.Second
err = Cfg.Section("database").MapTo(DatabaseSetting)
if err != nil {
log.Fatalf("Cfg.MapTo DatabaseSetting err: %v", err)
}
}
Here, we have done the following:
- Write structures (App, Server, Database) consistent with configuration items
- Use MapTo to map configuration items to structures
- Reassign some configuration items that need special settings
What you need to do:
- willmodels.go、setting.go、pkg/logging/log.goThe init function of is modified to the Setup method
- willmodels/models.goThe DB configuration item read independently is deleted and changed to read setting uniformly.
- willpkg/logging/fileThe independent LOG configuration item is deleted and changed to read setting uniformly.
I hope you can do it yourself. If you have any questions, you can turn right.Project address
第二步
In this step, we will set up the initialization process, open the main.go file and modify the contents:
func main() {
setting.Setup()
models.Setup()
logging.Setup()
endless.DefaultReadTimeOut = setting.ServerSetting.ReadTimeout
endless.DefaultWriteTimeOut = setting.ServerSetting.WriteTimeout
endless.DefaultMaxHeaderBytes = 1 << 20
endPoint := fmt.Sprintf(":%d", setting.ServerSetting.HttpPort)
server := endless.NewServer(endPoint, routers.InitRouter())
server.BeforeBegin = func(add string) {
log.Printf("Actual pid is %d", syscall.Getpid())
}
err := server.ListenAndServe()
if err != nil {
log.Printf("Server err: %v", err)
}
}
After modification, the initialization function of multiple modules was successfully put into the startup process (the sequence can also be controlled)
验证
At this point, the configuration optimization for this requirement is finished and you need to perform it.go run main.go
Verify that your function is normal
By the way, let’s leave a basic question for everyone to think about.
ServerSetting.ReadTimeout = ServerSetting.ReadTimeout * time.Second
ServerSetting.WriteTimeout = ServerSetting.ReadTimeout * time.Second
If these two lines in the setting.go file are deleted, what will be the problem and why?
Pull away from File
In previous versions, inlogging/file.goSome methods of os are used in. We found that this part can be reused in uploading pictures through early planning.
First step
Create a new file/file.go under the pkg directory, and write the following file contents:
package file
import (
"os"
"path"
"mime/multipart"
"io/ioutil"
)
func GetSize(f multipart.File) (int, error) {
content, err := ioutil.ReadAll(f)
return len(content), err
}
func GetExt(fileName string) string {
return path.Ext(fileName)
}
func CheckExist(src string) bool {
_, err := os.Stat(src)
return os.IsNotExist(err)
}
func CheckPermission(src string) bool {
_, err := os.Stat(src)
return os.IsPermission(err)
}
func IsNotExistMkDir(src string) error {
if exist := CheckExist(src); exist == false {
if err := MkDir(src); err != nil {
return err
}
}
return nil
}
func MkDir(src string) error {
err := os.MkdirAll(src, os.ModePerm)
if err != nil {
return err
}
return nil
}
func Open(name string, flag int, perm os.FileMode) (*os.File, error) {
f, err := os.OpenFile(name, flag, perm)
if err != nil {
return nil, err
}
return f, nil
}
Here we have packaged a total of seven methods
- GetSize: get file size
- GetExt: get file suffix
- CheckExist: check whether the file exists
- CheckPermission: check file permissions
- IsNotExistMkDir: create a new folder if it does not exist
- MkDir: new folder
- Open: open file
We used it heremime/multipart
Package, which mainly implements multipart parsing of MIME, is mainly applicable toHTTPAnd multipart bodies generated by common browsers
What is multipart?rfc2388Take a look at the multipart/form-data of
Second step
In the first step, we have repackaged file by one layer. In this step, we have modified the original logging package method.
1. open the pkg/logging/file.go file and modify the contents of the file:
package logging
import (
"fmt"
"os"
"time"
"github.com/EDDYCJY/go-gin-example/pkg/setting"
"github.com/EDDYCJY/go-gin-example/pkg/file"
)
func getLogFilePath() string {
return fmt.Sprintf("%s%s", setting.AppSetting.RuntimeRootPath, setting.AppSetting.LogSavePath)
}
func getLogFileName() string {
return fmt.Sprintf("%s%s.%s",
setting.AppSetting.LogSaveName,
time.Now().Format(setting.AppSetting.TimeFormat),
setting.AppSetting.LogFileExt,
)
}
func openLogFile(fileName, filePath string) (*os.File, error) {
dir, err := os.Getwd()
if err != nil {
return nil, fmt.Errorf("os.Getwd err: %v", err)
}
src := dir + "/" + filePath
perm := file.CheckPermission(src)
if perm == true {
return nil, fmt.Errorf("file.CheckPermission Permission denied src: %s", src)
}
err = file.IsNotExistMkDir(src)
if err != nil {
return nil, fmt.Errorf("file.IsNotExistMkDir src: %s, err: %v", src, err)
}
f, err := file.Open(src + fileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return nil, fmt.Errorf("Fail to OpenFile :%v", err)
}
return f, nil
}
We changed all references to methods in the file/file.go package.
2. open pkg/logging/log.go file and modify the file contents:
package logging
...
func Setup() {
var err error
filePath := getLogFilePath()
fileName := getLogFileName()
F, err = openLogFile(fileName, filePath)
if err != nil {
log.Fatalln(err)
}
logger = log.New(F, DefaultPrefix, log.LstdFlags)
}
...
Since the formal parameters of the original method have changed, the openLogFile also needs to be adjusted.
The interface for uploading pictures is realized.
In this section, we begin to realize some methods and functions related to the last picture.
First you need to add a field to blog_articlecover_image_url
, in the formatVarchar(255) DEFAULT '' COMMENT' cover picture address'
Step zero
Generally, the uploaded picture name will not be directly exposed, so we use MD5 to achieve this effect
Create a new md5.go in util directory and write the file contents:
package util
import (
"crypto/md5"
"encoding/hex"
)
func EncodeMD5(value string) string {
m := md5.New()
m.Write([]byte(value))
return hex.EncodeToString(m.Sum(nil))
}
First step
We have already packaged the underlying method previously, and this step is actually the processing logic for packaging image.
Create a new upload/image.go file under the pkg directory and write the contents of the file:
package upload
import (
"os"
"path"
"log"
"fmt"
"strings"
"mime/multipart"
"github.com/EDDYCJY/go-gin-example/pkg/file"
"github.com/EDDYCJY/go-gin-example/pkg/setting"
"github.com/EDDYCJY/go-gin-example/pkg/logging"
"github.com/EDDYCJY/go-gin-example/pkg/util"
)
func GetImageFullUrl(name string) string {
return setting.AppSetting.ImagePrefixUrl + "/" + GetImagePath() + name
}
func GetImageName(name string) string {
ext := path.Ext(name)
fileName := strings.TrimSuffix(name, ext)
fileName = util.EncodeMD5(fileName)
return fileName + ext
}
func GetImagePath() string {
return setting.AppSetting.ImageSavePath
}
func GetImageFullPath() string {
return setting.AppSetting.RuntimeRootPath + GetImagePath()
}
func CheckImageExt(fileName string) bool {
ext := file.GetExt(fileName)
for _, allowExt := range setting.AppSetting.ImageAllowExts {
if strings.ToUpper(allowExt) == strings.ToUpper(ext) {
return true
}
}
return false
}
func CheckImageSize(f multipart.File) bool {
size, err := file.GetSize(f)
if err != nil {
log.Println(err)
logging.Warn(err)
return false
}
return size <= setting.AppSetting.ImageMaxSize
}
func CheckImage(src string) error {
dir, err := os.Getwd()
if err != nil {
return fmt.Errorf("os.Getwd err: %v", err)
}
err = file.IsNotExistMkDir(dir + "/" + src)
if err != nil {
return fmt.Errorf("file.IsNotExistMkDir err: %v", err)
}
perm := file.CheckPermission(src)
if perm == true {
return fmt.Errorf("file.CheckPermission Permission denied src: %s", src)
}
return nil
}
Here we have implemented 7 methods, as follows:
- GetImageFullUrl: get the full picture access URL
- GetImageName: get the picture name
- GetImagePath: get the picture path
- GetImageFullPath: get the full path of the picture
- CheckImageExt: check the picture suffix
- CheckImageSize: check the image size
- CheckImage: check picture
This is basically a secondary encapsulation of the underlying code. In order to handle some picture-specific logic more flexibly and to facilitate modification, the underlying code is not directly exposed to the outside world.
Second step
This step will write the business logic for uploading pictures, create a new upload.go file under the routers/api directory, and write the contents of the file:
package api
import (
"net/http"
"github.com/gin-gonic/gin"
"github.com/EDDYCJY/go-gin-example/pkg/e"
"github.com/EDDYCJY/go-gin-example/pkg/logging"
"github.com/EDDYCJY/go-gin-example/pkg/upload"
)
func UploadImage(c *gin.Context) {
code := e.SUCCESS
data := make(map[string]string)
file, image, err := c.Request.FormFile("image")
if err != nil {
logging.Warn(err)
code = e.ERROR
c.JSON(http.StatusOK, gin.H{
"code": code,
"msg": e.GetMsg(code),
"data": data,
})
}
if image == nil {
code = e.INVALID_PARAMS
} else {
imageName := upload.GetImageName(image.Filename)
fullPath := upload.GetImageFullPath()
savePath := upload.GetImagePath()
src := fullPath + imageName
if ! upload.CheckImageExt(imageName) || ! upload.CheckImageSize(file) {
code = e.ERROR_UPLOAD_CHECK_IMAGE_FORMAT
} else {
err := upload.CheckImage(fullPath)
if err != nil {
logging.Warn(err)
code = e.ERROR_UPLOAD_CHECK_IMAGE_FAIL
} else if err := c.SaveUploadedFile(image, src); err != nil {
logging.Warn(err)
code = e.ERROR_UPLOAD_SAVE_IMAGE_FAIL
} else {
data["image_url"] = upload.GetImageFullUrl(imageName)
data["image_save_url"] = savePath + imageName
}
}
}
c.JSON(http.StatusOK, gin.H{
"code": code,
"msg": e.GetMsg(code),
"data": data,
})
}
Error codes involved (to be added at pkg/e/code.go, msg.go):
// 保存图片失败
ERROR_UPLOAD_SAVE_IMAGE_FAIL = 30001
// 检查图片失败
ERROR_UPLOAD_CHECK_IMAGE_FAIL = 30002
// 校验图片错误,图片格式或大小有问题
ERROR_UPLOAD_CHECK_IMAGE_FORMAT = 30003
In this large section of business logic, we have done the following:
- C.Request.FormFile: get the uploaded picture (return the first file of the provided form key)
- CheckImageExt, CheckImageSize check picture size, check picture suffix
- CheckImage: check the required (permissions, folders) for uploading pictures.
- SaveUploadedFile: save picture
In general, it is the application process saved by “enter the parameter-> check-“
Third step
Open the routers/router.go file to add routesr.POST("/upload", api.UploadImage)
For example:
func InitRouter() *gin.Engine {
r := gin.New()
...
r.GET("/auth", api.GetAuth)
r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))
r.POST("/upload", api.UploadImage)
apiv1 := r.Group("/api/v1")
apiv1.Use(jwt.JWT())
{
...
}
return r
}
Verification
Finally, we request the interface to upload pictures and test the functions written.
Check whether the directory contains files (pay attention to permissions)
$ pwd
$GOPATH/src/github.com/EDDYCJY/go-gin-example/runtime/upload/images
$ ll
... 96a3be3cf272e017046d1b2674a52bd3.jpg
... c39fa784216313cf2faa7c98739fc367.jpeg
Here we have returned a total of 2 parameters, one is the complete access URL and the other is the save path
File server
After completing the previous section, we also need to enable the front end to access the picture, which is generally as follows:
- CDN
- http.FileSystem
In the case of companies, CDN or self-built distributed file systems are in the majority and do not need too much attention. In practice, it must be built locally, Go itself has a good support for this, and Gin is another layer of encapsulation, only need to add a line of code in the route
r.StaticFS
Open the routers/router.go file to add routesr.StaticFS("/upload/images", http.Dir(upload.GetImageFullPath()))
For example:
func InitRouter() *gin.Engine {
...
r.StaticFS("/upload/images", http.Dir(upload.GetImageFullPath()))
r.GET("/auth", api.GetAuth)
r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))
r.POST("/upload", api.UploadImage)
...
}
What did it do
When accessing $HOST/upload/images, files under $ gopath/src/github.com/eddycjy/go-gin-example/runtime/upload/images will be read.
What does this line of code do? Let’s look at the prototype method
// StaticFS works just like `Static()` but a custom `http.FileSystem` can be used instead.
// Gin by default user: gin.Dir()
func (group *RouterGroup) StaticFS(relativePath string, fs http.FileSystem) IRoutes {
if strings.Contains(relativePath, ":") || strings.Contains(relativePath, "*") {
panic("URL parameters can not be used when serving a static folder")
}
handler := group.createStaticHandler(relativePath, fs)
urlPattern := path.Join(relativePath, "/*filepath")
// Register GET and HEAD handlers
group.GET(urlPattern, handler)
group.HEAD(urlPattern, handler)
return group.returnObj()
}
First, the use of * and: symbols is prohibited in the exposed URLcreateStaticHandler
After creating the static file service, the actual final call is stillfileServer.ServeHTTP
And some processing logic
func (group *RouterGroup) createStaticHandler(relativePath string, fs http.FileSystem) HandlerFunc {
absolutePath := group.calculateAbsolutePath(relativePath)
fileServer := http.StripPrefix(absolutePath, http.FileServer(fs))
_, nolisting := fs.(*onlyfilesFS)
return func(c *Context) {
if nolisting {
c.Writer.WriteHeader(404)
}
fileServer.ServeHTTP(c.Writer, c.Request)
}
}
http.StripPrefix
We can watch outfileServer := http.StripPrefix(absolutePath, http.FileServer(fs))
This statement is very common in static file services. What is its function?
http.StripPrefix
The main function is to remove the given prefix from the path of the request URL and finally return oneHandler
Usually http.FileServer is used in combination with http.StripPrefix, otherwise when you run:
http.Handle("/upload/images", http.FileServer(http.Dir("upload/images")))
You will not be able to correctly access the file directory because/upload/images
Also included in the URL path, must use:
http.Handle("/upload/images", http.StripPrefix("upload/images", http.FileServer(http.Dir("upload/images"))))
/*filepath
As you can see belowurlPattern := path.Join(relativePath, "/*filepath")
,/*filepath
Who are you, what are you doing here, are you the product of Gin?
The processing logic of the route can be known through semantics, while Gin’s route is based on httprouter, and the following information can be obtained through consulting documents
Pattern: /src/*filepath
/src/ match
/src/somefile.go match
/src/subdir/somefile.go match
*filepath
All file paths will be matched and*filepath
Must be at the end of the Pattern
Verification
Re-executiongo run main.go
To access the picture address just obtained in the upload interface and check whether http.FileSystem is normal
Modify article interface
Next, you are required to modify the two interfaces of routers/api/v1/article.go: AddArticle and EditArticle
- Add and update the article interface: support to enter cover_image_url
- Add and update article interface: add non-empty and longest length check for cover_image_url
As mentioned in the previous article, if you have any questions, please refer to the code of the project.
Summary
In this chapter, we simply analyzed the following requirements and made a small plan and implementation for the application.
After completing the function points and optimization in the list, it is also a common scene in actual projects. I hope you can taste them carefully and learn from them deeply.
References
This series of sample codes
This series of catalogues
- Serial One Golang Introduction and Environmental Installation
- Serialized 2 to build Blog API’s (1)
- Serial 3 to build Blog API’s (2)
- Serial 4 to build Blog API’s (3)
- Serial 5 Use JWT for Identity Verification
- Serial 6 Write a Simple File Log
- Serial Seven Golang Gracefully Restart HTTP Service
- Serial 8 Add Swagger to It
- Serial 9 Deploying Golang Applications to Docker
- Serial Ten Customized GORM Callbacks
- Serial Eleven Cron Scheduled Tasks
- Serial 12 Optimizing Configuration Structure and Realizing Picture Upload
- Serialization 13 Optimize Your Application Structure and Implement Redis Cache
- Serial 14 Realize Export and Import into Excel
- Serial 15 Generate Two-dimensional Code and Merge Posters
- Serial 16 Draw Text on Pictures
- Serial 17 Deploying Go Applications with Nginx
- Cross-compilation of safari Golang
- Please get started with Makefile