Easy coding | 中文
This repo contains an example structure for a monolithic Go Web Application.
Project Architecture
This project loosely follows Uncle Bob's Clean Architecture.
Project principle
Features
- 100% API defined by protobuf
- Auto generate grpc, grpc gateway, validate go files
- Provide both rest api and grpc api
- Auto generate swagger api document
- Breaking change detection(WIP)
- Builtin prometheus metrics
- Support import api definition by postman
- Run in docker
- Auto configuration generate
- Database migrate up and down
- Database mock testing
- Golang, Protobuf and basic text file linting
- Error definition and classification
- Auto logging and pretty format
- Unit test and test coverage
- Graceful stop
- Backend processes
- Health check
Prerequest
-
protoc plugins, go, grpc, grpc-gateway, openapi, validate
go install \
github.com/grpc-ecosystem/grpc-gateway/v2/[email protected] \
github.com/grpc-ecosystem/grpc-gateway/v2/[email protected] \
google.golang.org/protobuf/cmd/[email protected] \
google.golang.org/grpc/cmd/protoc-gen-go-gr[email protected] \
github.com/envoyproxy/[email protected]
-
golang 1.18+
-
docker and docker compose
-
(optional) pre-commit
pip3 install pre-commit
pre-commit install
- (optional) golang lint
go install github.com/golangci/golangci-lint/cmd/[email protected]
Getup and running
Modify the GOPROXY env in the dockerfile, when the download speed is slow
make deps
make run
The following files will be generated
- api/{module_name}/{module_name}.pb.go
- api/{module_name}/{module_name}.pb.validate.go
- api/{module_name}/{module_name}.pb.swagger.json
- api/{module_name}/rpc_grpc.pb.go
- api/{module_name}/rpc.pb.go
- api/{module_name}/rpc.pb.gw.go
- api/{module_name}/rpc.pb.validate.go
- api/{module_name}/rpc.swagger.json
There are three exported ports
- 10000: rest api server
- 10001: grpc api server
- 10002: swagger api and prometheus server
Check rest api server
curl http://localhost:10000/ping
Check grpc api server
go run cmd/client/main.go
Open the following url in the browser
Topic1 API management
Motivation
Api is an abstract concept, is language independent, in many cases, there are many files to describe an api
- Some struct/class in golang/java files
- Some class in typescript/javascript files
- Swagger/Openapi
- Readable document
That violate the Single source of truth principle, it should define the api in one place, and generate other files.
Get started
In this topic, we will add a new greet service
api/greet_apis/greet/greet.proto
syntax = "proto3";
package greet;
option go_package = 'easycoding/api/greet';
message HelloRequest {
string req = 1;
}
message HelloResponse {
string res = 1;
}
api/greet_apis/greet/rpc.proto
syntax = "proto3";
package greet;
option go_package = 'easycoding/api/greet';
import "google/api/annotations.proto";
import "greet/greet.proto";
// The greet service definition.
service GreetSvc {
rpc Hello(HelloRequest) returns (HelloResponse) {
option (google.api.http) = {
get: "/hello",
};
}
}
api/greet_apis/buf.yaml
version: v1
breaking:
use:
- FILE
lint:
use:
- DEFAULT
api/buf.work.yaml
- payment_apis
- pet_apis
- ping_apis
# add new line
- greet_apis
Run make gen-api in the workspace, the following files will be generated
api/greet/greet.pb.go
api/greet/greet.pb.validate.go
api/greet/greet.swagger.json
api/greet/rpc_grpc.pb.go
api/greet/rpc.pb.go
api/greet/rpc.pb.gw.go
api/greet/rpc.pb.validate.go
api/greet/rpc.swagger.json
api/api.swagger.json
Implement the greet service
internal/service/greet/service.go
package greet
import (
"context"
greet_pb "easycoding/api/greet"
"github.com/sirupsen/logrus"
)
type service struct{}
var _ greet_pb.GreetSvcServer = (*service)(nil)
func New(logger *logrus.Logger) *service {
return &service{}
}
func (s *service) Hello(
ctx context.Context,
req *greet_pb.HelloRequest,
) (*greet_pb.HelloResponse, error) {
return &greet_pb.HelloResponse{Res: req.Req}, nil
}
Update internal/service/register.go
var endpointFuns = []RegisterHandlerFromEndpoint{
ping_pb.RegisterPingSvcHandlerFromEndpoint,
pet_pb.RegisterPetStoreSvcHandlerFromEndpoint,
// add new line
greet_pb.RegisterGreetSvcHandlerFromEndpoint,
}
func RegisterServers(grpcServer *grpc.Server, logger *logrus.Logger, db *gorm.DB) {
ping_pb.RegisterPingSvcServer(grpcServer, ping_svc.New(logger))
pet_pb.RegisterPetStoreSvcServer(grpcServer, pet_svc.New(logger, db))
// add new line
greet_pb.RegisterGreetSvcServer(grpcServer, greet_svc.New())
}
Run the server
make run
Check the rest http server
curl localhost:10000/hello?req=hi
The new api document is in http://localhost:10002/swagger/, if you want to custom the output of openapi see protoc-gen-openapi for more information.
message MyMessage {
// This comment will end up direcly in your Open API definition
string uuid = 1 [(grpc.gateway.protoc_gen_openapiv2.options.openapiv2_field) = {description: "The UUID field."}];
}
The new metrics is in http://localhost:10002/metrics, you can use prometheus-client to custom metrics, see go-grpc-prometheus for example.
customizedCounterMetric = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "demo_server_say_hello_method_handle_count",
Help: "Total number of RPCs handled on the server.",
}, []string{"name"})
You can add some validate to your api like following
syntax = "proto3";
package greet;
option go_package = 'easycoding/api/greet';
// add new line
import "validate/validate.proto";
message HelloRequest {
// add validate
string req = 1[(validate.rules).string = {min_len: 0, max_len: 10}];
}
message HelloResponse {
string res = 1;
}
Stop the server and run make gen-api and make run again.
Send the following request and will return error, see protoc-gen-validate for more validate rule.
curl localhost:10000/hello?req=hiiiiiiiiii
Validator is checked request in the grpc middleware, you can find some common middlewares in grpc-middleware, or you can custom your own middleware.
Breaking change detection
cd api/pet_apis
buf breaking --against "../../.git#branch=master,subdir=api/pet_apis"
Incompatible changes
// api/pet_apis/pet/pet.proto
message Pet {
int32 pet_id = 1;
string name = 2;
// change the following type
// PetType pet_type = 3;
string pet_type = 3;
}
Check the compatibility
cd api/pet_apis
buf breaking --against "../../.git#branch=master,subdir=api/pet_apis"
pet/pet.proto:22:5:Field "3" on message "Pet" changed type from "enum" to "string".
Compatible changes
// api/pet_apis/pet/pet.proto
message Pet {
int32 pet_id = 1;
string name = 2;
PetType pet_type = 3;
// add the following field
string address = 4;
}
cd api/pet_apis
buf breaking --against "../../.git#branch=master,subdir=api/pet_apis"
Topic2 Database migrate
Motivation
Write raw sql to operate database is not easy to maintain, we use ORM to interact with database, Gorm for this project. Another situation we encounter is that we often upgrade the structure of the database, firstly, in many compaines, people who write the code and deploy applications are different, so it is necessary to manage upgrade and downgrade properly, secondly we can intergrate sql files into intergration test to ensure the correctness of database structure, thirdly, it is hard to write up and down sql file manully.
Get started
For the current time, the database test is totally empty, use the following command to create auto migration sql files
make migrate-create
The following files will be generated, see migrate for more information
migrations/pet/{timestamp}_pet.up.sql
migrations/pet/{timestamp}_pet.down.sql
Migrate sql to database, in the cloud native scenario, you usually need to start a kubernetes job to migrate the database, so the command is not intergrate with Makefile.
go run cmd/migrate/main.go step --latest
INFO[0000] Start buffering 20220723144816/u pet
INFO[0000] Read and execute 20220723144816/u pet
INFO[0000] Finished 20220723144816/u pet (read 5.465976ms, ran 57.983119ms)
Migrate successful, use describe pet to check the schema of table pet
+------------+----------+------+-----+-------------------+-------------------+
| Field | Type | Null | Key | Default | Extra |
+------------+----------+------+-----+-------------------+-------------------+
| id | int | YES | | NULL | |
| name | text | YES | | NULL | |
| type | int | YES | | NULL | |
| created_at | datetime | YES | | CURRENT_TIMESTAMP | DEFAULT_GENERATED |
+------------+----------+------+-----+-------------------+-------------------+
4 rows in set (0.01 sec)
Update pkg/orm/pet.go
--- a/pkg/orm/pet.go
+++ b/pkg/orm/pet.go
@@ -12,6 +12,7 @@ type Pet struct {
Name string
// TODO(qujiabao): replace int32 to pet_pb.PetType, because of `sqlize`
Type int32
+ Age int32
CreatedAt time.Time `gorm:"default:now()"`
}
Create migration files and two files will be generated, and there are four files in migrations/pet
make migrate-create
Step up
go run cmd/migrate/main.go step --latest
Check the current version of database
go run cmd/migrate/main.go version
Version: 20220723150428, Dirty: false
+------------+----------+------+-----+-------------------+-------------------+
| Field | Type | Null | Key | Default | Extra |
+------------+----------+------+-----+-------------------+-------------------+
| id | int | YES | | NULL | |
| name | text | YES | | NULL | |
| type | int | YES | | NULL | |
| age | int | YES | | NULL | |
| created_at | datetime | YES | | CURRENT_TIMESTAMP | DEFAULT_GENERATED |
+------------+----------+------+-----+-------------------+-------------------+
5 rows in set (0.00 sec)
Downgrade the database version
go run cmd/migrate/main.go step 1 --reverse
Version: 20220723144816, Dirty: false
+------------+----------+------+-----+-------------------+-------------------+
| Field | Type | Null | Key | Default | Extra |
+------------+----------+------+-----+-------------------+-------------------+
| id | int | YES | | NULL | |
| name | text | YES | | NULL | |
| type | int | YES | | NULL | |
| created_at | datetime | YES | | CURRENT_TIMESTAMP | DEFAULT_GENERATED |
+------------+----------+------+-----+-------------------+-------------------+
4 rows in set (0.00 sec)
Topic3 Linter
Motivation
- Cooperate in a same pattern
- Finds bugs during early stages of development
- Coding style as code
- Define rules to assist developers
| Bad | Good |
|---|---|
func (d *Driver) SetTrips(trips []Trip) {
d.trips = trips
}
trips := ...
d1.SetTrips(trips)
// Did you mean to modify d1.trips?
trips[0] = ...
|
func (d *Driver) SetTrips(trips []Trip) {
d.trips = make([]Trip, len(trips))
copy(d.trips, trips)
}
trips := ...
d1.SetTrips(trips)
// We can now modify trips[0] without affecting d1.trips.
trips[0] = ...
|
Alignment
| Bad | Good |
|---|---|
// 12 bytes
type Foo struct {
aaa bool
bbb int32
ссс bool
}
|
// 8 bytes
type Foo struct {
aaa bool
ссс bool
bbb int32
}
|
For more information see Uber golang style guide
Get started
make lint
Topic4 Error handling
Concept
Error vs Exception
Errors are the possible problems in program, is part of bussiness, like database connection failed.
Exception is unexpectable problems in program, is not part of bussiness, like nil pointer exception, array outof range.
Guideline
┌───────────────┐
│ │
│ handle error │
│ │
└───────────────┘
▲
│
┌───────┴───────┐
│ │
│ with message │
│ │
└───────────────┘
▲
│
┌───────┴───────┐
│ │
│ wrap error │
│ │
└───────────────┘
▲
│
┌───────┴───────┐
│ │
│ raw error │
│ │
└───────────────┘
Example
// pkg/orm/pet.go
func (pet *Pet) GetPet(db *gorm.DB, id int32) error {
// err is a raw err from gorm
if err := db.Take(pet, "id = ?", id).Error; err != nil {
// check the type of raw error
if errors.ErrorIs(err, gorm.ErrRecordNotFound) {
// wrap error
return errors.ErrNotFound(err)
}
// wrap error
return errors.ErrInternal(err)
}
return nil
}
// internal/service/pet/get_pet.go
func (s *service) getPet(
ctx context.Context,
req *pet_pb.GetPetRequest,
) (*pet_pb.GetPetResponse, error) {
pet := &orm.Pet{}
if err := pet.GetPet(s.DB, req.PetId); err != nil {
// with message in service
return nil, errors.WithMessage(err, "get pet failed")
}
}
// internal/middleware/log/interceptor.go
// Describe how to log error
func Interceptor(logger *logrus.Logger) func(
ctx context.Context,
req interface{},
_ *grpc.UnaryServerInfo,
handler grpc.UnaryHandler) (interface{}, error) {
ops := []grpc_logrus.Option{
// map to log level
grpc_logrus.WithLevels(levelFunc),
// add decider
grpc_logrus.WithDecider(decider),
}
entry := logrus.NewEntry(logger)
logInterceptorBefore := createBeforeInterceptor(entry)
logInterceptorAfter := createAfterInterceptor(entry)
return grpc_middleware.ChainUnaryServer(
logInterceptorBefore,
grpc_logrus.UnaryServerInterceptor(entry, ops...),
logInterceptorAfter,
)
}
// internal/middleware/error/interceptor.go
// error classification
func Interceptor(logger *logrus.Logger) grpc.UnaryServerInterceptor {
return func(
ctx context.Context,
req interface{},
reqinfo *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
res, err := handler(ctx, req)
if err == nil {
return res, err
}
var code codes.Code
switch {
case errors.ErrorIs(err, errors.InternalError):
code = codes.Internal
case errors.ErrorIs(err, errors.InvalidError):
code = codes.InvalidArgument
case errors.ErrorIs(err, errors.NotFoundError):
code = codes.NotFound
case errors.ErrorIs(err, errors.PermissionError):
code = codes.PermissionDenied
case errors.ErrorIs(err, errors.UnauthorizedError):
code = codes.Unauthenticated
default:
logger.WithError(err).WithField("method", reqinfo.FullMethod).
Warn("invalid err, without using easycoding/pkg/errors")
return res, err
}
s := status.New(code, err.Error())
return res, s.Err()
}
}
Topic5 Configuration
Motivation
Configuration is an abstract concept, language independent, in many cases, we describe configuration is many files
- yaml file
- json
- golang struct
That violate the Single source of truth principle, it should define the api in one place, and generate other files.
Configuration should combine the following sources and bind to struct
- explicit call
Set - command line flag
- env
- file
- default
Topic6 Unit test and coverage
Get started
Run all tests
make test
Run all tests with coverage
make coverage
Run all tests with coverage and open the report in browser
make coverage-html
Topic7 Release and deploy
Topic8 Monitor, logging and trace
TODO
- Use reflect in configration
- Benchmark
- Fix linting
- Intergration test
- Auth
- More options in configuration
- Property based test
- GraphQL server
- Use go generate to refactor proto and bindata generation
Inspirations
- https://github.com/OFFLINE-GmbH/go-webapp-example
- https://github.com/golang-standards/project-layout
