Building gRPC services
- 1. Building gRPC services With Go and Bazel
- 2. Backend Stack at TextNow • Build microservices using the Go programming language • Go pairs well with gRPC • Easy to create new networked services that are able to scale • Use a monorepo for all Go microservices • Easy to spin up new microservices • Take advantage of a common set of build & release tools • Linters, formatters, build rules, and more. • Use the gRPC framework for inter-service communication • Write clear, agreed upon contracts between our microservices • Improved the level of documentation for our internal microservices.
- 3. Backend Stack at TextNow • Began adopting gRPC/Go 2 years ago • Our monorepo now has: • roughly 3000 commits • 34 contributors • roughly 3 dozen microservice daemons • roughly 2 dozen CLI tools • We have learned a lot along the way!
- 4. Version 1.0 • textnow-mono’s first commit: • a tiny gRPC API • a daemon implementing the service • a CLI tool • The actual logic in the daemon wasn't significant • It provided a playground to try new things • Project structure? • Build tooling? • Proto generation?
- 5. Version 1.0 Project Structure • proto/ • test_service.proto • services/ • service name (for example, testService) • pb/ • test_service.pb.go • api.go • Other .go files exist that offer the required business logic used by the API type • cmd/ • testServiced/ • main.go • testServiceCli/ • main.go
- 6. Version 1.0 Proto Structure • All files in 1 directory • Each file defined it’s own namespace • Common types were imported across multiple protos • The option go_package directive was specified • Gave us control over mapping protos to services • Each service lived in the same package name • We called it ‘pb’ • It allowed us to distinguish auto-generated types from manually written types • Proto code was moved to the right package manually
- 7. Version 1.0 Build System • Script to auto-generate proto files • Manually run `go build …` or `go run <package>`
- 8. Version 1.0 Lessons Learned • Putting all the proto files into a single directory isn't a good idea! • Differentiating between public interfaces and private interfaces was very difficult. • A directory full of proto files didn't make it clear which were common types • Generating types in a separate package than the service adds unnecessary complexity. • Service definitions from different pb packages led to a lot of issues • Duplicate naming between manually written and proto-specified types is confusing • Can’t extend auto-generated types • Manual steps will be forgotten. • Edit the proto file • Build the .pb.go files • Copy them into the appropriate subdirectory • Leads to skew between what is committed in the proto file and what is built
- 9. Version 2.0 • Major goals of v2.0 • Ease of development • Support for multiple build targets (binary, Docker image, etc.) • Support for gRPC build targets • Better build times (caching, parallelism, etc.) • Consistent builds across platforms • OS X and Linux
- 10. Version 2.0 - Options • Makefiles � Tried and true � Easy to get started with � Makefiles get really complicated as the project grows � Remote caching is offered by the compiler, not the framework • Bazel � Fast, parallel builds with remote caching and build targets � Supports gRPC and build targets across many languages � Bleeding edge, going through active development � Documentation is sparse
- 11. Version 2.0 – Building with Bazel • We chose to adopt Bazel as our build framework • rules_go specifies the build targets for the Go programming language • Native support for both proto and gRPC build targets • Use Gazelle to create build files • Analyzes proto and Go files and generates build targets automatically • Finds dependencies and updates accordingly • Handles vendored references natively
- 12. Version 2.0 – Adopting Bazel 1. # brew install bazel 2. Create a WORKSPACE and BUILD.bazel file at the root of the project https://github.com/bazelbuild/rules_go has good defaults 3. # bazel run //:gazelle 4. # bazel build //… 5. Success? • In a new project? Yes • In an existing project? Maybe? • In textnow-mono? Definitely not
- 13. Version 2.0 – Adopting Bazel • What broke? • Gazelle does not like multiple proto packages in one target • Gazelle does not generate proto targets out of path • Needed to manually write the build targets for our protos • We had checked in a number of auto-generated targets • Gazelle happily generated build targets for these files • Conflicted with the auto-generated build targets
- 14. Version 2.0 – Project Structure • Changing our thinking about proto files solved our Bazel issues! • Internal gRPC service protos are declared in their service package • No more ‘pb’ packages • Types live where they are used • Can be extended if warranted • Public gRPC service protos are declared in a separate hierarchy • All services are versioned • Version exists in the package name • One or more services may implement the server interface • One service may easily implement multiple versions of the server interface
- 15. Version 2.0 – Project Structure (Internal API) • services/ • service name (for example, testService) • api.go • test_service.proto • Other .go files exist that offer the required business logic used by the API type • cmd/ • testServiced/ • main.go • testServiceCli/ • main.go
- 16. Version 2.0 – Project Structure (Public API) • api/ • test/ • v1/ • test_service.proto • services/ • service name (for example, testService) • api.go • Other .go files exist that offer the required business logic used by the API type • cmd/ • testServiced/ • main.go • testServiceCli/ • main.go
- 17. Version 2.0 – Project Structure • Gazelle generates Go build targets for all protos • Build target has both client and server stubs • Gazelle auto-links these build targets in our services • We manually add build targets for our Dockerized services • Use rules_docker and go_image • We manually add other build targets for public APIs • Bazel supports Java, CC, PHP, etc. • Manually specify other language build targets • Bazel build then auto-generates client dependencies
- 18. Version 2.0 - Example • https://github.com/rmrobinson-textnow/bazel-grpc-example • Public gRPC API with internal service • Service has both a Dockerized daemon and CLI tool • All built with Bazel!
- 19. Version 2.0 - Issues • All proto/gRPC targets are built out of path • Bazel has its own filesystem path for build artifacts • Can’t access these files in IDEs • No autocomplete • Build caches are necessary as projects get larger • Initial clones can take a long time to build • Build everything from scratch • Building for Docker on Mac is awkward • Must supply the --platform tag to Docker build targets • All said Version 2.0 is much better than Version 1.0
- 20. Version 2.1 – API Field Validation • Specify the allowed data on a field, not just the type, in the proto • Easy for both sides of the API to see what should be allowed • https://github.com/mwitkow/go-proto-validators • Generates code for Go • Build type included in Gazelle • Lacks some dependencies making building without vendor tricky • https://github.com/envoyproxy/protoc-gen-validate • Generates code for Go, Java and Python (with C++ in development) • Has Bazel targets for the validator logic and the Go dependencies • Gazelle requires some hacky directives in top-level BUILD.bazel
- 21. Version 2.1 – API Linting • We lint our code, we should also lint our APIs • Identify style, consistency or best-practice violations at commit time • https://github.com/uber/prototool • More than just a linter, a full proto build tool • Has several style guides, with rules for building your own • Lots of development underway • https://github.com/googleapis/api-linter • Enforces the Google style guide • Pretty simple
- 22. Version 2.2 – Auto-generated client libraries • Given a service description, can we generate a full client SDK? • Pre-specified hostnames, ports, etc. • Automatic handling of retries, backoffs, etc. • Applied on a per-route basis! • https://github.com/googleapis/gapic-generator • Still under active development • No support for newer client languages (Swift, Kotlin) • One possible end state – APIs as SDKs