![[234]멀티테넌트 하둡 클러스터 운영 경험기](https://cdn.slidesharecdn.com/ss_thumbnails/234-171017024419-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Cutting the Kubernetes Monorepo in pieces – never learnt more about git (20)
Ad
More from Stefan Schimanski (15)
Ad
Recently uploaded (20)
Cutting the Kubernetes Monorepo in pieces – never learnt more about git
- 1. Cutting the Kubernetes Monorepo in pieces Never learnt more about git Dr. Stefan Schimanski – @the_sttts – github.com/sttts [email protected]
- 3. „My other computer is a datacenter“
- 4. • Founded in 2014 by Google engineers • 100% OpenSource, Apache license • Governed by Cloud Native Computing Foundation (CNCF) • > 100 companies contributing Go
- 5. Monorepo + makes testing easier + makes releases easier + makes global changes easier + is easy to implement - is not how Golang works - is not what 3rd-parties expect - does not scale well github.com/kubernetes/kubernetes cmd pkg staging src k8s.io api apimachinery apiserver client-go apiextensions-apiserver kube-aggregator sample-apiserver sample-controller
- 6. Monorepo + makes testing easier + makes releases easier + makes global changes easier + is easy to implement - is not how Golang works - is not what 3rd-parties expect - does not scale well github.com/kubernetes/kubernetes cmd pkg staging src k8s.io api apimachinery apiserver client-go apiextensions-apiserver kube-aggregator sample-apiserver sample-controller vendor ... symlinks
- 7. Monorepo + makes testing easier + makes releases easier + makes global changes easier + is easy to implement - is not how Golang works - is not what 3rd-parties expect - does not scale well 😱 github.com/kubernetes/kubernetes cmd pkg staging src k8s.io api apimachinery apiserver client-go apiextensions-apiserver kube-aggregator sample-apiserver sample-controller vendor ... symlinks
- 8. github.com/kubernetes/kubernetes cmd pkg api cmd pkg apiserver cmd pkg apimachinery cmd pkg client-go cmd pkg aggregator cmd pkg apiextensions cmd pkg Publishing staging repos + automated, nightly + git filter-branch –subdirectory-filter - preserve history - tag across repos - always work 😱 preserve Github PR merge commits 😱 update dependencies: Godeps.json 😱 work with arbitrary merges 😱 some files live in staging repo Godeps.json, README.md, vendor/
- 9. commit author committer parents tree message $ git cat-file –p HEAD tree 4a541f0a3b26ab7ef8eb38090072558cd8aa54cd parent 3667154d8524e58db0e21b9338f335f5c53dd83e parent 951d2c060e0fa173f82bc221e388f9491709c52c author Kubernetes Submit Queue <[email protected] committer GitHub <[email protected]> 1508409674 -0700 Merge pull request #54192 from mkumatag/versioned_busybox message commit author committer parents tree message commit author committer parents
- 10. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash tree C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 11. git plumbing commands $ git hash-object –w <filename> # Copy into repository <blob-sha> $ git update-index --add --cacheinfo 100644 <blob-sha> <filename> $ git write-tree <tree-sha> $ git commit-tree -p <parent-sha> -m “Commit message“ <tree-sha> <commit-sha>
- 13. PR merge PR merge PR merge PR merge commit commit commit commit PR merge commit commit ... master
- 14. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash client-go C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 15. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash client-go C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 16. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash client-go C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 17. commit author committer parents tree message message commit author committer parents client-go C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“
- 18. commit author committer parents tree message message commit author committer parents client-go C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“
- 19. message commit author committer parents client-go C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „prune empty“ „subdirectory filter“
- 23. Merge pull request #4 author committer parents tree message message commit author committer parents tree C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“ „fast-forward merge“
- 24. Merge pull request #4 author committer parents tree message message commit author committer parents tree C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“ „fast-forward merge“ 😱
- 27. git filter-branch –subdirectory-filter staging/src/k8s.io/client-go instead: custom filter-branch with https://github.com/src-d/go-git „A highly extensible Git implementation in pure Go“ which prunes exactly the commits we want.
- 28. Lessons learnt • Git data structures are no rocket science • Choose: 1. 700 lines custom Git code in Go 2. 700 dense bash lines working around git filter-branch and friends • porcelain vs. plumbing commands Do not script the former, understand what they do!
- 29. References • Git Internals: https://git-scm.com/book/en/v2/Git-Internals-Git- Objects • Native (non-cgo) go library: https://github.com/src-d/go-git • Custom filter-branch algorithm: https://github.com/kubernetes/test- infra/pull/5035/commits/c659f8a49c02ec60bfecfbf1c87416e11e966 4c7