Making Wallstreet talk with GO (GO India Conference 2015)
2 likes1,066 views
Matthew Campbell discusses his experience running Go services with 300k users at Go Meetup in Bangkok, focusing on monitoring large-scale Go servers and managing remote teams. He addresses challenges such as garbage collection issues, latency-sensitive applications, and continuous delivery pipelines using Jenkins. The talk also covers plugin architecture, failover strategies, and ongoing development efforts including the removal of CGO code and the use of Go generate for configuration.
Making Wallstreet talk with GO (GO India Conference 2015)
- 1. Making WallStreet talk with GO Matthew Campbell
- 2. Who Am I • New York / Bangkok • Run GO Meetup in Bangkok • bloomberg.com (Rails) • gucci.com (Rails) • Co founder ErrPlane (GO) • Thomson Reuters Messenger (GO)
- 4. What is this talk about? • Running GO with a 300k users in production for multiple years • Monitoring Large scale GO Servers • Remote teams and GO
- 6. Eikon Messenger • 300k Traders / Fund Managers • Federation with 20+ Bank/Financial Networks • Strict compliance rules in 100+ countries • Global usage, largest in SE Asia • Downtime == Lost Trades == Upset customers
- 7. Protocols • XMPP (Most XEPS) • SIP (Microsoft SIP) • BOSH (http xmpp) • Chatrooms
- 8. Team • 80% remote team • Development New York/Canada/Portugal/Germany/Bangkok • Ops London • QA Bangkok
- 9. Remote Dev • Daily Google Hangouts • Pivotal Tracker • Github pull request, code reviews • Xmpp Chatroom, all day multi time zone • Email group
- 10. Issues with GO
- 11. Missing Features • Decent XML Dom Parser • Regex with decent performance • Serializing performance / allocations • Better CGO integration
- 12. Garbage Collection • Our single largest problem. Our app is latency sensitive. • We have automated test suite to detect garbage collection problems • Avoiding Heap allocations • Managing Size of Heap
- 13. IO Latency • Many Layers • In process cache • Local Redis instance on machine • Mysql primary storage • NSQ for delayed message queuing
- 14. IO Architecture
- 16. CGO Issues • Libxml2 (Gokogiri) , PCRE (Perl Regex) • Issues with to many Pthreads • 1.4 broke a lot of C libraries, with stack moving • Broke GDB for a while • Faster sometimes, sometimes significantly slower cause of thread dispatch
- 17. Following GO Libraries • For the first year, we tracked all GO dependencies • We often time found bugs in upstream libraries with our automated integration suite • Over time this got to risky, and we chose to freeze our GO Libraries
- 18. Tracking GO Lang • PAINLESS ! • We track Go Lang official releases, within 2 weeks of each release we have gone to production using the new compiler • We have had a few integration tests break during upgrades • Only production code that broke, was in CGO ;(
- 19. Continuous Delivery • Jenkins • Integration level tests, that fire up a cluster of our servers, and validates user level interactions at protocol level. Including federating with several open source messaging servers • Unit tests for functional areas
- 20. Plugin Architecture • Plugins are processes • Go doesn’t support dynamic linking • Plugins talk to main process via TCP • Plugins can be run in process, but still use tcp, to simulate exact same properties
- 22. Metrics • Annotate everything • Us GO-Metrics • Graphite / InfluxDb
- 24. Single Points of Failure • Mysql Multimaster • Redis Master / Slave • Sentinel for Mysql Failover • Custom scripts + ETCD for Mysql Failover
- 25. Packaging • RPMs • FPM Ruby Gem • Single binary + Configs
- 26. Managing Config • ETCD vs Zookeeper • Push • Pull
- 27. Compliance • NSQ • Elastic Search • Mysql • C# process
- 29. Upgrades • In place upgrades • Spawn a new process • Pass file handles for TCP via a Named Pipe • New process kills parent
- 30. What’s Next • Multiple live datacenters, currently Hot/Cold • Removal of all CGO Code • Using GO Generate for config classes
- 31. Q & A
- 32. Contact • Feel Free to email me any further questions • We’re Hiring! • [email protected]