Inside the IT Territory game server / Mark Lokshin (IT Territory)
Download as PPTX, PDF0 likes60 views
In my talk, I will talk about the internal structure of the game server, which support our games of the Vorodezh ITT division. What can be learned from the talk: - what technologies did we choose for the development of game servers (spoiler: Vert.X Hazelcast, Postgres, Kafka, Prometheus + Grafana, Consul, Photon Cloud); - how we use them (spoiler: not all for their intended purpose); - how we install updates; - some interesting mistakes that we caught while working with Vert.X and Hazelcast.
![GAME SHARD
Frontend
Mechanics[N]
Local cache
Hazelcast
Frontend
Mechanics[M]
Local cache
…
=
Shard](https://image.slidesharecdn.com/marklokshin-210615083211/85/Inside-the-IT-Territory-game-server-Mark-Lokshin-IT-Territory-37-320.jpg)
![[Start] Playing](https://cdn.slidesharecdn.com/ss_thumbnails/start-playing-121117211541-phpapp02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (12)
Similar to Inside the IT Territory game server / Mark Lokshin (IT Territory) (20)
![[NetherRealm Studios] Game Studio Perforce Architecture](https://cdn.slidesharecdn.com/ss_thumbnails/game-studio-perforce-architecture-slides-130523181224-phpapp02-thumbnail.jpg?width=560&fit=bounds)
Ad
More from DevGAMM Conference (20)
Ad
Recently uploaded (20)
Inside the IT Territory game server / Mark Lokshin (IT Territory)
- 1. INSIDE THE IT TERRITORY GAME SERVER Mark Lokshin Senior programmer
- 2. WE TALK ABOUT •Technology stack •Update specialty •Game cluster structure •Performance
- 3. • 200+ employees • 15+ successful projects • 100+ millions players Out studio
- 4. IT TERRITORY •Allods Online •Hawk: Freedom Squadron •Space Justice •World Above •Rush Royale
- 6. IN THE BEGINNING THERE WAS SILENCE AND DARKNESS • 2007 year • Pure Java SE • Proprietary protocol • Proprietary API • Hibernate, GWT, Trove collections, Protobuf …
- 7. INTERESTING, BUT NOT EFFECTIVE • Code generation, change byte-code on a fly • No documentation • No best-practice examples • Stack Overflow Driven Development • Need functionality - do it • Hard to support
- 8. WHAT ARE WE USE? • PostgreSQL • Photon Cloud • Kafka • Hazelcast • Vert.X • Prometheus + Grafana
- 10. PHOTON CLOUD • «Rooms» for the game • Datacenters in X point on all continents (penguins offended) • Dynamic hardware reservation • Made for games, but useful not only for games(f.e. – text/sound/video chat)
- 12. APACHE KAFKA • Horizontal scaling • Message persistence • Replication Distributed message broker
- 14. HAZELCAST • Integrate in Vert.X • Cluster from the box • Load balancing • Horizontal scaling • Add/failure node support • Distributed data structures • Queries In memory Data Grid
- 16. VERT.X • Open-source framework for building distributed event-driven applications based on JVM • Verticle – single thread service • Distributed message bus • Asynchrony • Parallelism • Multilanguage
- 17. REACTOR PATTERN
- 18. OPERATION EXECUTOR • Define entities for work (and specify lock level to access) • Take required locks • Retrieve required entities from Hazelcast • Execute operation code • Send changed entities to Hazelcast • Remove locks • Run callback when finish operation
- 19. DEADLOCK WHILE BLOCKING • Operation 1 take lock on A resource • Operation 2 try to take lock on A resource and wait • Operation 1 try to take lock on B resource • … and unfortunately deadlock
- 20. SOMEWHERE INSIDE VERT.X • T1:Lock(A) • T2:Lock(A) • T1:Lock(B) T1:A T2:A T1:B Locks queue
- 21. DEADLOCK WHILE BLOCKING • https://groups.google.com/g/vertx/c/- uvLMuubpz8/m/OB2JzdDdAwAJ • https://github.com/vert-x3/vertx-hazelcast/issues/41
- 22. AUTHORS REASONS The original reason executeBlocking defaults to ordered=true (it's more general than just hazelcast usage) is something like this: Imagine you have a web application and request #1 comes in - this requests inserts some data into a database (e.g. add to shopping basket) Immediately after this requests #2 comes in - this selects the same data from the database (e.g. view shopping basket)
- 23. VERT.X STRESS TESTING • We developed test server, that performs one operation • We developed client, that sends one operation to the server. • Testing performance in different configurations
- 24. • We developed test server, that performs one operation • We developed client, that sends one operation to the server. • Testing performance in different configurations • Everything is very bad… VERT.X STRESS TESTING
- 25. • We developed test server, that performs one operation • We developed client, that sends one operation to the server. • Testing performance in different configurations • Everything is very bad… • DB??? Remove interaction with database VERT.X STRESS TESTING
- 26. • We developed test server, that performs one operation • We developed client, that sends one operation to the server. • Testing performance in different configurations • Everything is very bad… • DB??? Remove interaction with database • Everything is very bad … VERT.X STRESS TESTING
- 27. • We developed test server, that performs one operation • We developed client, that sends one operation to the server. • Testing performance in different configurations • Everything is very bad… • DB??? Remove interaction with database • Everything is very bad … • Continue to test performance in different configurations VERT.X STRESS TESTING
- 28. • We developed test server, that performs one operation • We developed client, that sends one operation to the server. • Testing performance in different configurations • Everything is very bad… • DB??? Remove interaction with database • Everything is very bad … • Continue to test performance in different configurations • Everything is very bad … but sometimes good… VERT.X STRESS TESTING
- 30. «RAISE» INSIDE VERT.X WHEN CREATING A CHANNEL • Code take locks when user create new channel inside Vert.X • If you create many channels at same time – code wait most of time inside locks • We create one channels per user – in case of large users number the creation of channels was very slow • We change channels schema to one channel per message type • We fix channel creation in Vert.X • Create pull request to work Vert.X + Prometheus
- 31. TIM FOX
- 32. UPDATE FEATURES
- 33. GAME UPDATES • Downtime min • Development price min • Revenue max Business requirements
- 34. UPDATE PROCESS • Delaying update availability • Shards with different versions • Unified database • «Soft» update • «Hard» update
- 35. SWITCHING PLAYERS TO THE NEW VERSION
- 37. GAME SHARD Frontend Mechanics[N] Local cache Hazelcast Frontend Mechanics[M] Local cache … = Shard
- 38. PROJECT INFRASTRUCTURE Hazelcast Shard 1 Shard 2 Shard 3 Account Server Kafka ETL Gametool Web Server Gametool Lobby Account
- 39. STRUCTURES IN HAZLECAST Inside game shard (player information) • Player description • Player items • Player quests • Player counters • e.t.c. PlayerI D 1 2 3
- 40. СТРУКТУРЫ В HAZLECAST Общий для шардов (например - турнир) PlayerI D 1 2 3 • Battle history • Battle history • Battle history № bucket 1 2 3 • Players • Scores • Attempts PlayerID № bucket 1 1 2 1 3 2 Players counter 3
- 41. REPEATABLE COUNTER • We made tournament, it works fine for a month, players are registered and playing.
- 42. REPEATABLE COUNTER • We made tournament, it works fine for a month, players are registered and playing. • Ticket from support – users see x2 players inside tournament buckets.
- 43. REPEATABLE COUNTER • We made tournament, it works fine for a month, players are registered and playing. • Ticket from support – users see x2 players inside tournament buckets. • Fix counter
- 44. REPEATABLE COUNTER • We made tournament, it works fine for a month, players are registered and playing. • Ticket from support – users see x2 players inside tournament buckets. • Fix counter • Check code for errors – nothing suspect
- 45. REPEATABLE COUNTER • We made tournament, it works fine for a month, players are registered and playing. • Ticket from support – users see x2 players inside tournament buckets. • Fix counter • Check code for errors – nothing suspect • Remember, that some times before we shutdown old node.
- 46. REPEATABLE COUNTER • We made tournament, it works fine for a month, players are registered and playing. • Ticket from support – users see x2 players inside tournament buckets. • Fix counter • Check code for errors – nothing suspect • Remember, that some times before we shutdown old node. • Read documentation…
- 47. REPEATABLE COUNTER • IAtomicLong have only 2 copy • When 2 copies lay in old nodes for shutdown, there was a problem • Shutdown node with kill -9 – bad idea!
- 48. PERFORMANCE
- 49. 50K ACTIVE USERS • Game Servers: 2x8 2.5GHz Xeon + 32Gb RAM x3 • GameDB Server: 2x4 2.5GHz Xeon + 256Gb RAM • Account Server: 2x4 2.5GHz Xeon + 4Gb RAM
- 50. 50K ACTIVE USERS • Game Servers: 2x8 2.5GHz Xeon + 32Gb RAM x3 • GameDB Server: 2x4 2.5GHz Xeon + 256Gb RAM • Account Server: 2x4 2.5GHz Xeon + 4Gb RAM • AccountDB Server • Kafka • ETL
- 51. SCALING CAPABILITIES • CPU – horizontal scaling hardware in cluster • БД – vanilla PostgreSQL → no sharding → better hardware • Write in IMDG, delayed sync with PostgreSQL • Different shards
- 52. CONCLUSIONS • The technology stack scales well and really works • Errors happens • Universalism is good, but not for all purposes
- 54. THANK YOU!