12-Step Program for Scaling Web Applications on PostgreSQL
119 likes69,149 views
The document discusses scaling a web application called Wanelo that is built on PostgreSQL. It describes 12 steps for incrementally scaling the application as traffic increases. The first steps involve adding more caching, optimizing SQL queries, and upgrading hardware. Further steps include replicating reads to additional PostgreSQL servers, using alternative data stores like Redis where appropriate, moving write-heavy tables out of PostgreSQL, and tuning PostgreSQL and the underlying filesystem. The goal is to scale the application while maintaining PostgreSQL as the primary database.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to 12-Step Program for Scaling Web Applications on PostgreSQL (20)
Ad
Recently uploaded (20)
12-Step Program for Scaling Web Applications on PostgreSQL
- 1. Proprietary and WHILE THE OLD PRESENTATION IS STILL AVAILABLE AFTER THIS SLIDE, I THINK YOU SHOULD SEE THE NEW ONE INSTEAD! PLEASE CLICK ON THE LINK: http://bit.ly/1MXxlBL WARNING: There is a more recent and updated version of this talk โจ available on slideshare, itโs called: โFROM OBVIOUS TO INGENIOUS: โจ INCREMENTALLY SCALING WEB APPLICATIONS ON POSTGRESQLโ presented at
- 2. Proprietary and Con๏ฌdential Konstantin Gredeskoulโจ CTO, Wanelo.com 12-Step Program for Scaling Web Applications on PostgreSQL @kig @kigster
- 3. Proprietary and And why should you care? What does it mean, to scale on top of PostgreSQL?
- 4. Proprietary and Scaling means supporting more work load concurrently, where work is often interchanged with users But why on PostgreSQL? Because NoNoSQL is hawt! (again)
- 5. Proprietary and Relational databases are great at supporting constant change in software They are not as great in โpure scalingโ, like RIAK or Cassandra So the choice critically depends on what you are trying to build
- 6. Proprietary and Large majority of web applications are represented extremely well by the relational model So if I need to build a new product or a service, my default choice would be PostgreSQL for critical data, + whatever else as needed
- 7. Proprietary and This presentation is a walk-through ๏ฌlled with practical solutions Itโs based on a story of scaling wanelo.com to sustain 10s of thousand concurrent users, 3k req/sec Not Twitter/Facebook scale, but stillโฆ So letโs explore the application to learn a bit about wanelo for our scalability journey
- 8. Proprietary and Founded in 2010, Wanelo (โwah-nee-loh,โ from Want, Need, Love) is a community and a social network for all of the world's shopping. Wanelo is a home to 12M products, millions of users, 200K+ stores, and products on Wanelo have been saved into collections โจ over 2B times
- 9. โข move fast with product development โข scale as needed, stay ahead of the curve โข keep overall costs low โข but spend where it matters โข automate everything โข avoid reinventing the wheel โข learn as we go โข remain in control of our infrastructure Early on we wanted to:
- 10. Heroku or Not? Proprietary and Assuming we want full control of our application layer, places like Heroku arenโt a great ๏ฌt But Heroku can be a great place to start. It all depends on the size and complexity of the app we are building. Ours would have been cost prohibitive.
- 11. Foundations of web apps Proprietary and โข app server (we use unicorn) โข scalable web server in front (we use nginx) โข database (we use postgresql) โข hosting environment (we use Joyent Cloud) โข deployment tools (capistrano) โข server con๏ฌguration tools (we use chef) โข programming language + framework (RoR) โข many others, such as monitoring, alerting
- 12. Letโs reviewโฆ Basic Web App Proprietary and /var/pgsql/data incoming http PostgreSQL Server /home/user/app/current/public nginx Unicorn / Passenger Ruby VM N x Unicorns Ruby VM โข no redundancy, no caching (yet) โข can only process N concurrent requests โข nginx will serve static assets, deal with slow clients โข web sessions probably in the DB or cookie
- 13. First optimizations: โจ cheap early on, well worth it Proprietary and โข Personalization via AJAX, so controller actions can be cached entirely using caches_action โข Page returned unpersonalized, additional AJAX request loads personalization
- 14. A few more basic performance tweaks that go a long way Proprietary and โข Install 2+ memcached servers for caching and use Dalli gem to connect to it for redundancy โข Switch to memcached-based web sessions. Use sessions sparingly, assume transient nature โข Setup CDN for asset_host and any user generated content. We use fastly.com โข Redis is also an option, but I prefer memcached for redundancy
- 15. Proprietary and browser PostgreSQL Server /home/user/app/current/public nginx Unicorn / Passenger Ruby VM N x Unicorns Ruby VM memcachedCDN cache images, JS Caching goes a long wayโฆ โข geo distribute and cache your UGC and CSS/JS assets โข cache html and serialize objects in memcached โข can increase TTL to alleviate load, if tra๏ฌc spikes
- 16. Proprietary and Adding basic redundancy โข Multiple app servers require haproxy between nginx and unicorn โข Multiple long-running tasks (such as posting to Facebook or Twitter) require background job processing framework โข Multiple load balancers require DNS round robin and short TTL (dyn.com)
- 17. Proprietary and PostgreSQL Unicorn / Passenger Ruby VM (times N) haproxy incoming http DNS round robin or failover / HA solution nginx memcached redis CDN cache images, JS Load Balancers App Servers single DB Object Store User Generated Content Sidekiq / Resque Background WorkersData stores Transient to Permanent this architecture can horizontally scale up as far the database at itโs center every other component can be scaled by adding more of it, to handle more tra๏ฌc
- 18. Proprietary and As long as we can scale the data store on the backend, we can scale the app! Mostly :) At some point we may hit a limit on TCP/IP network throughput, # of connections, but this is at a whole another scale level
- 19. The tra๏ฌc keeps climbingโฆ
- 20. Performance limits are near Proprietary and โข First signs of performance problems start creeping up โข Symptoms of read scalability problems โข Pages load slowly or timeout โข Users are getting 503 Service Unavailable โข Database is slammed (very high CPU or read IO) โข Symptoms of write scalability problems โข Database write IO is maxed out, CPU is not โข Update operations are waiting on each other, piling up โข Application โlocks upโ, timeouts โข Replicas are not catching up โข Some pages load (cached?), some donโt
- 21. Proprietary and Both situations may easily result in downtime
- 22. Proprietary and Even though we achieved 99.99% uptime in 2013, in 2014 we had a couple short downtimes caused by overloaded replica that lasted around 5 minutes. But users quickly noticeโฆ
- 23. Proprietary and
- 24. Proprietary and Perhaps not :)
- 25. Proprietary and Common patterns for scaling high tra๏ฌc web applications, based on wanelo.com โจ 12-Step Program for curing your dependency on slow application latency
- 26. Proprietary and โข For small / fast HTTP services, 10-12ms or lower โข If your app is high tra๏ฌc (100K+ RPM) I recommend 80ms or lower Whatโs a good latency?
- 27. Proprietary and โข RubyVM (30ms) + Garbage collection (6ms) is CPU burn, easy to scale by adding more app servers โข Web services + Solr (25ms), memcached (15ms), database (6ms) are all waiting on IO CPU burn vs Waiting on IO?
- 28. Proprietary and Step 1: โจ Add More Cache!
- 29. Moar Cache!!! Proprietary and โขAnything that can be cached, should be โขCache hit = many database hits avoided โขHit rate of 17% still saves DB hits โขWe can cache many types of thingsโฆ โขCache is cheap and fast (memcached)
- 30. Cache many types of things Proprietary and โข caches_action in controllers is very e๏ฌective โข fragment caches of reusable widgets โข we use gem Compositor for JSON API. We cache serialized object fragments, grab them from memcached using multi_get and merge them โข Shopify open sourced IdentityCache, which caches AR models, so you can Product.fetch(id) https://github.com/wanelo/compositorโจ https://github.com/Shopify/identity_cache
- 31. But Caching has itโs issues Proprietary and โข Expiring cache is not easy โข CacheSweepers in Rails help โข We found ourselves doing 4000 memcached deletes in a single request! โข Could defer expiring caches to background jobs, or use TTL if possible โข But we can cache even outside of our app: โจ we cache JSON API responses using CDN (fastly.com)
- 32. Proprietary and Step 2: โจ Optimize SQL
- 33. SQL Optimization โข Find slow SQL (>100ms) and either remove it, cache the hell out of it, or ๏ฌx/rewrite the query โข Enable slow query log in postgresql.conf:โจ โจ log_min_duration_statement = 80โจ log_temp_files = 0 โข pg_stat_statements is an invaluable contrib module:โจ โจ
- 34. Fixing Slow Query Proprietary and Run explain plan to understand how DB runs the query Are there adequate indexes for the query? Is the database using appropriate index? Has the table been recently analyzed? Can a complex join be simpli๏ฌed into a subselect? Can this query use an index-only scan? Can โorder byโ column be added to the index? pg_stat_user_indexes and pg_stat_user_tables for seq scans, unused indexes, cache info
- 35. SQL Optimization, ctd Proprietary and โข Instrumentation software such as NewRelic shows slow queries, with explain plans, and time consuming transactions
- 37. Proprietary and One day, I noticed lots of temp ๏ฌles created in the postgres.log
- 38. Proprietary and Letโs run this queryโฆ This join takes a whole second to return :(
- 39. Proprietary and โข Follows tableโฆ
- 40. Proprietary and โข Stories tableโฆ
- 41. Proprietary and So our index is partial, only on state = โactiveโ So this query is a full table scanโฆ But there state isnโt used in the query, a bug? Letโs add state = โactiveโ It was meant to be there anyway
- 42. Proprietary and After adding extra condition IO drops signi๏ฌcantly:
- 43. Proprietary and Step 3: โจ Upgrade Hardware and RAM
- 44. Hardware + RAM Proprietary and โข Sounds obvious, but better or faster hardware is an obvious choice when scaling out โข Large RAM will be used as ๏ฌle system cache โข On Joyentโs SmartOS ARC FS cache is very e๏ฌective โข shared_buffers should be set to 25% of RAM or 12GB, whichever is smaller โข Using fast SSD disk array can make a huge di๏ฌerence โข Joyentโs native 16-disk RAID managed by ZFS instead of controller provides excellent performance
- 45. Hardware in the cloud Proprietary and โข SSD o๏ฌerings from Joyent and AWS โข Joyents โmaxโ SSD node $12.9/hr โข AWS โmaxโ SSD node $6.8/hr
- 46. So whoโs better? Proprietary and โข JOYENT โข 16 SSD drives: RAID10 + 2 โข SSD Make: DCS3700 โข CPU: E5-2690โจ 2.9GHz โข AWS โข 8 SSD drives โข SSD Make: ? โข CPU: E5-2670โจ 2.6Ghz Perhaps you get what you pay for after allโฆ.
- 47. Proprietary and Step 4: โจ Scale Reads by Replication
- 48. Scale Reads by Replication Proprietary and โข postgresql.conf (both master & replica) โข These settings have been tuned for SmartOS and our application requirements (thanks PGExperts!)
- 49. How to distribute reads? Proprietary and โข Some people have success using this setup for reads:โจ app haproxy pgBouncer replicaโจ pgBouncer replica โข Iโd like to try this method eventually, but we choose to deal with distributing read tra๏ฌc at the application level โข We tried many ruby-based solutions that claimed to do this well, but many werenโt production ready
- 50. Proprietary and โข Makara is a ruby gem from TaskRabbit that we ported from MySQL to PostgreSQL for sending reads to replicas โข Was the simplest library to understand, and port to PG โข Worked in the multi-threaded environment of Sidekiq Background Workers โข automatically retries if replica goes down โข load balances with weights โข Was running in production
- 51. Special considerations Proprietary and โข Application must be tuned to support eventual consistency. Data may not yet be on replica! โข Must explicitly force fetch from the master DB when itโs critical (i.e. after a user accountโs creation) โข We often use below pattern of ๏ฌrst trying the fetch, if nothing found retry on master db
- 52. Replicas can specialize Proprietary and โข Background Workers can use dedicated replica not shared with the app servers, to optimize hit rate for ๏ฌle system cache (ARC) on both replicas PostgreSQL Master Unicorn / Passenger Ruby VM (times N) App Servers Sidekiq / Resque Background Workers PostgreSQL Replica 1 PostgreSQL Replica 2 PostgreSQL Replica 3 ARC cache warm with queries from web traf๏ฌc ARC cache warm with background job queries
- 53. Big heavy reads go there Proprietary and โข Long heavy queries should run by the background jobs against a dedicated replica, to isolate their e๏ฌect on web tra๏ฌc PostgreSQL Master Sidekiq / Resque Background Workers PostgreSQL Replica 1 โข Each type of load will produce a unique set of data cached by the ๏ฌle system
- 54. Proprietary and Step 5: โจ Use more appropriate tools
- 55. Leveraging other tools Proprietary and Not every type of data is well suited for storing in a relational DB, even though initially it may be convenient โข Redis is a great data store for transient or semi- persistent data with list, hash or set semantics โข We use it for ActivityFeed by precomputing each feed at write time. But we can regenerate it if the data is lost from Redis โข We use twemproxy in front of Redis which provides automatic horizontal sharding and connection pooling. โข We run clusters of 256 redis shards across many virtual zones; sharded redis instances use many cores, instead of one โข Solr is great for full text search, and deep paginated sorted lists, such as trending, or related products
- 56. Proprietary and True story: applying WAL logs onโจ replicas creates signi๏ฌcant disk write load But we still have single master DB taking all the writesโฆ Replicas are unable to both serve live tra๏ฌc and โจ catch up on replication. They fall behind. Back to PostgreSQL
- 57. Proprietary and When replicas fall behind, application generates errors, unable to ๏ฌnd data it expects
- 58. Proprietary and Step 6: โจ Move write-heavy tables out: Replace with non-DB solutions
- 59. Move event log out Proprietary and โข We were appending all user events into this table โข We were generating millions of rows per day! โข We solved it by replacing user event recording system to use rsyslog, appending to ASCII ๏ฌles โข We discovered from pg_stat_user_tables top table by write volume was user_events Itโs cheap, reliable and scalable We now use Joyentโs Manta to analyze this data in parallel. Manta is an object store + native compute on
- 60. Proprietary and For more information about how we migrated user events to a ๏ฌle-based append-only log, and analyze it with Manta, please read http://wanelo.ly/event-collection
- 61. Proprietary and Step 7: โจ Tune PostgreSQL and your Filesystem
- 62. Tuning ZFS Proprietary and โข Problem: zones (virtual hosts) with โwrite problemsโ appeared to be writing 16 times more data to disk, compared to what virtual ๏ฌle system reports โข vfsstat says 8Mb/sec write volume โข So whatโs going on? โข iostat says 128Mb/sec is actually written to disk
- 63. Proprietary and โข Turns out default ZFS block size is 128Kb, and PostgreSQL page size is 8Kb. โข Every small write that touched a page, had to write 128Kb of a ZFS block to the disk Tuning Filesystem โข This may be good for huge sequential writes, but not for random access, lots of tiny writes
- 64. Proprietary and โข Solution: Joyent changed ZFS block size for our zone, iostat write volume dropped to 8Mb/sec โข We also added commit_delay Tuning ZFS & PgSQL
- 65. Proprietary and โข Many such settings are pre-de๏ฌned in our open-source Chef cookbook for installing PostgreSQL from sources Installing and Con๏ฌguring PG โข https://github.com/wanelo-chef/postgres โข It installs PG in eg /opt/local/postgresql-9.3.2 โข It con๏ฌgures itโs data in /var/pgsql/data93 โข It allows seamless and safe upgrades of minor or major versions of PostgreSQL, never overwriting binaries
- 66. Additional resources online Proprietary and โข Josh Berkusโs โ5 steps to PostgreSQL Performanceโ on SlideShare is fantastic โข PostgreSQL wiki pages on performance tuning are excellent โข Run pgBench to determine and compare performance of systems http://www.slideshare.net/PGExperts/๏ฌve-steps-perform2013 http://wiki.postgresql.org/wiki/Performance_Optimization http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
- 67. Proprietary and Step 8: โจ Bu๏ฌer and serialize frequent updates
- 68. Counters, countersโฆ Proprietary and โข Problem: products.saves_count is incremented every time someone saves a product (by 1) โข At 100s of inserts/sec, thatโs a lot of updates How can we reduce number of writes and โจ lock contention? โข Worse: 100s of concurrent requests trying to obtain a row level lock on the same popular product
- 69. Bu๏ฌering and serializing Proprietary and โข Sidekiq background job framework has two inter-related features: โข scheduling in the future (say 10 minutes ahead) โข UniqueJob extension โข Once every 10 minutes popular products are updated by adding a value stored in Redis to the database value, and resetting Redis value to 0 โข We increment a counter in redis, and enqueue a job that says โupdate product in 10 minutesโ
- 70. Bu๏ฌering explained Proprietary and Save Product Save Product Save Product 1. enqueue update request for product with a delay PostgreSQL Update Request already on the queue 3. Process Job Redis Cache 2. increment counter 4. Read & Reset to 0 5. Update Product
- 71. Bu๏ฌering conclusions Proprietary and โข If not, to achieve read consistency, we can display the count as database value + redis value at read time โข If we show objects from the database, they might be sometimes behind on the counter. It might be okโฆ
- 72. Proprietary and Step 9: โจ Optimize DB schema
- 73. MVCC does copy on write Proprietary and โข Problem: PostgreSQL rewrites the row for most updates (some exceptions exist, ie non-indexed column, a counter, timestamp) โข But we often index these so we can sort by them โข Rails and Hibernateโs partial updates are not helping โข Are we updating User on each request? โจ โจ โข So updates can become expensive on wide tables
- 74. Schema tricks Proprietary and โข Solution: split wide tables into several 1-1 tables to reduce update impact โข Much less vacuuming required when smaller tables are frequently updated
- 75. Proprietary and Donโt update anything on each request :) id email encrypted_password reset_password_token reset_password_sent_at remember_created_at sign_in_count current_sign_in_at last_sign_in_at current_sign_in_ip last_sign_in_ip con๏ฌrmation_token con๏ฌrmed_at con๏ฌrmation_sent_at uncon๏ฌrmed_email failed_attempts unlock_token locked_at authentication_token created_at updated_at username avatar state followers_count saves_count collections_count stores_count following_count stories_count Users id email created_at username avatar state Users user_id encrypted_password reset_password_token reset_password_sent_at remember_created_at sign_in_count current_sign_in_at last_sign_in_at current_sign_in_ip last_sign_in_ip con๏ฌrmation_token con๏ฌrmed_at con๏ฌrmation_sent_at uncon๏ฌrmed_email failed_attempts unlock_token locked_at authentication_token updated_at UserLogins user_id followers_count saves_count collections_count stores_count following_count stories_count UserCounts refactor
- 76. Proprietary and Step 10: โจ Shard Busy Tables Vertically
- 77. Vertical sharding Proprietary and โข Heavy tables with too many writes, can be moved into their own separate database โข For us it was saves: now @ 2B+ rows โข At hundreds of inserts per second, and 4 indexes, we were feeling the pain โข It turns out moving a single table (in Rails) out is a not a huge e๏ฌort: it took our team 3 days
- 78. Vertical sharding - how to Proprietary and โข Update code to point to the new database โข Implement any dynamic Rails association methods as real methods with 2 fetches โข ie. save.products becomes a method on Save model, lookup up Products by IDs โข Update development and test setup with two primary databases and ๏ฌx all the tests
- 79. Proprietary and Web App PostgreSQL Master (Main Schema) PostgreSQL Replica (Main Schema) Vertically Sharded Database PostgreSQL Master (Split Table) Here the application connects to main master DB + replicas, and a single dedicated DB for the busy table we moved
- 80. Vertical sharding, deploying Proprietary and Drop in write IO on the main DB after splitting o๏ฌ the high IO table into a dedicated compute node
- 81. Proprietary and For a complete and more detailed account of our vertical sharding e๏ฌort, please read our blog post: http://wanelo.ly/vertical-sharding
- 82. Proprietary and Step 11: โจ Wrap busy tables with services
- 83. Splitting o๏ฌ services Proprietary and โข Vertical Sharding is a great precursor to a micro-services architecture โข New service: Sinatra, client and server libs, updated tests & development, CI, deployment without changing db schema โข 2-3 weeks a pair of engineers level of e๏ฌort โข We already have Saves in another database, letโs migrate it to a light-weight HTTP service
- 84. Adapter pattern to the rescue Proprietary and Main App Unicorn w/ Rails PostgreSQL HTTP Client Adapter Service App Unicorn w/Sinatra Native Client Adaptor โข We used Adapter pattern to write two client adapters: native and HTTP, so we can use the lib, but not yet switch to HTTP
- 85. Services conclusions Proprietary and โข Now we can independently scale service backend, in particular reads by using replicas โข This prepares us for the next inevitable step: horizontal sharding โข At a cost of added request latency, lots of extra code, extra runtime infrastructure, and 2 weeks of work โข Do this only if you absolutely have to
- 86. Proprietary and Step 12: โจ Shard Services Backend Horizontally
- 87. Horizontal sharding in ruby Proprietary and โข We wanted to stick with PostgreSQL for critical data such as saves โข Really liked Instagramโs approach with schemas โข Built our own schema-based sharding in ruby, on top of Sequel gem, and open sourced it โข It supports mapping of physical to logical shards, and connection pooling https://github.com/wanelo/sequel-schema-sharding
- 88. Schema design for sharding Proprietary and https://github.com/wanelo/sequel-schema-sharding user_id product_id collection_id created_at index__on_user_id_and_collection_id UserSaves Sharded by user_id product_id user_id updated_at index__on_product_id_and_user_id index__on_product_id_and_updated_at ProductSaves Sharded by product_id We needed two lookups, by user_id and by product_id hence we needed two tables, independently sharded Since saves is a join table between user, product, collection, we did not need unique ID generated Composite base62 encoded ID: fpua-1BrV-1kKEt
- 89. Spreading your shards Proprietary and โข We split saves into 8192 logical shards, distributed across 8 PostgreSQL databases โข Running on 8 virtual zones spanning 2 physical SSD servers, 4 per compute node โข Each database has 1024 schemas (twice, because we sharded saves into two tables) https://github.com/wanelo/sequel-schema-sharding 2 x 32-core 256GB RAM 16-drive SSD RAID10+2 PostgreSQL 9.3 1 3 4 2
- 90. Proprietary and Sample con๏ฌguration of shard mapping to physical nodes with read replicas, supported by the library
- 91. Proprietary and How can we migrate the data from old non- sharded backend to the new sharded backend without a long downtime?
- 92. New records go to both Proprietary and HTTP Service Old Non-Sharded Backend New Sharded Backend 1 3 4 2 Read/Write Background Worker Enqueue Sidekiq Queue Create Save
- 93. Proprietary and HTTP Service Old Non-Sharded Backend New Sharded Backend 1 3 4 2 Read/Write Background Worker Enqueue Sidekiq Queue Create Save Migration Script Migrate old rows We migrated several times before we got this rightโฆ
- 94. Proprietary and Swap old and new backends HTTP Service Old Non-Sharded Backend New Sharded Backend 1 3 4 2Read/Write Background Worker Enqueue Sidekiq Queue Create Save
- 95. Horizontal sharding conclusions Proprietary and โข This is the ๏ฌnal destination of any scalable architecture: just add more boxes โข Pretty sure we can now scale to 1,000, or 10,000 inserts/second by scaling out โข Took 2 months of 2 engineers, including migration, but zero downtime. Itโs an advanced level e๏ฌort and our engineers really nailed this. https://github.com/wanelo/sequel-schema-sharding
- 96. Putting it all together Proprietary and โข This infrastructure complexity is not free โข It requires new automation, monitoring, graphing, maintenance and upgrades, and brings with it a new source of bugs โข In addition, micro-services can be โownedโ by small teams in the future, achieving organizational autonomy โข But the advantages are clear when scaling is one of the requirements
- 97. Proprietary and Systems Diagram incoming http requests 8-core 8GB zones haproxy nginx Fastly CDN cache images, JS Load Balancers Amazon S3 Product Images User Pro๏ฌle Pictures 32-core 256GB 16-drive SSD RAID10+2 Supermicro "Richmond" SSD Make: Intel DCS3700, CPU: Intel E5-2690, 2.9GHz PostgreSQL 9.2 Master Primary Database Schema 4-core 16GB zones memcached User and Product Saves, Horizontally Sharded, Replicated 32-core 256GB RAM 16-drive SSD RAID10+2 PostgreSQL 9.3 1 3 4 2 Read Replicas (non SSD) 2 4 2 1 Read Replica (SSD) PostgreSQL Async Replicas 32-core 32GB high-CPU instances Unicorn Main Web/API App, Ruby 2.0 Unicorn Saves Service haproxy pgbouncer iPhone, Android, Desktop clients Makara distributes DB load across 3 replicas and 1 master MemCached Cluster Redis Clusters for various custom user feeds, such as product feed 1-core 1GB zones twemproxy Redis Proxy Cluster 16GB high-mem 4-core zones 32 redis instances per server redis-001 redis-256 8GB High CPU zones Solr Replica 8GB High CPU zone Solr Master App Servers + Admin Servers Cluster of MemCached Servers is accessed via Dali fault tolerant library one or more can go down Apache Solr Clusters 32-core 32GB high-CPU instances Sidekiq Background Worker Unicorn Saves Service haproxy pgbouncer to DBs Solr Reads Solr Updates Background Worker Nodes redis Redis Sidekiq Jobs Queue / Bus
- 98. Systems Status: Dashboard Monitoring & Graphing with Circonus, NewRelic, statsd, nagios
- 99. Backend Stack & Key Vendors Proprietary and โ MRI Ruby, jRuby, Sinatra, Ruby on Rails โ PostgreSQL, Solr, redis, twemproxyโจ memcached, nginx, haproxy, pgbouncer โ Joyent Cloud, SmartOS, Manta Object Storeโจ ZFS, ARC Cache, superb IO, SMF, Zones, dTrace, humans โ DNSMadeEasy, MessageBus, Chef, SiftScience โ LeanPlum, MixPanel, Graphite analytics, A/B Testing โ AWS S3 + Fastly CDN for user / product images โ Circonus, NewRelic, statsd, Boundary, โจ PagerDuty, nagios:trending / monitoring / alerting
- 100. Proprietary and We are hiring! DevOps, Ruby, Scalability, iOS & Android Talk to me after the presentation if you are interested in working on real scalability problems, and on a product used and loved by millions :) http://wanelo.com/about/play Or email [email protected]
- 101. Thanks! โจ github.com/wanelo github.com/wanelo-chef โจ wanelo technical blog (srsly awsm) building.wanelo.com Proprietary and @kig @kig @kigster