DjangoCon 2010 Scaling Disqus

Editor's Notes

• #2: Hi. I'm Jason (and I'm David), and we're from Disqus.
• #3: Show of hands, How many of you know what DISQUS is?
• #4: For those of you who are not familiar with us, DISQUS is a comment system that focuses on connecting communities. We power discussions on such sites as CNN, IGN, and more recently Engadget and TechCrunch. Our company was founded back in 2007 by my co-founder, Daniel Ha, and I back where we started working out of our dorm room. Our decision to use Django came down primarily to our dislike for PHP which we were previously using. Since then, we've grown Disqus to over 250+ million visitors a month.
• #5: We've peaked at over 17,000 requests per second, to Django, and we currently power comments on nearly half a million websites which accounts for more than 15 million profiles who have left over 75 million comments.
• #6: As you can imagine we have some big challenges when it comes to scaling a large Django application. For one, it’s hard to predict when events happen like last year with Michael Jackson’s death, and more recently, the Gulf Oil Spill. Another challenge we have is the fact that discussions never expire. When you visit that blog post from 2008 we have to be ready to serve those comments immediately. Not only does THAT make caching difficult, but we also have to deal with things such as dynamic paging, realtime commenting, and other personal preferences. This makes it even more important to be able to serve those quickly without relying on the cache.
• #7: So we also have some interesting infrastructure problems when it comes to scaling Disqus. We're not a destination website, so if we go down, it affects other sites as well as ours. Because of this, it's difficult for us to schedule maintenance, so we face some interesting scaling and availbility challenges.
• #8: As you can see, we have tried to keep the stack pretty thin. This is because, as we've learned, the more services we try to add, the more difficult it is to support. And especially because we have a small team, this becomes difficult to manage. So we use DNS load balancing to spread the requests to multiple HAProxy servers which are our software load balancers. These proxy requests to our backend app servers which run mod_wsgi. We use memcache for caching, and we have a custom wrapper using syslog for our queue. For our data store, we use PostgreSQL, and for replication, we use Slony for failover and read slaves.
• #9: As I said, we use HAProxy for HTTP load balancing. It's a high performance software load balancer with intelligent failure detection. It also provides you with nice statistics of your requests. We use heartbeat for high availability and we have it take over the IP address of the down machine.
• #10: We have about 100GB of cache. Because of our high availability requirements, 20% are allocated to high availability and load balancing.
• #11: Our web servers are pretty standard. We use mod_wsgi mostly because it just works. Performance wise, you're really going to be bottlenecked on the application. The cool thing we do is that we actually hasve a custom middleware that does performance monitoring. What this does is ship data from our application about external calls like database, cache calls, and we collect it and graph it with Ganglia.
• #12: The more interesting aspect of our server architecture is how we have our database setup. As I mentioned, we use Postgres as our database. Honestly, we used it because Django recommended it, and my recommendation is that if you’re not already an expert in a database, you're better off going with Postgres. We use slony for replication Slony is trigger-based which means that every write is captured and strored in a log table and those events are replayed to slave databases. This is nice over otehr methods such as log shipping because it allows us to have flexible schemas across read lsaves. For example, some of our read slaves have different indexes. We also use slony for failover for high availbility.
• #13: There are a few things we do to keep our database healthy. We keep our indexes in memory, and when we can't, we partition our data. We also have application-specific indexes on our readslaves. Another important thing we've done is measure I/O. Any time we've seen high I/O is usually because we're missing indexes or indexes aren't fitting in memory. Lastly, we monitor slow queries. We send logs to pgfouine via syslog which genererates a nice report showing you which queries are the slowest.
• #14: The last thing we've found to be really helpful is switching to database connection pool. Remember, Django doesn't do this for you. We use pgbouncer for this, and there are a few easy wins for using it. One is that it limits the maximum connections to the database so it doesn't have to handle as many concurrent connections. Secpondly, you save the cost of opening and tearing down new connections per request.
• #15: Moving on to our application, we’ve found that most of the struggle is with the database layer. We’ve got a pretty standard layout if you’re familiar with forums. Forum has many threads, which has many posts. Posts use an adjacency list model, and also reference Users. With this kind of data model, one of our quickest wins has been the ability to partition data.
• #16: It’s almost entirely done at the application level, which makes it fairly easy to implement. The only thing not handled by the app is replication, and Slony does that for us. We handle partitioning in a couple of ways.
• #17: The first of which are vertical partitions. This is probably the simplest thing you can implement in your application. Kill off your joins and spread out your applications on multiple databases. Some database engines might make this easier than others, but Slony allows us to easily replicate very specific data.
• #18: Using this method you’ll need to handle joins in your Python application. We do this by performing two separate queries and mapping the foreign keys to the parent objects. For us the easiest way has been to throw them into a dictionary, iterate through the other queryset, and set the foreignkey cache’s value to the instance.
• #19: A few things to keep in mind when doing pythonic joins. They’re not going to be as fast in the database. You can’t avoid this, but it’s not something you should worry about. With this however, you get plain and simple vertical partitions. You also can cache things a lot easier, and more efficiently fetch them using things like get_many and a singular object cache. Overall your’e trading performance for scale.
• #20: Another benefit that comes from vertical partitioning is the ability to designate masters. We do this to alleviate some of the load on our primary application master. So for example, server FOO might be the source for writes on the Users table, while server BAR handles all of our other forum data. Since we’re using Django 1.2 we also get routing for free through the new routers.
• #21: Here’s an example of a simple application router. It let’s us specify a read-slave based on our app label. So if its users, we go to FOO, if its forums, we go to BAR. You can handle this logic any way you want, pretty simple and powerful.
• #22: While we use vertical partitioning for most cases, eventually you hit an issue where your data just doesn’t scale on a single database. You’re probably familiar with the word sharding, well that’s what we do with our forum data. We’ve set it up so that we can send certain large sites to dedicated machines. This also uses designated masters as we mentioned with the other partitions.
• #23: We needed this when write and read load combined became so big that it was just hard to keep up on a single set of machines. It also gives the nice added benefit of high availability in many situations. Mostly though, it all goes back to scaling our master databases.
• #24: So again we’re using the router here to handle partitioning of the forums. We can specify that CNN goes to this database alias, which could be any number of machines, and everything else goes to our default cluster. The one caveat we found with this, is sometimes hints aren’t present in the router. I believe within the current version of Django they are only available when using a relational lookup, such as a foreign key. All in all it’s pretty powerful, and you just need to be aware of it while writing your queries.