Server architecture & scaling strategy for a sports website
1 like1,408 views
- The document discusses the server architecture, availability, and scaling strategy used for the Greek sports website sport.gr. It describes transitioning from dedicated servers to using Amazon Web Services (AWS) like S3, EC2, CloudFront, Route 53, and CloudWatch. - This provided significant cost reductions and improved ability to scale up during traffic peaks. Techniques used included auto-scaling EC2 instances, CDN delivery with CloudFront, load balancing with ELB, and monitoring with CloudWatch. - While AWS helped scale the growing video-focused site, the document notes some limitations like slow EBS disk I/O, provisioning times, and that Akamai is still faster than Cloud
![# who’s talking
leonidas tsementzis
aka @goldstein
* software architect, engineer
[all major web/mobile platforms]
* devOps
[enthusiast, not a real sysadmin]
* entrepreneur
[n00b]
CTO Mobile architect CTO](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-2-320.jpg)
![# i can see clearly now :)
* videos
[goals, match coverage]
* comments
[the blogging age, remember?]
* live streaming
[ustream does not exist, yet]
* live coverage of events
[cover it live does not exist, yet]
* user-centric design
[personalization, ratings]
* even more videos
[I can haz more LOLCats]](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-4-320.jpg)
![# the problem :(
* we are planning for a 150% traffic growth but
[6 months planning ahead]
$ video costs
[bandwidth cost: 1€/GB]
* comments costs
[DB writes, CPU, disk i/o]
* live streaming costs
[bandwidth cost: 1€/GB]
* limited iron resources, not happy with our current host
[dedicated managed servers in top GR Datacenter]](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-5-320.jpg)
![# S3 to the rescue
* 87% cost reduction
[0.13€/GB VS 1€/GB]
* made videos section possible...
* ...and advertisers loved it ($$$+)
* first GR site to focus on video, key competitive advantage
* 6TB video traffic in the first month
* hired a video editing team to support the demand](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-6-320.jpg)
![# EC2 servers on demand
* 3x(n) Application servers for the main website
[Windows 2003, IIS 6]
* 2x(n) Application servers for APIs
[Windows 2003, IIS 6]
* 2x(n) Servers for banner managers
[CentOS, Apache, OpenX]
* 1x Storage server
* 2x Database servers
[MS SQL Server 2008 with failover]
* 2x Reverse Proxy cache servers
[Squid]
* 2x Load Balancers
[HAProxy with failover]
* 1x monitoring server
[munin with a lot of custom plugins]](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-7-320.jpg)
![# wait, there’s more!
* CDN & media streaming with CloudFront
* use multiple CNAMES with CloudFront to boost HTTP
requests
[as per YSlow recommends]
* CloudFront custom domains are sexy
* robust DNS with Route 53
* simple monitoring with CloudWatch
[you still need an external monitoring tool]](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-11-320.jpg)
![# lessons learned
* test, iterate, test, iterate
* reserved instances saves you $$
* EC2 is a hacker playground
[prepare for DOS attacks]
* backup entire AMIs to S3
[instances *WILL* #FAIL]
* EBS disk I/O is slow, but amazon is working on this
[problems with DB writes]
* spawning new instances is slow
[15 mins provisioning can be a show stopper on scaling]
* S3 uploads/downloads are slow
* sticky session is a must
[we replaced AWS ELB with HAProxy just for this]
* SLAs can't guarantee high availability
[AWS *WILL* #FAIL]](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-13-320.jpg)
![# more lessons learned
* devOps are hard to find
[interested? I’m hiring]
* automate everything
[makes you sleep at night]
* monitor everything
[munin is your friend]
* disaster prevention
[work *ALWAYS* around the worst case scenario]
* windows server administration is a mess
[and AWS is not making this prettier]
* DB scale is the hardest part
[code changes]
* legacy software *IS* a problem
** on scaling
** on hiring
** on growing (have you tried to use XMPP via ASP?)](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-14-320.jpg)
![# AWS is not perfect
* AKAMAI is still faster compared to CloudFront
[especially in Greece]
* not affordable for large architectures
[if you’re running 300+ instances, you should consider making your own
datacenter]](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-15-320.jpg)
![# questions? challenges?
?
@goldstein
aka leonidas tsementzis
leotsem [at] gmail.com](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-16-320.jpg)
![# thank you
!
@goldstein
aka leonidas tsementzis
leotsem [at] gmail.com](https://image.slidesharecdn.com/awspresentation-120217023252-phpapp02/85/Server-architecture-scaling-strategy-for-a-sports-website-17-320.jpg)
![[Perforce] Adventures in Build](https://cdn.slidesharecdn.com/ss_thumbnails/perforce-adventures-in-build-slides-130523180506-phpapp02-thumbnail.jpg?width=560&fit=bounds)
![Serhiy Kalinets "Console: why should we care?"[Lightning talk]](https://cdn.slidesharecdn.com/ss_thumbnails/kalinetswhyconsole-180418120639-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Server architecture & scaling strategy for a sports website (20)
Ad
Recently uploaded (20)
Ad
Server architecture & scaling strategy for a sports website
- 1. server architecture, availability & scaling strategy CTO leonidas tsementzis
- 2. # who’s talking leonidas tsementzis aka @goldstein * software architect, engineer [all major web/mobile platforms] * devOps [enthusiast, not a real sysadmin] * entrepreneur [n00b] CTO Mobile architect CTO
- 3. # the high-level requirements * 2007 * take sport.gr to the next level... * ... make sure it works smoothly... * ... and fast enough
- 4. # i can see clearly now :) * videos [goals, match coverage] * comments [the blogging age, remember?] * live streaming [ustream does not exist, yet] * live coverage of events [cover it live does not exist, yet] * user-centric design [personalization, ratings] * even more videos [I can haz more LOLCats]
- 5. # the problem :( * we are planning for a 150% traffic growth but [6 months planning ahead] $ video costs [bandwidth cost: 1€/GB] * comments costs [DB writes, CPU, disk i/o] * live streaming costs [bandwidth cost: 1€/GB] * limited iron resources, not happy with our current host [dedicated managed servers in top GR Datacenter]
- 6. # S3 to the rescue * 87% cost reduction [0.13€/GB VS 1€/GB] * made videos section possible... * ...and advertisers loved it ($$$+) * first GR site to focus on video, key competitive advantage * 6TB video traffic in the first month * hired a video editing team to support the demand
- 7. # EC2 servers on demand * 3x(n) Application servers for the main website [Windows 2003, IIS 6] * 2x(n) Application servers for APIs [Windows 2003, IIS 6] * 2x(n) Servers for banner managers [CentOS, Apache, OpenX] * 1x Storage server * 2x Database servers [MS SQL Server 2008 with failover] * 2x Reverse Proxy cache servers [Squid] * 2x Load Balancers [HAProxy with failover] * 1x monitoring server [munin with a lot of custom plugins]
- 8. # a nice headache :( :( :’(
- 9. # a typical week * peaks at 3k hits/sec once or twice/week * normal ratio at 300 hits/sec * you can’t afford the 1st * you can’t deliver on the 2nd
- 10. # auto-scaling to the rescue * if average CPU usage grows over 60% for 2 minutes, add another application server * if average CPU usage falls below 30% for 5 minutes, kill gracefully an application server * 20 instances on peaks * 3 instances (minimum) on normal operations * no more “Server is busy” errors * pay only what you (really) need * you can now sleep at nights * 60% overall cost reduction
- 11. # wait, there’s more! * CDN & media streaming with CloudFront * use multiple CNAMES with CloudFront to boost HTTP requests [as per YSlow recommends] * CloudFront custom domains are sexy * robust DNS with Route 53 * simple monitoring with CloudWatch [you still need an external monitoring tool]
- 12. # SUM() * S3 * CloudFront Photos Video streaming Videos CDN Static banners * ELB * EC2 Load balancing Main website SQL Databases RDS * Backoffice MySQL databases APIs Banner Managers * Route 53 Cache servers Load Balancers DNS Resolution * CloudWatch Auto-scaling Simple Monitoring
- 13. # lessons learned * test, iterate, test, iterate * reserved instances saves you $$ * EC2 is a hacker playground [prepare for DOS attacks] * backup entire AMIs to S3 [instances *WILL* #FAIL] * EBS disk I/O is slow, but amazon is working on this [problems with DB writes] * spawning new instances is slow [15 mins provisioning can be a show stopper on scaling] * S3 uploads/downloads are slow * sticky session is a must [we replaced AWS ELB with HAProxy just for this] * SLAs can't guarantee high availability [AWS *WILL* #FAIL]
- 14. # more lessons learned * devOps are hard to find [interested? I’m hiring] * automate everything [makes you sleep at night] * monitor everything [munin is your friend] * disaster prevention [work *ALWAYS* around the worst case scenario] * windows server administration is a mess [and AWS is not making this prettier] * DB scale is the hardest part [code changes] * legacy software *IS* a problem ** on scaling ** on hiring ** on growing (have you tried to use XMPP via ASP?)
- 15. # AWS is not perfect * AKAMAI is still faster compared to CloudFront [especially in Greece] * not affordable for large architectures [if you’re running 300+ instances, you should consider making your own datacenter]
- 16. # questions? challenges? ? @goldstein aka leonidas tsementzis leotsem [at] gmail.com
- 17. # thank you ! @goldstein aka leonidas tsementzis leotsem [at] gmail.com