AWS re:Invent 2016 - Scality's Open Source AWS S3 Server
Download as PPTX, PDF1 like1,697 views
Scality's AWS S3 Server is an open-source, object-based software-defined storage solution optimized for cloud environments, capable of running on standard x86 servers and providing reliable performance for high-capacity applications. Released under an Apache-2.0 license, it has gained significant adoption due to the growing demand for AWS S3 as a de-facto interface standard. The solution supports local development and scalable deployment scenarios, catering to use cases in video on demand, cloud services, and long-term archiving.
AWS re:Invent 2016 - Scality's Open Source AWS S3 Server
- 1. AWS re:Invent 2016 Scality’s Open Source AWS S3 Server Giorgio Regni, CTO @GiorgioRegni
- 2. Cloud Field Days #1 © Scality 20162 Scality RING: Automatize storage for Digital Business The Scality RING is object-based software-defined storage for the cloud. We run on standard x86 servers and create a giant pool of storage. We protect the data and provide 100% reliable, high performance access for any capacity-driven application. FILE OBJECT OPENSTACK
- 3. Disrupting storage – unlimited & everywhere
- 4. © Scality 20164 Open source object storage server https://github.com/scality/s3 Written in Node.js Single instance running in a docker container Uses docker volumes for persistent storage Same code as Scality’s RING S3 interface What is Scality AWS S3 Server?
- 5. © Scality 20165 Scality AWS S3 server released under an Apache-2.0 license in July -> >15K downloads on docker hub! 5
- 6. Customer & partner push for AWS S3 has swelled in last 18 months AWS S3 has become de-facto interface standard Growing demand 6 AWS S3 Adoption keeps rising
- 7. Why you should use AWS S3?
- 8. 8 Object vs File Bucket 1 Object A Object B Object C Bucket 2 Object A Object B Object C Object/S3: • Flat Data Model with collections called “buckets” • Objects are written and overwritten not byte-wise modified • Scales to billions of objects File/Posix: • Hierarchical Data Model • Files are randomly writable in byte- wise fashion • Scales to hundreds of thousands files
- 9. 9 Why object scales better Separation of Metadata and Data • Bucket listing and object locations are stored separately from the Data • Objects can be spread out anywhere • Direct access to data - no need to traverse a tree structure Shared nothing: • Consistency rules only apply to one object or bucket at a time • Clients are stateless - Last writer wins • No relationships exist between objects Data Data Data MD MD MD Data… … SDS/Object Service
- 10. © Scality 2016 Scality AWS S3 Server achitecture Protocol Stack Buckets, Objects, MPU (REST API) AUTH BUCKETS DATA LevelDB* Scality RING v6 Kinetic IP drives Docker Volume * Persisted in Docker Volume Scality RING v6Vault LevelDB* S3 API S3 Server Public Cloud
- 11. © Scality 201612 Developers can install and develop S3- based apps locally Enterprises can host a local test/dev environment to learn about object storage Enterprises can host a small, local object storage system in production Scality AWS S3 Server: From 0 to S3 in Under 5 Minutes S3 Server S3 Server S3 Server Backup Application S3
- 12. Steps: 1. Launch Kitematic UI to access Docker Hub 2. Pull the S3 Server container 3. Start the container 4. Use the Cyberduck UI to create Buckets, PUT, GET & DELETE Objects Quick, Live AWS S3 Server Demo!
- 13. © Scality 201614 Why it matters Supports many simultaneous clients to the same bucket, even across sites Simplifies access Linear performance scaling AWS S3 Server for RING - Scale-out & Performance Scalable Bucket Namespace with simple access point Scales-out in simple uniform building blocks Optimized for low latency, high bandwidth, and fast listing Multi-site deployment preserves availability during site or network failure (initial support for two sites) Scalable S3 Bucket Namespace: any-to-any access APP A APP B APP D APP C
- 14. Challenges & how to overcome them
- 15. Logging is hard • Challenges • Logging is expensive as it taxes the Node.js process • UDP datagrams have expensive DNS lookups • Redundant transformations by bunyan and bunyan-logstash • Solution: Werelogs • Produces raw JSON logs with the least resistive path • Forward logs to ELK using Filebeat for indexing • Avoids expensive and redundant transformations • Ability to track requests across the components with UIDs • Dump log history on errors Open source -> http://github.com/scality/werelogs
- 16. Performance, performance & performance
- 17. The performance cycle Code, Benchmark, … Repeat • Socket & Nagle algorithm on by default -> very high latencies • The event loop can get backed up quickly -> hunt for all cpu intensive tasks in the main loop • Buffers are much more efficient when writing server response • Micro optimizations: Date.now() > new Date() • Beware of libraries doing way too many things for you • ES6 support, Babel5 was killing performance -> Babel6
- 19. Download Scality AWS S3 Server! http://s3.scality.com/
- 21. © Scality 2016 VOD/Live Streaming: Deluxe RING was deployed as an origin server to store and distribute all of Deluxe OnDemand’s transcoded titles (including Theatrical and TV) to large cable providers, retailers, and end consumers. The initial deployment was 1.5PB, which the customer expects to add 10-15PB for different applications. Highest valued features: • Performance – serves video to millions of users at 100Gb/s. • Hardware-freedom – allows for competitive hardware pricing, and utilization of newer disks and hardware over time to stay in the sweet spot of cost. • Scalability – meet 10X expansion with no data migration About the customer: Deluxe OnDemand is a cloud-based multiscreen VOD catalog service that simplifies the access and delivery of content to any device. What was the challenge? Customer needed to transcode and store up to 10TB of content daily. Needed a platform that could scale to petabytes and remain cost-effective. Video Testimonial: https://vimeo.com/134065438
- 22. © Scality 2016 Web & Cloud: Daisy Group (fka Phoenix IT) GEO-stretched RING across 3 UK-based data centers. Highly resilient configuration. Running on HPE SL4540 servers with DL360 servers used for connectors. Highest valued features: – Reliability – Reduce risk with data resiliency – Scalability – Easily grow to 1 PB and beyond – Application support – Integration with archiving app. – Future proof – Can use the same platform to host other services Video Testimonial: https://vimeo.com/131127269 About the customer: Leading provider of business continuity and managed services in the UK What was the challenge? Launching new Archive as a Service. Needed the solution to be significantly lower cost than existing infrastructure
- 23. © Scality 2016 Long Term Archive: AB TV France AB TV deployed a HP SL4540-based 700TB active/passive RING at two fiber-connected data centers. Initially, they migrated their 700TB LTO archive to the RING and recently added 300TB on both sites to grow to a PB. They are currently using SGT for Media Asset Management. Highest valued features: • Geo-distribution and 100% reliability – second site for disaster recovery ensures data availability • Performance – Superior to LTO-based access performance; Simplified management and automation. • Scalability – ability to meet future storage needs About the customer: The AB Group, a French broadcasting company, produces content for both TV and the Web. The company owns 14 French language channels such as AB Moteurs, RTL9 and Ciné FX. What was the challenge? First, AB TV needed to migrate its tape- based assets. In addition, they were frustrated with their hardware dependent RAID system.
- 24. Download Scality AWS S3 Server! http://s3.scality.com/
Editor's Notes
- #3: The Scality RING is software-defined storage. We run on standard x86 servers and create a giant pool of storage. We protect the data and provide 100% reliable, high performance access for any capacity-driven application. Why, what. [On the same slide you can also talk about benefits] Amazing capabilities and benefits: 100% reliability with no maintenance windows, for superior SLAs to your customers – case in point, our first customer hasn’t been down since they installed the RING five years ago! The only storage platform with native file, object, and OpenStack support – legacy file, digital, and cloud workloads all in one platform for better utilization, holistic protection, and lower costs Real hardware choice means the best hardware for you at any point in the lifecycle, and no data migration – better economics and lower risk
- #4: A little bit about Scality – founded in 2009 with our first customer in 2010. Offices around the world - with local language support in US, France, and Japan Managing over 800 billion objects in production at 100 customers around the world Ranked #1 by IDC
- #7: Why we decided to refresh our S3 interface Database have SQL Files have NFS Object has S3
- #9: De haut niveau = modèle de donnée.
- #11: So we thought, how can we help people try object?
- #15: Only object storage with scale-out access to any number of Buckets and objects Fully redundant system design Stateless S3 Servers integrate with standard load balancers Highly-durable data protection through RING storage
- #26: Deluxe on Demand recognized it was not financially feasible to store content with the CDN provider. In this case, the CDN was Limelight. The hardware/software solution was a replacement for an Isilon system.
- #27: In addition: Scality/HPE solution provided much lower TCO than EMC platform Scality strong partnership with HP an advantage
- #28: Used to use Front Porch MAM, DIVA (data mover), small caching NAS (100TB) and LTO tape. With Scality, they only need a MAM and Scality. No need to move data between tiers. Front Porch charged annually per terabyte, so the cost was very high to maintain and scale.