OSDC 2010 | Use Distributed Filesystem as a Storage Tier by Fabrizio Manfred
0 likes106 views
Storage is one of the most important part of a data center, the complexity to design, build and delivering 24/forever availability service continues to increase every year. For these problems one of the best solution is a distributed filesystem (DFS) This talk describes the basic architectures of DFS and comparison among different free software solutions in order to show what makes DFS suitable for large-scale distributed environments. We explain how to use, to deploy, advantages and disadvantages, performance and layout on each solutions. We also introduce some Case Studies on implementations based on openAFS, GlusterFS and Hadoop finalized to build your own Cloud Storage.
Ad
More Related Content
What's hot (19)
Similar to OSDC 2010 | Use Distributed Filesystem as a Storage Tier by Fabrizio Manfred (20)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
Ad
OSDC 2010 | Use Distributed Filesystem as a Storage Tier by Fabrizio Manfred
- 1. Fabrizio Manfred Furuholmen! Use Distributed File system as a Storage Tier !
- 2. 6/23/10! 2! Agenda Introduction Next Generation Data Center Distributed File system Distributed File system OpenAFS GlusterFS HDFS Ceph Case Studies Conclusion
- 3. 6/23/10! 3! Class Exam What do you know about DFS ? How can you create a Petabyte storage ? How can you make a centralized system log ? How can you allocate space for your user or system, when you have a thousands of users/systems ? How can you retrieve data from everywhere ?
- 4. 6/23/10! 4! Introduction Next Generation Data Center: the “FABRIC” Key categories: Continuous data protection and disaster recovery File and block data migration across heterogeneous environments Server and storage virtualization Encryption for data in-flight and at-rest In other words: Cloud data center
- 5. 6/23/10! 5! Introduction Storage Tier in the “FABRIC” High Performance Scalability Simplified Management Security High Availability Solutions Storage Area Network Network Attached Storage Distributed file system
- 6. 6/23/10! 6! Introduction What is a Distributed File system ? “A distributed file system takes advantage of the interconnected nature of the network by storing files on more than one computer in the network and making them accessible to all of them..”
- 7. What do you expected from a distributed file system ? • Uniform Access: file names global support • Security: to provide a global authentication/authorization • Reliability: the elimination of each single point of failure • Availability: administrators perform routine maintenance while the file server is in operation, without disrupting the user’s routines • Scalability: Handle terabytes of data • Standard conformance: some IEEE POSIX file system semantics standard • Performance: high performance Introduction 7!
- 8. Part II Implementations 8! How many DFS do you know ?
- 9. 6/23/10! 9! OpenAFS: introduction Key ideas: Make clients do work whenever possible. Cache whenever possible. Exploit file usage properties. Understand them. One-third of Unix files are temporary. Minimize system-wide knowledge and change. Do not hardwire locations. Trust the fewest possible entities. Do not trust workstations. Batch if possible to group operations. is the open source implementation of Andrew File system of IBM
- 11. • Cell is collection of file servers and workstation • The directories under /afs are cells, unique tree • Fileserver contains volumes Cell • Volumes are "containers" or sets of related files and directories • Have size limit • 3 type rw, ro, backup Volumes • Access to a volume is provided through a mount point • A mount point is just like a static directory Mount Point Directory OpenAFS: components Server A Server A+B Server C 11!
- 12. OpenAFS: performances OpenAFS OpenAFS OSD 2 Servers
- 13. 6/23/10! 13! OpenAFS: features Uniform name space: same path on all workstations Security: base to krb4/krb5, extended ACL, traffic encryption Reliability: read-only replication, HA database, read/write replica in OSD version Availability: maintenance tasks without stopping the service Scalability: server aggregation Administration: administration delegation Performance: client side disk base persistent cache, big rate client per Server
- 14. • Internal usage • Storage: 450 TB (ro)+ 15 TB (rw) • Client: 22.000 Morgan Stanley IT • Online picture album • Storage: 265TB ( planned growth to 425TB in twelve months) • Volumes: 800,000. • Files: 200 000 000. Pictage, Inc • Internet Shared folder • Storage: 500TB • Server: 200 Storage server • 300 App server Embian • Internal usage 210TB RZH openAFS: who uses it ? 14!
- 15. Good • Wide Area Network • Heterogeneous System • Read operation > write operation • Large number of clients/systems • Usage directly by end-users • Federation Bad • Locking • Database • Unicode • Large File • Some limitations on .. OpenAFS: good for ... 15!
- 16. 6/23/10! 16! GlusterFS “Gluster can manage data in a single global namespace on commodity hardware..” Keys: Lower Storage Cost—Open source software runs on commodity hardware Scalability—Linearly scales to hundreds of Petabytes Performance—No metadata server means no bottlenecks High Availability—Data mirroring and real time self-healing Virtual Storage for Virtual Servers—Simplifies storage and keeps VMs always-on Simplicity—Complete web based management suite
- 18. 6/23/10! 18! GlusterFS: components • Volume is the basic element for data export • The volumes can be stacked for extension Volume • Specific options (features) can be enabled for each volume (cache, pre fetch, etc.) • Simple creation for custom extensions with api interface Capabilities • Access to a volume is provided through services like tcp, unix socket, infiniband Services volume posix1! type storage/posix! option directory /home/export1! end-volume! volume brick1! type features/posix-locks! option mandatory! subvolumes posix1! end-volume! volume server! type protocol/server! option transport-type tcp ! option transport.socket.listen-port 6996! subvolumes brick1! option auth.addr.brick1.allow * ! end-volume!
- 21. 6/23/10! 21! Gluster: carateristics Uniform name space: same path on all workstation Reliability: read-1 replication, asynchronous replication for disaster recovery Availability: No system downtime for maintenance (better in the next release) Scalability: Truly linear scalability Administration: Self Healing, Centralized logging and reporting, Appliance version Performance: Stripe files across dozens of storage blocks, Automatic load balancing, per volume i/o tuning
- 22. Gluster: who uses it ? Avail TVN (USA) 400TB for Video on demand, video storage Fido Film (Sweden) visual FX and Animation studio University of Minnesota (USA) 142TB Supercomputing Partners Healthcare (USA) 336TB Integrated health system Origo (Switzerland) open source software development and collaboration platform 22!
- 23. Good • Large amount of data • Access with different protocols • Directly access from applications (api layer) • Disaster recover (better in the next release) • SAN replacement, vm storage Bad • User-space • Low granularity in security setting • High volumes of operations on same file Gluster: good for ... 23!
- 24. 6/23/10! 24! Implementations Implementations Old way Metadata and data in the same place Single stream per file New way Multiple streams are parallel channels through which data can flow Files are striped across a set of nodes in order to facilitate parallel access OSD Separation of file metadata management (MDS) from the storage of file data
- 25. 6/23/10! 25! HDFS: Hadoop HDFS is part of the Apache Hadoop project which develops open-source software for reliable, scalable, distributed computing. Hadoop was inspired by Google’s MapReduce and Google File system
- 26. 6/23/10! 26! HDFS: Google File System “ Design of a file systems for a different environment where assumptions of a general purpose file system do not hold—interesting to see how new assumptions lead to a different type of system…” Key ideas: Component failures are the norm. Huge files (not just the occasional file) Append rather than overwrite is typical Co-design of application and file system API—specialization. For example can have relaxed consistency.
- 27. Map! • Split and mapped in key- value pairs! Combine! • For efficiency reasons, the combiner works directly to map operation outputs .! Reduce! • The files are then merged, sorted and reduced! “Moving Computation is Cheaper than Moving Data” HDFS: MapReduce 27!
- 28. Goals! Scalable: can reliably store and process petabytes.! Economical: It distributes the data and processing across clusters of commonly available computers. ! Efficient: can process data in parallel on the nodes where the data is located.! Reliable: automatically maintains multiple copies of data and automatically redeploys computing tasks based on failures.! HDFS: goals 28!
- 29. HDFS: design 29!
- 30. • An HDFS cluster consists of a single NameNode • It is a master server that manages the file system namespace and regulates access to files by clients. Namenode • Datanode manage storage attached to the system it run on • Applay the map rule of MapReduce Datanodes • File is split into one or more blocks and these blocks are stored in a set of DataNodes Blocks HDFS: components 30!
- 31. 6/23/10! 31! HDFS: features Uniform name space: same path on all workstations Reliability: rw replication, re-balancing, copy in different locations Availability: hot deploy Scalability: server aggregation Administration: HOD Performance: “grid” computation, parallel transfer
- 32. HDFS: who uses it ? Major players Yahoo! A9.com AOL Booz Allen Hamilton EHarmony Facebook Freebase Fox Interactive Media IBM ImageShack ISI Joost Last.fm LinkedIn Metaweb Meebo Ning Powerset (now part of Microsoft) Proteus Technologies The New York Times Rackspace Veoh Twitter … 32!
- 33. Good • Task distribution (Basic GRID infrastructure) • Distribution of content (High throughput of data access ) • Archiving • Etherogenous envirorment Bad • Not General purpose File system • Not Posix Compliant • Low granularity in security setting • Java HDFS: good for ... 33!
- 34. Ceph “Ceph is designed to handle workloads in which tens thousands of clients or more simultaneously access the same file or write to the same directory– usage scenarios that bring typical enterprise storage systems to their knees.” Keys: Seamless scaling — The file system can be seamlessly expanded by simply adding storage nodes (OSDs). However, unlike most existing file systems, Ceph proactively migrates data onto new devices in order to maintain a balanced distribution of data. Strong reliability and fast recovery — All data is replicated across multiple OSDs. If any OSD fails, data is automatically re-replicated to other devices. Adaptive MDS — The Ceph metadata server (MDS) is designed to dynamically adapt its behavior to the current workload. 34!
- 36. • Metadata Storage • Dynamic Subtree Partitioning • Traffic Control Dynamic Distributed Metadata • Data Distribution • Replication • Data Safety • Failure Detection • Recovery and Cluster Updates Reliable Autonomic Distributed Object Storage Ceph: features 36!
- 37. Pseudo-random data distribution function (CRUSH)! Reliable object storage service (RADOS)! Extent B-tree object File System (today btrfs) ! 37! Ceph: features
- 38. Splay Replication • Only after it has been safely committed to disk is a final commit notification sent to the client. Ceph: features 38!
- 39. Good • Scientific application, High throughput of data access • Heavy Read / Write operations • It is the most advance distributed file system Bad • Young (Linux 2.6.34) • Linux only • Complex Ceph: good for … 39!
- 42. 6/23/10! 42! Class Exam What can DFS do for you ? How can you create a Petabyte storage ? How can you make a centralized system log ? How can you allocate space for your user or system, when you have a thousands of users/systems ? How can you retrieve data from everywhere ?
- 43. • Share Documents across a wide network area • Share home folder across different Terminal servers Problem • OpenAFS • Samba Solution • Single ID, Kerberos/ldap • Single file system Results • 800 users • 15 branch offices • File sharing /home dir Usage File sharing 43!
- 44. • Big Storage on a little budget Problem • Gluster Solution • High Availability data storage • Low price Results • 100 TB image archive • Multimedia content for web site Usage Web Service 44!
- 45. • Data from everywhere • Disaster Recover Problems • myS3 • Hadoop / OpenAFS Solution • High Availability • Access through HTTP protocol (REST Interface) • Disaster Recovery Results • Users backup • Application backend • 200 Users • 6 TB Usage Internet Disk: myS3 45!
- 46. • Log concentrator Problem • Hadoop cluster • Syslog-NG Solution • High availability • Fast search • “Storage without limits” Results • Security audit and access control Usage Log concentrator 46!
- 47. • Low cost VM storage • VM self provisioning Problems • GlusterFS • openAFS • Custom provisioning Solution • Auto provisioning • Low cost • Flexible solution Rresults • Development env • Production env Usage Private cloud
- 48. 6/23/10! 48! Conclusion: problems Failure For 10 PB of storage, you will have an average of 22 consumer-grade SATA drives failing per day. Read/write time Each of the 2TB drives takes approximately best case 24,390 seconds to be read and written over the network. Data Replication Data replication is the number of the disk drives, plus difference. Do you have enough bandwidth ? !
- 49. Environment Analysis! • No true Generic DFS! • Not simple move 800TB btw different solutions! Dimension ! • Start with the right size! • Servers number is related to speed needed and number of clients ! • Network for Replication ! Divide system in Class of Service! • Different disk Type ! • Different Computer Type! System Management! • Monitoring Tools! • System/Software Deploy Tools! Conclusion 49!
- 51. OpenAFS! • www.openafs.org! • www.beolink.org! Gluster! • www.gluster.org! Hadoop! • Hadoop.apache.org! • Isabel Drost! Ceph! • ceph.newdream.n et! • Publication! • Mailing list! Links 51!
- 52. I look forward to meeting you… XVII European AFS meeting 2010 PILSEN - CZECH REPUBLIC September 13-15 Who should attend: Everyone interested in deploying a globally accessible file system Everyone interested in learning more about real world usage of Kerberos authentication in single realm and federated single sign-on environments Everyone who wants to share their knowledge and experience with other members of the AFS and Kerberos communities Everyone who wants to find out the latest developments affecting AFS and Kerberos More Info: http://afs2010.civ.zcu.cz/ 6/23/10! 52!