Setting up repositories

Aug 3, 2009Download as PPT, PDF2 likes649 views

1. The document discusses the hardware and software requirements for setting up an institutional repository, noting that repositories can run on a variety of server types from basic to high-powered and require reasonably good server hardware, storage, and memory. 2. It provides examples of specific hardware configurations used by some repositories, including servers from HP, Sun, and Dell with various processors, RAM amounts, and storage capacities. 3. The document states that the repository software installed and resulting user interface are what primarily determine a repository's functionality and appearance to users, giving the example of DSpace which is written in Java and can run on various platforms.

Setting up repositories: Technical Requirements, Repository Software, Metadata & Workflow. Repository services Iryna Kuchma, eIFL Open Access program manager, eIFL.net Presented at “ Open Access: Maximising Research Quality and Impact ” wor kshop, July 2 3 2009, Kathmandu, Nepal

H ardware Repositories can sit on dedicated servers, shared services or as virtual machines on larger servers. In the first few years a basic or moderately specified server will perform sufficiently. (Repositories Support Project)

H ardware 2 In order to re-cycle apparently obsolete computers and transform the ageing PCs into high-speed multi-purpose library terminals using open source software you can use the LTSP guide “ Extending the life of older computer hardware in your library: How to Set Up a Linux Terminal Server Project Installation”, the Birzeit University Library Experience by Dr. Wasel Ghanem, Head of Computer Systems, and Mrs. Diana Sayej-Naser, Library Director, Birzeit University

H ardware 3 http://www. dspace .org/index. php ?/FAQ/FAQ.html#hardware “ There are no specific server requirements for DSpace except UNIX. (Because the application is written in Java, in theory it will run on other platforms as well.) DSpace is built on top of free, open-source tools, such as the Apache Web server, the Tomcat Servlet engine, and the postgreSQL relational database system… The system runs on anything from a laptop to a $500K server, but there are a few general recommendations for hardware architectures…

H ardware 4 http://www. dspace .org/index. php ?/FAQ/FAQ.html#hardware “ For a research university, DSpace requires a reasonably good server and a decent amount of memory and disk storage. Some examples from the community (not necessarily endorsements): HP Server rx2600, powered by dual 64-bit Intel Itanium 2 processors (900MHz), 2GB RAM, 26 GB internal disk storage. HP StorageWorks Modular SAN Array 1000 (msa1000) with a single high-performance controller. Options include a second controller and, with the addition of two more drive enclosures, controls up to 42 Ultra2, Ultra 3, or Ultra320 SCSI drives. Total capacity can be six terabytes…

H ardware 5 http :// www . dspace . org / index . php ?/FAQ/FAQ. html # hardware “ SunFire 280R Server, two 900MHz UltraSPARC-III Cu processors, 8MB E-cache, 2GB memory, two 36GB 10,000rpm HH internal FCAL disk drives, DVD, 436-GB, or 12 x 26.4 Gbyte 10K RPM disks, Sun StorEdge A1000 rackmountable w/ 1 HW RAID controller, 24MB std cache. Dell PowerEdge 2650 with dual Xeon processors (2.4GHz), 2GB RAM, 2x73GB scsi disks. One 2.5TB Apple XServe. A DLT tape library to back up the DB/jsps etc. Of course, your mileage (and costs) will vary depending on what you plan to do with the system… ”

H ardware & Software Apart from the obvious computer equipment needed to set up a repository, e.g. server, storage space, network connection, it is the software that is installed and the web interface for users that determines what a repository looks like for most people (Repositories Support Project)

http :// www . rsp . ac . uk / pubs /briefingpapers- docs /repoadmin- metadata . pdf

Thanks to the SHERPA Team Especially Peter Millington, Technical Development Officer for the slides

Services Updated CVs Back-up of publications Usage and download statistics Publications / bibliographic databases Print-on-demand Management Information Systems Personal web pages and automated publication Lists Marketing Systems Virtual Learning Environment

Thank you ! Questions ? Iryna Kuchma iryna.kuchma[at]eifl.net; www. eifl .net The presentation is licensed with Creative Commons Attribution 3.0 License

This document provides an introduction to HDFS (Hadoop Distributed File System). It discusses what HDFS is, its core components, architecture, and key elements like the NameNode, metadata, and blocks. HDFS is designed for storing very large files across commodity hardware in a fault-tolerant manner and allows for streaming access. While HDFS can handle small datasets, its real power is with large and distributed data.

Hadoop Distributed File SystemAnand Kulkarni

The document discusses the Hadoop Distributed File System (HDFS), which was created by Doug Cutting to address the need for large-scale data processing. HDFS is designed for streaming data across commodity hardware and uses a master/slave architecture with one NameNode master and multiple DataNodes. The NameNode manages the file system namespace and regulates access to files by clients via the DataNodes, which store data blocks and ensure replication for fault tolerance.

Hadoop Distributed File SystemVaibhav Jain

HDFS allows storing large amounts of data across multiple machines by splitting files into blocks and replicating those blocks for reliability. It addresses challenges of big data like volume, velocity, and variety by providing a distributed storage solution that scales horizontally. Traditional systems are limited by network bandwidth, storage capacity of individual machines, and single points of failure. HDFS introduces a scalable architecture with a master NameNode and slave DataNodes that stores data blocks, addressing these issues through data distribution and fault tolerance.

Hdfs architectureAisha Siddiqa

The Hadoop Distributed File System (HDFS) has a master/slave architecture with a single NameNode that manages the file system namespace and regulates client access, and multiple DataNodes that store and retrieve blocks of data files. The NameNode maintains metadata and a map of blocks to files, while DataNodes store blocks and report their locations. Blocks are replicated across DataNodes for fault tolerance following a configurable replication factor. The system uses rack awareness and preferential selection of local replicas to optimize performance and bandwidth utilization.

Modeling, estimating, and predicting Ceph (Linux Foundation - Vault 2015)Lars Marowsky-Brée

This document discusses modeling and predicting performance for Ceph storage clusters. It describes many of the hardware, software, and configuration factors that impact Ceph performance, including network setup, storage nodes, disks, redundancy, placement groups and more. The document advocates for developing standardized benchmarks to better understand Ceph performance under different workloads and cluster configurations in order to answer customers' questions.

Hadoop descriptionHadoop online training

Hadoop is an open source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It provides reliable, scalable storage and processing using the Hadoop Distributed File System (HDFS) and MapReduce programming model. Hadoop has been shown to sort terabytes of data in minutes using thousands of nodes. It is widely used to analyze large, diverse datasets such as web logs, social media, and sensor data.

Ceph Days 2014 Paul Evans Slide DeckDaystromTech

The document discusses building a data lake or data grid using Ceph for storage. It describes how organizations are facing large amounts of data being generated too quickly and in too many variants. A data lake or data grid provides a solution by storing raw data in its native format and allowing flexible processing. Ceph is proposed as the underlying storage layer because it is Hadoop-native, offers locality-awareness and erasure coding for efficiency and scalability. The data lake or grid architecture with Ceph provides a robust, expandable yet simple solution to store growing amounts of data and effectively manage it in a decentralized manner.

Ceph Day Berlin: Scaling an Academic CloudCeph Community

This document summarizes a presentation on scaling academic clouds with Ceph. It discusses using Ceph as the storage component of software-defined datacenters and clouds to provide automated, agile, and efficient storage. Examples are provided of large Ceph deployments at universities, including a collaboration between 4 universities with a 6.9PB Ceph cluster for shared microbial data. Design considerations for Ceph infrastructure are also covered, such as storage node configuration, use of SSDs, network topology, and site infrastructure needs. The document concludes with an overview of Dell's Ceph reference architecture offering hardware, software, and services.

An introduction to Big-Data processing applying hadoopAmir Sedighi

Hadoop distributed file systemAmeya Vijay Gokhale

Hadoop Distributed File System (HDFS) is an open-source software framework that provides distributed storage and processing of large datasets across clusters of commodity hardware. HDFS has two main components - HDFS for storage and MapReduce for distributed processing. HDFS uses a master-slave architecture with a NameNode master and DataNodes slaves. The NameNode manages the file system namespace and metadata, while DataNodes store data blocks and report to the NameNode.

Hadoop Distributed File Systemelliando dias

Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has very large files (over 100 million files) and is optimized for batch processing huge datasets across large clusters (over 10,000 nodes). HDFS stores multiple replicas of data blocks on different nodes to handle failures. It provides high aggregate bandwidth and allows computations to move to where data resides.

Research computing at ILRIILRI

This document summarizes the history and current state of research computing at ILRI. It describes how the computing environment has evolved from a cluster of 32 dual-core nodes in 2003 to a current cluster of 152 cores and 32 TB of storage. The primary computing platforms are managed using Slurm for job scheduling and modules for managing applications. Data and applications are made available across all nodes using GlusterFS for scalable network storage. User authentication is handled through LDAP and identities are consistent across all systems.

Hadoop HDFS Architeture and Designsudhakara st

HDFS is a distributed file system designed for storing very large data files across commodity servers or clusters. It works on a master-slave architecture with one namenode (master) and multiple datanodes (slaves). The namenode manages the file system metadata and regulates client access, while datanodes store and retrieve block data from their local file systems. Files are divided into large blocks which are replicated across datanodes for fault tolerance. The namenode monitors datanodes and replicates blocks if their replication drops below a threshold.

HDFS introductioninjae yeo

This document provides an overview of Hadoop and its core components HDFS and MapReduce. It describes how HDFS uses a master/slave architecture with a single NameNode master and multiple DataNode slaves to store and retrieve data in a fault-tolerant manner. The NameNode manages the filesystem namespace and monitors data replication, while DataNodes store data blocks and perform read/write operations. It also discusses high availability techniques for the NameNode and core functions like block placement, garbage collection and stale replica detection in HDFS.

Hadoop Research Shreyansh Ajit kumar

The Hadoop platform uses the Hadoop Distributed File System (HDFS) to reliably store large files across thousands of nodes. It requires a minimum of computing power, memory, storage, and network bandwidth. A recommended cluster size depends on linear relationships between resources and efficiency. Dashboards can be created using data extracted from HDFS to SQL for analytics. The Hadoop architecture is designed to scale easily by adding more servers as data and workloads increase.

Cassandra useful featuresSandeep Sharma IIMK Smart City,IoT,Bigdata,Cloud,BI,DW

This document discusses features of self-tuning row caching in Cassandra including: 1) Caching entire merged rows to avoid fragment caching and reduce CPU and disk IO. 2) Automatically determining memory usage for each row cache. 3) Supporting SSD and HDD volumes in the same cluster by pinning tables to optimal media. 4) Providing the most recent data for requested rows across the distributed system through locking and commit logs.

Hadoop HDFS by rohitkapakapa rohit

HDFS is a distributed file system designed for storing very large data sets reliably and efficiently across commodity hardware. It has three main components - the NameNode, Secondary NameNode, and DataNodes. The NameNode manages the file system namespace and regulates access to files. DataNodes store and retrieve blocks when requested by clients. HDFS provides reliable storage through replication of blocks across DataNodes and detects hardware failures to ensure data is not lost. It is highly scalable, fault-tolerant, and suitable for applications processing large datasets.

March 2011 HUG: HDFS FederationYahoo Developer Network

HDFS Federation allows HDFS to scale beyond the limitations of a single namenode by federating the namespace and block management across multiple independent namenodes. This simplifies the design and implementation compared to a distributed namenode approach. Existing single namenode deployments are not impacted and can continue running as is. Federation preserves the robustness of individual namenodes while scaling to more namenodes. It generalizes the block storage layer to allow multiple namenodes to share the same blocks storage. This improves isolation and availability while providing a simpler way to scale HDFS in the near term.

Pillars of Heterogeneous HDFS StoragePete Kisich

1. Hadoop HDFS supports tiered storage using different storage policies (e.g. Hot, Warm, Cold) that map files to specific disk types (e.g. SSD, disk, archive). 2. A file's storage policy and disk type can be set individually or inherited from its parent directory, and moving a file between tiers involves changing its storage policy and triggering a mover process. 3. FactorData HDFSplus is a tool that sits outside a Hadoop cluster and collects metadata to generate reports on data usage, small files, replication, etc. It can then automate optimizing storage by archiving, promoting, or changing replication of data based on user-defined rules

Word count program execution steps in hadoopjijukjoseph

Redis vs MemcachedGaurav Agrawal

Redis and Memcached are both open-source, in-memory key-value data stores used for caching, but they differ in features and use cases. Redis stores data in memory and can persist to disk, supports data types beyond strings like lists and sets, and allows replication. Memcached only stores data in memory and does not support persistence or replication. While both are fast, Redis is more full-featured but requires more memory, making Memcached better for simple caching use cases with high traffic.

Cassandra architectureT Jake Luciani

The document compares Cassandra and PostgreSQL when deployed at scale. It outlines that Cassandra uses a peer-to-peer and masterless architecture with tunable consistency levels and can scale up and down easily. Cassandra also integrates tightly with Hadoop and offers the CQL query language similar to SQL. The document provides examples of basic SQL commands and their Cassandra equivalents using the CQL language.

SanDisk: Persistent Memory and CassandraDataStax Academy

Hadoop HDFS NameNode HAHanborq Inc.

This document outlines and compares two NameNode high availability (HA) solutions for HDFS: AvatarNode used by Facebook and BackupNode used by Yahoo. AvatarNode provides a complete hot standby with fast failover times of seconds by using an active-passive pair and ZooKeeper for coordination. BackupNode has limitations including slower restart times of 25+ minutes and supporting only two-machine failures. While it provides hot standby for the namespace, block reports are sent only to the active NameNode, making it a semi-hot standby solution. The document also briefly mentions other experimental HA solutions for HDFS.

MySQL HAKris Buytaert

MySQL Cluster provides high availability through data replication across multiple nodes, automatic failover, and synchronous replication to ensure data integrity, but it has limitations in that the entire database must reside in memory and database size is restricted by available memory. Other options for high availability with MySQL include using MySQL proxy to split reads and writes across nodes, replication with multi-master setups, and technologies like DRBD to replicate data for recovery. Planning for failures, keeping implementations simple, and separating data and connectivity high availability are important principles for highly available MySQL architectures.

Introduction of mesos persistent storageZhou Weitao

What Brian Cant Never Taught You About MetadataDrew McLellan

The document is a presentation by Drew McLellan about metadata. It discusses how what Brian Cant taught about metadata may be wrong and how the presenter learned to accept metadata. It also lists some topics that will be covered, including metadata, HTML, robots from children's television programming from the 1970s/80s, tofu, truth, honesty, and made up rules stated as facts. It provides background about the presenter and their websites before planning to discuss Brian Cant and what he taught, as well as other sources of information beyond Cant.

Current Accounting and Reporting Developments Webcast Series Q4PwC

This document provides a summary of a webcast on current accounting and reporting developments from the fourth quarter of 2015. It covered topics like non-GAAP measures, segments, risk disclosures, equity method investments, debt modifications, and standards effective in 2016. The webcast was intended to provide 1.5 hours of CPE credit and included polls to verify attendance. It was presented by several partners from PwC and was meant to highlight issues discussed at a recent AICPA conference on SEC and PCAOB developments.

More Related Content

What's hot (20)

Ceph Days 2014 Paul Evans Slide DeckDaystromTech

Ceph Day Berlin: Scaling an Academic CloudCeph Community

An introduction to Big-Data processing applying hadoopAmir Sedighi

Hadoop distributed file systemAmeya Vijay Gokhale

Hadoop Distributed File Systemelliando dias

Research computing at ILRIILRI

Hadoop HDFS Architeture and Designsudhakara st

HDFS introductioninjae yeo

Hadoop Research Shreyansh Ajit kumar

Cassandra useful featuresSandeep Sharma IIMK Smart City,IoT,Bigdata,Cloud,BI,DW

Hadoop HDFS by rohitkapakapa rohit

March 2011 HUG: HDFS FederationYahoo Developer Network

Pillars of Heterogeneous HDFS StoragePete Kisich

Word count program execution steps in hadoopjijukjoseph

Redis vs MemcachedGaurav Agrawal

Cassandra architectureT Jake Luciani

SanDisk: Persistent Memory and CassandraDataStax Academy

Hadoop HDFS NameNode HAHanborq Inc.

MySQL HAKris Buytaert

Introduction of mesos persistent storageZhou Weitao

Ceph Days 2014 Paul Evans Slide DeckDaystromTech

Ceph Day Berlin: Scaling an Academic CloudCeph Community

An introduction to Big-Data processing applying hadoopAmir Sedighi

Hadoop distributed file systemAmeya Vijay Gokhale

Hadoop Distributed File Systemelliando dias

Research computing at ILRIILRI

Hadoop HDFS Architeture and Designsudhakara st

HDFS introductioninjae yeo

Hadoop Research Shreyansh Ajit kumar

Cassandra useful featuresSandeep Sharma IIMK Smart City,IoT,Bigdata,Cloud,BI,DW

Hadoop HDFS by rohitkapakapa rohit

March 2011 HUG: HDFS FederationYahoo Developer Network

Pillars of Heterogeneous HDFS StoragePete Kisich

Word count program execution steps in hadoopjijukjoseph

Redis vs MemcachedGaurav Agrawal

Cassandra architectureT Jake Luciani

SanDisk: Persistent Memory and CassandraDataStax Academy

Hadoop HDFS NameNode HAHanborq Inc.

MySQL HAKris Buytaert

Introduction of mesos persistent storageZhou Weitao

Viewers also liked (20)

What Brian Cant Never Taught You About MetadataDrew McLellan

Current Accounting and Reporting Developments Webcast Series Q4PwC

Real-World Data Governance Webinar: Data Governance and Metadata Best PracticeDATAVERSITY

Best practices are defined as a method or technique that has consistently shown results superior to those achieved with other means, and that is used as a benchmark. In addition the definition goes on to say that a "best" practice can evolve to become better as improvements are discovered. A best practice can also be considered a target behavior to which you can compare your organization to deliver the actionable steps you can follow to achieve best practice. In this Real-World Data Governance webinar, Bob Seiner focuses on defining, assessing and deploying Data Governance and metadata best practice that will move your organization in the best possible direction of success. Participants can expect to leave the webinar with a working list that can be used for self or contracted assessment. This session will cover: Criteria to Determine if Something is Best Practice Development of Data Governance Best Practice The Process to Complete the Best Practice Assessment The Delivery of the Assessment to Management How to Use the Assessment to Deliver Action

Preservation MetadataDigitalPreservationEurope

Metadata can play a vital role in enabling the effective management, discovery, and re-usability of digital information. Digital preservation metadata provides provenance information, supports and documents preservation activity, identifies technical features, and aids in verifying the authenticity of a digital object. This presentation gives and introduction to Digital preservation matadata and preservation metada in practise. Presentation was delivered during the joint DPE/Planets/CAPAR/nestor training event, ‘The Preservation challenge: basic concepts and practical applications’ (Barcelona, March 2009)

Better Cross-Channel Experiences With Metadata - Information Architecture Sum...aungstad

Metadata provides a foundation for the cross-channel experiences we consume in the dawn of the 21st century by enabling consistency, context and interoperability. While Metadata is key to many of the User Experience methods we use – from triggering Content Strategy workflows to enabling faceted search through groups, sorts and filters, it remains a mystery to many of us. This session is a crash course on Metadata in the 21st century for UX Designers. We'll look at recent developments in the field, and build an understanding of how Metadata enables the exchange of information between parties, channels, systems and platforms through semantic, syntax and lexical specifications. Attendees will understand how Metadata fits into UX methods and tools they already know – using an ontology to support a mental model for example. This session is a must for all UX Designers looking to build meaningful cross-channel experiences in an increasingly interlinked world.

Metadata Mapping & CrosswalksNikos Palavitsinis, PhD

Hansen Metadata for Institutional RepositoriesNational Information Standards Organization (NISO)

Data Systems Integration & Business Value Pt. 1: MetadataDATAVERSITY

Certain systems are more data focused than others. Usually their primary focus is on accomplishing integration of disparate data. In these cases, failure is most often attributable to the adoption of a single pillar (silver bullet). The three webinars in the Data Systems Integration and Business Value series are designed to illustrate that good systems development more often depends on at least three DM disciplines (pie wedges) in order to provide a solid foundation. Much of the discussion of metadata focuses on understanding it and the associated technologies. While these are important, they represent a typical tool/technology focus and this has not achieved significant results to date. A more relevant question when considering pockets of metadata is: Whether to include them in the scope organizational metadata practices. By understanding what it means to include items in the scope of your metadata practices, you can begin to build systems that allow you to practice sophisticated ways to advance their data management and supported business initiatives. After a bit of practice in this manner you can position your organization to better exploit any and all metadata technologies.

Semantic Web, Cataloging, & Metadatarobin fay

Metadata Creation In OBIEEiWare Logic Technologies Pvt. Ltd.

Does metadata matter?Eduserv Foundation

The document discusses the history and importance of metadata. It describes how metadata standards have evolved over time, from MARC for library catalogs to Dublin Core for web pages to RDF for the semantic web. It also discusses how metadata is now commonly used for repositories of research publications and data to help discover and manage institutional assets. While search engines ignore most embedded metadata, standards and policies around metadata have still spread widely across domains like education, libraries, and cultural heritage.

The linked open government data and metadata lifecycleOpen Data Support

This document discusses the lifecycle of linked open government data and metadata. It begins by examining existing data and metadata lifecycles, noting that they primarily focus on the supply side. It then presents a hybrid lifecycle model that includes both supply and demand sides. The supply side covers the selection, modeling, publishing and linking of data and metadata by governments. The demand side involves finding, integrating, reusing and providing feedback on open data by consumers. The document also provides best practices for publishing data and metadata at various stages of the lifecycle.

What is the role of cloud computing, web 2.0, and web 3.0 semantic technologi...Mills Davis

The US has a new administration that values transparency, citizen participation, collaboration, information sharing, and internet technology. This presentation maps the role of information and communication technologies (specifically, cloud computing, Web 2.0, and Web 3.0 semantic technologies) in the evolution of government information systems from e-gov (silos with web front ends) to connected governance (e.g. distributed social computing environments for collaborative work, information sharing, knowledge management, and participatory decision-making.)

RWDG Webinar: Metadata to Support Data GovernanceDATAVERSITY

This document describes a webinar on using metadata to support data governance. It provides definitions of key terms like data governance, metadata, and non-invasive data governance. It explains that metadata is a byproduct of good governance practices like formalizing accountability and standards. The webinar will cover selecting important initial metadata, using metadata to support the governance program, and incorporating governance into processes to manage metadata. It promotes integrating governance roles and responsibilities into existing methodologies.

Metadata is a Love Note to the FutureRachel Lovinger

The document discusses the importance of metadata for archiving digital content and history. It describes how Jason Scott transformed from a "metadata skeptic" to a "metadata warrior" after his experiences rescuing data from Geocities. Proper metadata made the rescued data more useful, efficient to archive, and prevented duplication. The document advocates for taking a long-term view of digital content and using metadata to ensure information can be discovered and understood in the future.

Plantas Tìpicas da Caatingagoretteleal

Turning big data into big revenuePwC

Web 3.0 The Semantic WebHatem Mahmoud

The document discusses the Semantic Web as Web 3.0. It explains that while current web pages use HTML to describe structure, not meaning, the Semantic Web aims to allow computers to understand the meaning behind information by recognizing things like people, places, events. This is done through techniques like embedding semantic annotations directly into data using standards like RDFa, microformats, and querying data with SPARQL. The Semantic Web will enable new applications by making the web more machine-readable.

Data warehouse architecturepcherukumalla

Grooming PresentationNikhil Mathur

Personal grooming and appearance are essential for working in the aviation and hospitality industries. Maintaining good hygiene, dressing professionally, and paying attention to details like hair, nails, teeth and attire help make a strong first impression and influence how people perceive you. Proper grooming habits demonstrate traits like professionalism, intelligence and credibility that are important in these customer-facing roles. The document provides extensive guidelines on maintaining a neat appearance, from dressing appropriately to practicing good hygiene.

What Brian Cant Never Taught You About MetadataDrew McLellan

Current Accounting and Reporting Developments Webcast Series Q4PwC

Real-World Data Governance Webinar: Data Governance and Metadata Best PracticeDATAVERSITY

Preservation MetadataDigitalPreservationEurope

Better Cross-Channel Experiences With Metadata - Information Architecture Sum...aungstad

Metadata Mapping & CrosswalksNikos Palavitsinis, PhD

Hansen Metadata for Institutional RepositoriesNational Information Standards Organization (NISO)

Data Systems Integration & Business Value Pt. 1: MetadataDATAVERSITY

Semantic Web, Cataloging, & Metadatarobin fay

Metadata Creation In OBIEEiWare Logic Technologies Pvt. Ltd.

Does metadata matter?Eduserv Foundation

The linked open government data and metadata lifecycleOpen Data Support

What is the role of cloud computing, web 2.0, and web 3.0 semantic technologi...Mills Davis

RWDG Webinar: Metadata to Support Data GovernanceDATAVERSITY

Metadata is a Love Note to the FutureRachel Lovinger

Plantas Tìpicas da Caatingagoretteleal

Turning big data into big revenuePwC

Web 3.0 The Semantic WebHatem Mahmoud

Data warehouse architecturepcherukumalla

Grooming PresentationNikhil Mathur

Similar to Setting up repositories (20)

Accelerate Big Data Processing with High-Performance Computing TechnologiesIntel® Software

Big Data Meets HPC - Exploiting HPC Technologies for Accelerating Big Data Pr...inside-BigData.com

In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: Big Data Meets HPC - Exploiting HPC Technologies for Accelerating Big Data Processing. "This talk will provide an overview of challenges in accelerating Hadoop, Spark and Memcached on modern HPC clusters. An overview of RDMA-based designs for Hadoop (HDFS, MapReduce, RPC and HBase), Spark, Memcached, Swift, and Kafka using native RDMA support for InfiniBand and RoCE will be presented. Enhanced designs for these components to exploit NVM-based in-memory technology and parallel file systems (such as Lustre) will also be presented. Benefits of these designs on various cluster configurations using the publicly available RDMA-enabled packages from the OSU HiBD project (http://hibd.cse.ohio-state.edu) will be shown." Watch the video: https://youtu.be/iLTYkTandEA Learn more: http://web.cse.ohio-state.edu/~panda.2/ and http://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

HPC and cloud distributed computing, as a journeyPeter Clapham

Research OnThe IT Infrastructure Of Globe Pharmaceutical Ltd - All ServerZiaul Hoque Prince

The document summarizes the IT infrastructure of Globe Pharmaceuticals Ltd. It includes details about the rack servers, backup servers, mail servers, firewall servers, Oracle's database management server, production printers, process servers, and Cisco switches used by the company. The rack servers are Dell PowerEdge R620 servers running Windows Server or Linux. Backup is provided by Dell PowerVault MD1220 direct attached storage arrays. Mail services use Novell GroupWise. Firewalls utilize HP F5000 appliances. Oracle databases are managed by Oracle SuperCluster M6 servers. Printing needs are met with HP T300 printers. Process servers use HP Proliant DL380p Gen8 servers. Networking leverages Cisco Catalyst 6800 series switches

Walk Through a Software Defined Everything PoCCeph Community

This document summarizes a proof of concept for a software defined data center using OpenStack and Midokura MidoNet software defined networking. The POC used 4 controllers, 8 Ceph storage nodes, and 16 compute nodes with Midokura providing logical layer 2-4 networking services. Key lessons learned included planning the underlay network configuration, optimizing Zookeeper connections, and improving OpenStack deployment processes which can be complex. Performance testing showed Ceph throughput was higher for reads than writes and SSD journaling improved IOPS. The streamlined workflow provided by the software defined infrastructure could help reduce costs and management complexity for organizations.

From Rack scale computers to Warehouse scale computersRyousei Takano

This document discusses the transition from rack-scale computers to warehouse-scale computers through the disaggregation of technologies. It provides examples of rack-scale architectures like Open Compute Project and Intel Rack Scale Architecture. For warehouse-scale computers, it examines HP's The Machine project using application-specific cores, universal memory, and photonics fabric. It also outlines UC Berkeley's FireBox project utilizing 1 terabit/sec optical fibers, many-core systems-on-chip, and non-volatile memory modules connected via high-radix photonic switches.

Making clouds: turning opennebula into a productCarlo Daffara

OpenNebulaConf 2013 - Making Clouds: Turning OpenNebula into a Product by Car...OpenNebula Project

What does it takes to bring innovations like private clouds to small and medium enterprises? In the course of this talk we will present our experience in creating a self-service toolkit for creating a complete virtualization and cloud platform based on OpenNebula, as well as our experience gathered in tens of installations of all sizes. From scalable storage (with benchmarks!) to autonomic optimization, we will present what in our view is needed to bring private clouds to everyone, what components and additions we created to better solve our customers’ problems (from replacing industrial control systems to medium scale virtual desktop infrastructures), and why OpenNebula has been chosen over other competing cloud toolkits. Bio: Carlo Daffara the Technical director of Cloudweavers, and formerly head of research and development at Conecta, a consulting firm specializing in open source systems and distributed computing; Italian member of the European Working Group on Libre Software and co-coordinator of the working group on SMEs of the EU ICT task force on competitiveness. Since 1999, works as evaluator for IST programme submissions in the field of component-based software engineering, GRIDs and international cooperation. Coordinator of the open source platforms technical area of the IEEE technical committee on scalable computing, co-chair of the SIENA EU cloud initiative roadmap editorial board and part of the editorial review board of the International Journal of Open Source Software & Processes (IJOSSP).

Making Clouds: Turning OpenNebula into a ProductNETWAYS

Big Data and its emergencekoolkalpz

The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...OpenStack

Audience Level Intermediate Synopsis High performance computing and cloud computing have traditionally been seen as separate solutions to separate problems, dealing with issues of performance and flexibility respectively. In a diverse research environment however, both sets of compute requirements can occur. In addition to the administrative benefits in combining both requirements into a single unified system, opportunities are provided for incremental expansion. The deployment of the Spartan cloud-HPC hybrid system at the University of Melbourne last year is an example of such a design. Despite its small size, it has attracted international attention due to its design features. This presentation, in addition to providing a grounding on why one would wish to build an HPC-cloud hybrid system and the results of the deployment, provides a complete technical overview of the design from the ground up, as well as problems encountered and planned future developments. Speaker Bio Lev Lafayette is the HPC and Training Officer at the University of Melbourne. Prior to that he worked at the Victorian Partnership for Advanced Computing for several years in a similar role.

Sanger HPC infrastructure Report (2007)Guy Coates

Sharing High-Performance Interconnects Across Multiple Virtual Machinesinside-BigData.com

In this deck from the Stanford HPC Conference, Mohan Potheri from VMware presents: Sharing High-Performance Interconnects Across Multiple Virtual Machines. "Virtualized devices offer maximum flexibility: sharing of hardware between virtual machines, the use of VMware vMotion to handle migration and take snapshots. However, when performance is the most critical requirement there are other options. VMware Direct Path I/O delivers excellent performance, but only for a single virtual machine. Single root I/O virtualization (SR-IOV), on the other hand, offers the performance of pass-through mode while allowing devices to be shared by multiple virtual machines. This session introduces SR-IOV, explains how it is enabled in VMware vSphere, and provides details of specific use cases that important for machine learning and high-performance computing. It includes performance comparisons that demonstrate the benefits of SR-IOV and information on how to configure and tune these configurations." Watch the video: https://youtu.be/-iYYmsBw8SU Learn more: https://www.vmware.com and http://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Accelerating Apache Hadoop through High-Performance Networking and I/O Techno...DataWorks Summit/Hadoop Summit

The document discusses accelerating Apache Hadoop through high-performance networking and I/O technologies. It describes how technologies like InfiniBand, RoCE, SSDs, and NVMe can benefit big data applications by alleviating bottlenecks. It outlines projects from the High-Performance Big Data project that implement RDMA for Hadoop, Spark, HBase and Memcached to improve performance. Evaluation results demonstrate significant acceleration of HDFS, MapReduce, and other workloads through the high-performance designs.

Oracle Database Appliance X5Amro Ibrahim

The Oracle Database Appliance X5-2 is a fully integrated high availability database appliance that features: - Two Oracle Linux servers with 18-core Intel Xeon processors and 256GB RAM each - Shared direct-attached storage with 64TB capacity that is double or triple mirrored - InfiniBand interconnect for high availability and redundancy - Software including Oracle Database Enterprise Edition and Oracle Real Application Clusters for active-active or active-passive failover - Management software that simplifies deployment, patching, and diagnostics

Acunu Whitepaper v1Acunu

The document describes Acunu's data platform, which reengineers the storage stack in the Linux kernel to take advantage of massive datasets and scale-out commodity hardware. Acunu provides high performance storage solutions for NoSQL databases like Apache Cassandra through its open source Acunu Storage Core. It also allows multiple data stores to interoperate on the same data through versioning and isolation tools. Acunu's research has led to innovations like fast full data versioning and data structures optimized for SSD performance.

Designing High-Performance and Scalable Middleware for HPC, AI and Data ScienceObject Automation

This document discusses designing high-performance middleware for HPC, AI, and data science applications. It provides an overview of the MVAPICH2 project, which develops an open-source MPI library supporting modern HPC architectures and networking technologies. MVAPICH2 aims to provide a converged software stack for HPC, deep learning, and data science through libraries like MVAPICH2, HiDL, and HiBD. The document outlines challenges in communication library design for exascale systems and MVAPICH2's architecture supporting programming models across domains.

Ceph Day Melbourne - Walk Through a Software Defined Everything PoCCeph Community

This document summarizes a proof of concept for a software defined everything architecture using OpenStack and Midokura software defined networking. The objectives are to abstract compute, network and storage resources and dynamically allocate them as needed. It aims to provide scalability, availability, isolation, metering and automation of infrastructure. The proof of concept uses OpenStack for compute with Ceph for storage, and Midokura for software defined networking to provide logical switching, routing, firewalling and load balancing. It details the architecture, infrastructure, configuration and lessons learned from the implementation.

OpenNebulaConf2017EU: Hyper converged infrastructure with OpenNebula and Ceph...OpenNebula Project

Hp Integrity ServersEddy Jennekens