Building a Scalable and Modern Infrastructure at CARFAX

Aug 5, 2014Download as PPTX, PDF7 likes5,661 views

The document discusses CARFAX's transition from a proprietary key-value store to MongoDB. It describes CARFAX's production MongoDB environment including 12 sharded servers with 128GB RAM each. It details how CARFAX loads millions to billions of records per day via distributed processing. It also discusses implementing high availability reads through tagging data centers and replica sets to scale to millions of reports per day. The presentation emphasizes automating processes and sharding early and often for scaling their MongoDB deployment.

A Scalable and Modern
Infrastructure at CARFAX

About Me
• Jai Hirsch – Senior Systems Architect, Data
Technologies at CARFAX
• Long-time Java and Database Developer
• Data and Distributed Processing Enthusiast
• Github: https://github.com/JaiHirsch
Twitter: @JaiHirsch https://twitter.com/JaiHirsch
LinkedIn: http://www.linkedin.com/pub/jai-hirsch/8/a89/335

“CARFAX helps millions of people buy and sell used cars with
more confidence”

NoSQL Before it Was
Cool
Proprietary Key Value Store on
OpenVMS Developed by
CARFAX in 1984

Why MongoDB?
Legacy structures mapped to
documents
High availability using replica sets
Platform Independence
Support

MongoDB at CARFAX
Our Production Environment
The Legacy Database and High
Volume Loads
High Availability Reads

Server Deployment
AUTOMATE
AUTOMATE
AUTOMATE
AUTOMATE

Server Configuration
12 Shards with two spare
servers racked for failover
• OS: Linux
• MongoDB 2.4.9
• 128 GIGs of RAM
• 1.8 TB of Drive Space
• 10K RPM SAS Drives

Building a Scalable and Modern Infrastructure at CARFAX

Loading Millions to Billions of Records per
Day
AUTOMATE
AUTOMATE
AUTOMATE
AUTOMATE

First Attempt To Load Was
Completely CPU Bound

Not Acceptable!
45 Days to
Backload the
Legacy Database

Acceptable!
Billion+ inserts per
Day!
9 Days to Backload

The MongoDB
Implementation
13 billion+ documents
1.5 billion+ new documents per
year
 Document size: ~ 795 Bytes
 VHR uses 200+ documents

Millions of Reports per
Day
AUTOMATE
AUTOMATE
AUTOMATE

Each Data
center is
Tagged
Each Replica
Set is Tagged

Lets Wrap It Up
Don’t buy a used car without a
CARFAX report
➢Grok your data and working set
➢Architect for your load volume
➢Scale your reads to meet demand
29

Keys To Success
➢AUTOMATE EVERYTHING
➢Test Many Configurations
➢Grid Computing is Awesome
➢Shard Early, Shard Often

Friends Don’t
Let Friends Use
Default Ulimits!

Thank You!
The migration was a
success due to the
incredible teams at
CARFAX and MongoDB
We are always looking
for great people to join
us.
www.carfax.com/careers

This session will be a case study of eBay’s experience running MongoDB for project Zoom, in which eBay stores all media metadata for the site. This includes references to pictures of every item for sale on eBay. This cluster is eBay's first MongoDB installation on the platform and is a mission critical application. Yuri Finkelstein, an Enterprise Architect on the team, will provide a technical overview of the project and its underlying architecture.

A Mobile-First, Cloud-First Stack at PearsonMongoDB

Pearson has transitioned to using open source, NoSQL, and MongoDB as the foundation of its technology stack. MongoDB was chosen because it is open source, agile, scalable, and has fast reads and queries. Pearson uses MongoDB for identity and access management, adaptive learning and analytics, and its activity framework. Going forward, Pearson aims to extend MongoDB globally while addressing challenges around configuration, licensing, and developer adoption of NoSQL.

Mongo db 3.4 OverviewNorberto Leite

MongoDB 3.2 Feature PreviewNorberto Leite

MongoDB Certification Study Group - May 2016Norberto Leite

An Enterprise Architect's View of MongoDBMongoDB

This document discusses how MongoDB can help enterprises meet modern data and application requirements. It outlines the many new technologies and demands placing pressure on enterprises, including big data, mobile, cloud computing, and more. Traditional databases struggle to meet these new demands due to limitations like rigid schemas and difficulty scaling. MongoDB provides capabilities like dynamic schemas, high performance at scale through horizontal scaling, and low total cost of ownership. The document examines how MongoDB has been successfully used by enterprises for use cases like operational data stores and as an enterprise data service to break down silos.

Prepare for Peak Holiday Season with MongoDBMongoDB

This document discusses preparing for the holiday season by providing a seamless customer experience. It covers expected trends for the 2014 holiday season including increased spending and an extended shopping window. The opportunity is to provide personalized and relevant experiences for customers. The document then provides an overview of how MongoDB can be used to power various retail functions like product catalogs, real-time inventory and orders, and consolidated customer views to enable a modern seamless retail experience. Technical details are discussed for implementing product catalogs and real-time inventory using MongoDB.

Experian Health: Moving Universal Identity Manager from ANSI SQL to MongoDBMongoDB

Speaker: Sean Reisz, Senior Software Architect, Experian Health Speaker: Dimuthu Wijetilleke, Senior Software Developer, Experian Health Track: RDBMS to MongoDB Experian Health provides industry-leading revenue cycle management, identity management, patient engagement and care management solutions to power opportunities in the healthcare industry. The company has developed the Universal Identity Manager (UIM) platform to match records and across healthcare clients to address duplicate record challenges and facilitate information exchange within and across healthcare enterprises. The product was originally architected around a standard ANSI SQL database. This solution worked for a time, but volume and performance requirements, as well as increasing complexity of the data model, prompted the exploration of other options. In this discussion, we will cover how we were able to modify the existing SQL structure to take advantage of MongoDB’s document object model. We will also discuss improvements that MongoDB allowed us to implement to simplify document retrieval and modification. Audience members only need basic MongoDB understanding, but should have a grasp of the core concepts. What You Will Learn: - 1. The steps to migrate from a traditional SQL architecture to a MongoDB architecture, including improvements and iterations *to simplify the migration. - Understand the design of a reference architecture for a performant and highly available application using MongoDB - Architect performance and scale comparisons between traditional SQL database schemas and the MongoDB document model

Webinar: What's New in MongoDB 3.2MongoDB

Webinar: Faster Big Data Analytics with MongoDBMongoDB

Learn how to leverage MongoDB and Big Data technologies to derive rich business insight and build high performance business intelligence platforms. This presentation includes: - Uncovering Opportunities with Big Data analytics - Challenges of real-time data processing - Best practices for performance optimization - Real world case study This presentation was given in partnership with CIGNEX Datamatics.

MongoDB Operations for DevelopersMongoDB

MongoDB vs Mysql. A devops point of viewPierre Baillet

Introduction to MongoDB EnterpriseMongoDB

MongoDB is a leading database technology that combines the foundations of RDBMS with the innovations of NoSQL, allowing organizations to simultaneously boost productivity and lower TCO. MongoDB Enterprise Advanced is a finely-tuned package of advanced software, enterprise-grade support, and other services designed to accelerate your success with MongoDB in every stage of your app lifecycle, from early development to the scale-out of mission-critical production environments. With the release of 3.2, MongoDB Enterprise Advanced now includes: MongoDB Ops Manager 2.0 MongoDB Compass, the MongoDB GUI MongoDB Connector for Business Intelligence Encrypted Storage Engine In-Memory Storage Engine (beta) Attend this webinar to learn how MongoDB Enterprise Advanced can help you get to market faster and de-risk your mission critical deployments.

When to Use MongoDB...and When You Should Not...MongoDB

MongoDB is well-suited for applications that require: - A flexible data model to handle diverse and changing data sets - Strong performance on mixed workloads involving reads, writes, and updates - Horizontal scalability to grow with increasing user needs and data volume Some common use cases that leverage MongoDB's strengths include mobile apps, real-time analytics, content management, and IoT applications involving sensor data. However, MongoDB is less suited for tasks requiring full collection scans under load, high write availability, or joins across collections.

3 scenarios when to use MongoDB!Edureka!

The document discusses MongoDB and provides an overview of its features and use cases. It begins with definitions of NoSQL and SQL databases, and the CAP theorem. It then covers MongoDB's features such as document-oriented storage, indexing, replication for high availability, and sharding for horizontal scaling. Examples are given of companies like Expedia using MongoDB for real-time personalized offers, flexible data models, and indexing to power travel search features. In conclusion, three scenarios are presented where MongoDB is well-suited: when dynamic schemas are needed, for flexible data modeling, and to enable powerful indexing.

eHarmony - Messaging Platform with MongoDB Atlas MongoDB

eHarmony is moving their messaging platform to MongoDB Atlas to improve performance and scalability. They are redesigning their 18 step communication flow into a simpler real-time chat system. This will require restructuring their relational database tables into a flexible NoSQL schema in MongoDB Atlas. They modeled the data as collections for conversations, chat history, and recently asked questions. MongoDB Atlas provides high availability, automatic scaling, and worry-free management. Load testing showed performance and latency improvements over their on-premise solution. Monitoring tools in Atlas will provide visibility into key metrics like response times, storage usage, and traffic volumes to support over 300 million users.

NoSQL benchmarkingPrasoon Kumar

Securing Your MongoDB DeploymentMongoDB

Security is more critical than ever with new computing environments in the cloud and expanding access to the internet. There are a number of security protection mechanisms available for MongoDB to ensure you have a stable and secure architecture for your deployment. We'll walk through general security threats to databases and specifically how they can be mitigated for MongoDB deployments. Topics will include general security tools and how to configure those for MongoDB, an overview of security features available in MongoDB, including LDAP, SSL, x.509 and Authentication.

MongoDB Introduction talk at Dr Dobbs Conference, MongoDB Evenings at Bangalo...Prasoon Kumar

MongoDB is a leading nosql database. It is horizonatally scalable, document datastore. In this introduction given at Dr Dobbs Conference, Bangalore and Pune in April 2014, I show schema design with an example blog application and Python code snippets. I delivered the same in the maiden MongoDB Evening event at Delhi and Gurgaon in May 2014. When constructing a data model for your MongoDB collection for CMS, there are various options you can choose from, each of which has its strengths and weaknesses. The three basic patterns are: 1.Store each comment in its own document. 2.Embed all comments in the “parent” document. 3.A hybrid design, stores comments separately from the “parent,” but aggregates comments into a small number of documents, where each contains many comments. Code sample and wiki documentation is available on https://github.com/prasoonk/mycms_mongodb/wiki.

MongoDB Evenings DC: Get MEAN and Lean with Docker and KubernetesMongoDB

This document discusses running MongoDB and Kubernetes together to enable lean and agile development. It proposes using Docker containers to package applications and leverage tools like Kubernetes for deployment, management and scaling. Specifically, it recommends: 1) Using Docker to containerize applications and define deployment configurations. 2) Deploying to Kubernetes where services and replication controllers ensure high availability and scalability. 3) Treating databases specially by running them as "naked pods" assigned to labeled nodes with appropriate resources. 4) Demonstrating deployment of a sample MEAN stack application on Kubernetes with MongoDB and discussing future work around experimentation and blue/green deployments.

MongoDB AtlasMongoDB

MongoDB 3.4 webinarAndrew Morgan

In the age of digital transformation and disruption, your ability to thrive depends on how you adapt to the constantly changing environment. MongoDB 3.4 is the latest release of the leading database for modern applications, a culmination of native database features and enhancements that will allow you to easily evolve your solutions to address emerging challenges and use cases. In this webinar, we introduce you to what’s new, including: - Multimodel Done Right. Native graph computation, faceted navigation, rich real-time analytics, and powerful connectors for BI and Apache Spark bring additional multimodel database support right into MongoDB. - Mission-Critical Applications. Geo-distributed MongoDB zones, elastic clustering, tunable consistency, and enhanced security controls bring state-of-the-art database technology to your most mission-critical applications. - Modernized Tooling. Enhanced DBA and DevOps tooling for schema management, fine-grained monitoring, and cloud-native integration allow engineering teams to ship applications faster, with less overhead and higher quality.

MongoDB: How We Did It – Reanimating Identity at AOLMongoDB

AOL experienced explosive growth and needed a new database that was both flexible and easy to deploy with little effort. They chose MongoDB. Due to the complexity of internal systems and the data, most of the migration process was spent building a new identity platform and adapters for legacy apps to talk to MongoDB. Systems were migrated in 4 phases to ensure that users were not impacted during the switch. Turning on dual reads/writes to both legacy databases and MongoDB also helped get production traffic into MongoDB during the process. Ultimately, the project was successful with the help of MongoDB support. Today, the team has 15 shards, with 60-70 GB per shard.

MongoDB & The McGraw-Hill Education Learning Analytics PlatformMongoDB

Business Track: How MongoDB Helps Telefonia Digital Accelerate Time to MarketMongoDB

Migration from SQL to MongoDB - A Case Study at TheKnot.com MongoDB

8 out of 10 couples use TheKnot.com to help plan their wedding. A key part of planning involves selecting articles, photographs, and other resources and storing these in the user's Favorites. Recently we migrated major parts of our technology stack to open source technologies. As part of our migration strategy, we zeroed in on MongoDB, since it better suited our requirements for speed and data structure as well as eliminating the need for a caching layer. The transition required a period in which both our legacy and new API where working concurrently with data being persisted on both databases (SQL and Mongo) and all records were being synched with every request. We resourced to many strategies and applications to achieve this goal, including: Pentaho, AWS SQS and SNS, a queue messenger system and some proprietary ruby gems. In this session we will review our strategy and some of the lessons we learned about successfully migrating with zero downtime.

Apache Spark and MongoDB - Turning Analytics into Real-Time ActionJoão Gabriel Lima

This document discusses combining Apache Spark and MongoDB for real-time analytics. It provides an overview of MongoDB's native analytics capabilities including querying, data aggregation, and indexing. It then discusses how Apache Spark can extend these capabilities by providing additional analytics functions like machine learning, SQL queries, and streaming. Combining Spark and MongoDB allows organizations to perform real-time analytics on operational data without needing separate analytics infrastructure.

An Introduction to MongoDB Ops ManagerMongoDB

MongoDB Ops Manager is the easiest way to manage/monitor/operationalize your MongoDB footprint across your enterprise. Ops Manager automates key operations such as deployments, scaling, upgrades, and backups, all with the click of a button and integration with your favorite tools. It also provide the ability to monitor and alert on dozens of platform specific metrics. In this webinar, we'll cover the components of Ops Manager, as well as how it integrates and accelerates your use of MongoDB.

FINAL CARFAX PPT_ TEAM PANTHERSSera Su

Carfax plans to expand into China's growing used car market by providing vehicle history reports. Key points: 1) Carfax will analyze China's used car market opportunity, facing competition from local vehicle inspection services. 2) The plan outlines market segmentation, targeting dealers and young buyers, and differentiating Carfax reports as comprehensive and electronic. 3) Marketing will position Carfax as promoting transparency, with strategies including digital advertising, partnerships, and 24/7 customer support.

gannett 1Qtranscript2006finance30

- Gannett reported first quarter earnings of $0.99 per share, consistent with prior guidance but down from $1.03 per share in the first quarter of 2005. - Revenue growth at community newspapers was offset by softness in the UK market. Broadcasting saw record revenues due to the Winter Olympics. - Several one-time factors impacted expenses, including expensing of stock options and changes in newspaper partnerships. Excluding these items, costs increased only 0.6% overall and less than 1% for newspapers.

More Related Content

What's hot (20)

Webinar: What's New in MongoDB 3.2MongoDB

Webinar: Faster Big Data Analytics with MongoDBMongoDB

MongoDB Operations for DevelopersMongoDB

MongoDB vs Mysql. A devops point of viewPierre Baillet

Introduction to MongoDB EnterpriseMongoDB

When to Use MongoDB...and When You Should Not...MongoDB

3 scenarios when to use MongoDB!Edureka!

eHarmony - Messaging Platform with MongoDB Atlas MongoDB

NoSQL benchmarkingPrasoon Kumar

Securing Your MongoDB DeploymentMongoDB

MongoDB Introduction talk at Dr Dobbs Conference, MongoDB Evenings at Bangalo...Prasoon Kumar

MongoDB Evenings DC: Get MEAN and Lean with Docker and KubernetesMongoDB

MongoDB AtlasMongoDB

MongoDB 3.4 webinarAndrew Morgan

MongoDB: How We Did It – Reanimating Identity at AOLMongoDB

MongoDB & The McGraw-Hill Education Learning Analytics PlatformMongoDB

Business Track: How MongoDB Helps Telefonia Digital Accelerate Time to MarketMongoDB

Migration from SQL to MongoDB - A Case Study at TheKnot.com MongoDB

Apache Spark and MongoDB - Turning Analytics into Real-Time ActionJoão Gabriel Lima

An Introduction to MongoDB Ops ManagerMongoDB

Webinar: What's New in MongoDB 3.2MongoDB

Webinar: Faster Big Data Analytics with MongoDBMongoDB

MongoDB Operations for DevelopersMongoDB

MongoDB vs Mysql. A devops point of viewPierre Baillet

Introduction to MongoDB EnterpriseMongoDB

When to Use MongoDB...and When You Should Not...MongoDB

3 scenarios when to use MongoDB!Edureka!

eHarmony - Messaging Platform with MongoDB Atlas MongoDB

NoSQL benchmarkingPrasoon Kumar

Securing Your MongoDB DeploymentMongoDB

MongoDB Introduction talk at Dr Dobbs Conference, MongoDB Evenings at Bangalo...Prasoon Kumar

MongoDB Evenings DC: Get MEAN and Lean with Docker and KubernetesMongoDB

MongoDB AtlasMongoDB

MongoDB 3.4 webinarAndrew Morgan

MongoDB: How We Did It – Reanimating Identity at AOLMongoDB

MongoDB & The McGraw-Hill Education Learning Analytics PlatformMongoDB

Business Track: How MongoDB Helps Telefonia Digital Accelerate Time to MarketMongoDB

Migration from SQL to MongoDB - A Case Study at TheKnot.com MongoDB

Apache Spark and MongoDB - Turning Analytics into Real-Time ActionJoão Gabriel Lima

An Introduction to MongoDB Ops ManagerMongoDB

Viewers also liked (20)

FINAL CARFAX PPT_ TEAM PANTHERSSera Su

gannett 1Qtranscript2006finance30

Building Real Time Systems on MongoDB Using the Oplog at StripeMongoDB

How We Test MongoDB: EvergreenMongoDB

Comprehensive testing is a critical part of how we develop MongoDB. Because we support myriad features on multiple operating systems and architectures, it takes hundreds of hours to fully test a single commit to the MongoDB server repository. So how do we keep up? In this talk, we will detail our automated testing process and introduce Evergreen, the distributed continuous integration system enabling our engineers to get feedback on their code like never before.

Projects 2012 - Sans souciSans Souci Prague - Luxury glass solutions

The document provides specifications for several lighting fixtures and glass sculptures. It includes dimensions, glass parts used, light sources, wattages, designers, and notes that reproduction of the designs requires authorization. The fixtures range in size from 50x180cm to 350x750cm and use various glass combinations and light sources including LEDs, fiber optics, and GU10 bulbs. Materials include crystal, amber, and technical glasses set in stainless steel, brass, or nickel fittings.

Presentacion conclusiones grupo 1SaresAragon

Medienethik - Präsentation von Philippe WampflerPhilippe Wampfler

Protectia attorneys at law ::: Corporate presentationProtectia Patentes y Marcas

We are located in Madrid close to the Spanish Patent and Trademark Office (SPTO). Our modern management and communication system along with the experience of our professionals and correspondents allows us to successfully process, advise as well as monitor with full guarantees the files belonging both to SMEs working only at a national level as well as big intellectual and industrial property portfolios belonging to international companies

Wf. fitnessRollef

Mobile today-tomorrow-201006-aberlaNetcetera

In the future, there will be two types of companies: those that are "mobile ready" and those that will fail. Mobile technologies are advancing rapidly, with better connectivity, faster devices, and new interactive capabilities. The mobile experience will continue to converge, with phones replacing more dedicated devices and integrating diverse sensors and input methods. Content must be optimized for mobile contexts and allow sharing and interaction within social communities. Companies should start preparing their online presence and content strategies for a mobile-first future.

SPRING_PROSPECT_MAGAZINEGraeme Robertson

Presentación de la Factoría de Innovación MurciaEOI Escuela de Organización Industrial

La mujer. grupo 1presentaciones96

El documento resume la evolución histórica de la mujer, desde su rol tradicional en la familia y la sociedad hasta su lucha por la igualdad de derechos. Explica las diferencias laborales entre hombres y mujeres y el aumento de la participación política femenina a nivel global. También destaca la vulnerabilidad que enfrentan las mujeres y el movimiento feminista en pro de la dignidad y los derechos humanos de la mujer.

Security China 2016 ImpressionAnn He

ATOSS Software AG Geschäftsbericht 2015ATOSS Software AG

Nigel frank dynamics salary survey 2016 CRM DinamicsDaniel Sastre del Río ★ [LION]

This document summarizes the key findings of the 2016 Nigel Frank International Microsoft Dynamics Salary Survey. Some highlights include: - Over 18,700 Microsoft Dynamics professionals across 131 countries provided salary and employment data. - 55% of respondents held Microsoft certifications, with over 1/3 seeing a pay rise post-certification. - 53.6% of respondents worked for Microsoft Partners while 42% worked as end users. - 28.1% of respondents planned to leave their current employer within 12 months, down from 31.9% last year. - Laptop, bonus, and health insurance were the most common benefits provided by employers.

Cv[1].Veronica F. Benitezgasparbenitezbenitez

Casa gasparrodrigomolinalaserna

El documento describe la Casa Gaspar, una casa diseñada por el arquitecto español Alberto Campo Baeza. La casa tiene una planta cuadrada de 18 metros cuadrados dividida en tres ambientes: el centro es más alto que los laterales, que albergan servicios. Las paredes laterales son muy altas, de 8 metros, y cubren solo la parte central, creando porches delante y detrás para proteger de la luz del sol.

Manual para usuarios urkund 2015javierelpapi

Este manual explica cómo usar el software URKUND para realizar análisis de plagio de documentos estudiantiles. Los tutores reciben los análisis en sus cuentas de URKUND y revisan cada coincidencia identificada, desactivando las que no sean plagio real. Una vez finalizado el análisis, comparten los resultados con los estudiantes para que realicen los cambios necesarios y vuelvan a enviar el trabajo para una nueva revisión.

Salt Stack - Subhankar SenguptaDevOpsBangalore

This document discusses Salt Stack, an open source configuration management and remote execution system. Salt Stack allows users to manage configuration of systems through states written in YAML and execute commands remotely. It uses a master-minion architecture with asynchronous communication over ZeroMQ. Minions connect to the master and subscribe to events to receive configuration updates. Salt Stack provides modules to manage users, packages, files, services and more. It offers a simple and fast way to provision and manage systems at scale.

FINAL CARFAX PPT_ TEAM PANTHERSSera Su

gannett 1Qtranscript2006finance30

Building Real Time Systems on MongoDB Using the Oplog at StripeMongoDB

How We Test MongoDB: EvergreenMongoDB

Projects 2012 - Sans souciSans Souci Prague - Luxury glass solutions

Presentacion conclusiones grupo 1SaresAragon

Medienethik - Präsentation von Philippe WampflerPhilippe Wampfler

Protectia attorneys at law ::: Corporate presentationProtectia Patentes y Marcas

Wf. fitnessRollef

Mobile today-tomorrow-201006-aberlaNetcetera

SPRING_PROSPECT_MAGAZINEGraeme Robertson

Presentación de la Factoría de Innovación MurciaEOI Escuela de Organización Industrial

La mujer. grupo 1presentaciones96

Security China 2016 ImpressionAnn He

ATOSS Software AG Geschäftsbericht 2015ATOSS Software AG

Nigel frank dynamics salary survey 2016 CRM DinamicsDaniel Sastre del Río ★ [LION]

Cv[1].Veronica F. Benitezgasparbenitezbenitez

Casa gasparrodrigomolinalaserna

Manual para usuarios urkund 2015javierelpapi

Salt Stack - Subhankar SenguptaDevOpsBangalore

Similar to Building a Scalable and Modern Infrastructure at CARFAX (20)

Building a Scalable and Modern Infrastructure at CARFAXMongoDB

The CARFAX vehicle history database contains over twelve billion documents in a twelve shard cluster that replicates to multiple data centers. This will be a step by step walk through of how we deploy our servers, manage high volume reads and writes, and our configuration for high availability. By automating everything from the operating system install up we are able deploy complete replica clusters quickly and efficiently. Using distributed processing and message queuing we load millions of new documents each day with a projected growth over a billion records per year. Through the use of tagging, server configuration, and read settings we deliver content with high consistency and availability.

Deploy Apache Spark™ on Rackspace OnMetal™ for Cloud Big Data PlatformRackspace

There's an elephant in the room when it comes to Big Data. Apache Hadoop and Spark offer the promise to transform how businesses leverage Big Data, finding the right mix of flexible deployments, elastic scalability, and performance can be daunting. Introducing Rackspace OnMetal™ for Apache Spark™ an industry first that combines the performance and efficiency of bare metal with the ease and flexibility of cloud. With Rackspace OnMetal for Cloud Big Data Platform you can transform how you run Hadoop and Spark workloads: •Deploy in minutes, not months •Spin instances up or down on demand •Process data in-memory for faster query times •Get bare metal performance and say goodbye to virtualization taxes Sign up and learn how Rackspace OnMetal for Cloud Big Data Platform can rapidly move your organization from planning to deploying.

Aspera For Microsoft Sharepoint June 2016Aspera, an IBM Company

Aspera for Microsoft SharePoint is for organizations who need to quickly, predictably, and securely store and access high volumes of large files in Share Point. Aspera has seamlessly integrated patented FASP transfer technology into SharePoint for high-speed document upload and download workflows. Using Aspera, customers can not only overcome file and total repository size limitations of SharePoint, they also can reliably transfer files into and out of SharePoint at much higher speeds with auditable and predictable results. That means users can now upload large video files, imagery files, laser scan data, and other large files leveraging SharePoint document library structures and metadata for organization and search. Watch the Aspera for SharePoint Webinar at https://youtu.be/J8BOFBK-glc

HBaseCon 2013: General SessionCloudera, Inc.

Welcome! Michael Stack, Software Engineer, Cloudera & HBase PMC Chair 9:00-9:05am Conference MC Michael Stack, Chair of the HBaseCon 2013 Program Committee, welcomes you to the conference and offers a preview of the day. The Apache HBase Community: Best Ever and Getting Better Amr Awadallah, CTO and Co-founder, Cloudera 9:05-9:15am Amr comments on the explosion of interest in Apache HBase over the past few years, how that interest has influenced the Hadoop stack overall, and why Cloudera considers its involvement in the HBase community to be so important. State of the Apache HBase Union Michael Stack & Lars Hofhansl, Architect, Salesforce.com 9:15-9:40am Release-managers-in-crime Michael and Lars offer a look back, and a look forward, at HBase releases and what they have brought us (and will bring us in the future). The Apache HBase Ecosystem Aaron Kimball, Chief Architect, WibiData 9:40-10:05am Today, HBase stands as Apache Hadoop did years ago, a project with a growing and vibrant community in its own right. In this talk, Aaron will overview some of the projects built on top of HBase that you’ll get a chance to learn about during the day – each of these projects having grown out of a need to use HBase for an application that requires real-time atomic access to data. As an example, he’ll present the motivations for Kiji and how it is helping organizations create amazing new applications using HBase and Hadoop. Overview of Apache HBase at Facebook (Slides Not Available) Liyin Tang, Software Engineer, Facebook & HBase PMC Member 10:05-10:30am In this keynote, you’ll get an overview of how HBase is used at Facebook. Explore Facebook’s applications using HBase as an OLTP service, which require high reliability, efficiency, and scalability, and how HBase can tolerate small network glitches and rack failures. You’ll also learn the use cases for adopting HBase as a batch processing service and various optimizations to scale processing throughput. Finally, learn Facebook’s thoughts about the future of HBase.

Michael stack -the state of apache h basehdhappy001

The document provides an overview of Apache HBase, an open source, distributed, scalable, big data non-relational database. It discusses that HBase is modeled after Google's Bigtable and built on Hadoop for storage. It also summarizes that HBase is used by many large companies for applications such as messaging, real-time analytics, and search indexing. The project is led by an active community of committers and sees steady improvements and new features with each monthly release.

Wasp2 - IoT and Streaming PlatformPaolo Platter

Liferay & Big Data Dev Con 2014Miguel Pastor

The document discusses big data architectures and technologies. It introduces concepts like Hadoop, HDFS, MapReduce, Spark, Storm and Kafka. It proposes a reference architecture using these technologies with data sources like databases, user tracking, logs and streaming data. The architecture includes an event broker to handle streaming data which is then processed via Spark, Storm or Hadoop and stored in data warehouses or search indexes. It also provides examples of using these technologies for analytics, machine learning and graph processing.

Apache hadoop technology : BeginnersShweta Patnaik

HPCC Systems Engineering Summit: Community Use Case: Because Who Has Time for...HPCC Systems

Presenter: John Andleman, Staff Database Engineer, Citrix In this session, John will share some interesting use cases leveraging the HPCC Systems platform, including those beyond traditional big data uses. John will also share his roadmap of HPCC projects being planned for the next few months and why he feels HPCC Systems is a more suitable solution than Hadoop based on experiences and lessons learned. NOTE: The video of this presentation is the 3rd one shown in the accompanying YouTube link.

How companies use NoSQL and Couchbase - NoSQL Now 2013Dipti Borkar

Introduction to Big Data & Hadoop Architecture - Module 1Rohit Agrawal

Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineData Con LA

In this talk, we will discuss how we use Spark as part of a hybrid RDBMS architecture that includes Hadoop and HBase. The optimizer evaluates each query and sends OLTP traffic (including CRUD queries) to HBase and OLAP traffic to Spark. We will focus on the challenges of handling the tradeoffs inherent in an integrated architecture that simultaneously handles real-time and batch traffic. Lessons learned include: - Embedding Spark into a RDBMS - Running Spark on Yarn and isolating OLTP traffic from OLAP traffic - Accelerating the generation of Spark RDDs from HBase - Customizing the Spark UI The lessons learned can also be applied to other hybrid systems, such as Lambda architectures. Bio:- John Leach is the CTO and Co-Founder of Splice Machine. With over 15 years of software experience under his belt, John’s expertise in analytics and BI drives his role as Chief Technology Officer. Prior to Splice Machine, John founded Incite Retail in June 2008 and led the company’s strategy and development efforts. At Incite Retail, he built custom Big Data systems (leveraging HBase and Hadoop) for Fortune 500 companies. Prior to Incite Retail, he ran the business intelligence practice at Blue Martini Software and built strategic partnerships with integration partners. John was a key subject matter expert for Blue Martini Software in many strategic implementations across the world. His focus at Blue Martini was helping clients incorporate decision support knowledge into their current business processes utilizing advanced algorithms and machine learning. John received dual bachelor’s degrees in biomedical and mechanical engineering from Washington University in Saint Louis. Leach is the organizer emeritus for the Saint Louis Hadoop Users Group and is active in the Washington University Elliot Society.

Spark volume requirements 2018Rachit Arora

This document discusses storage requirements for running Spark workloads on Kubernetes. It recommends using a distributed file system like HDFS or DBFS for distributed storage and emptyDir or NFS for local temp scratch space. Logs can be stored in emptyDir or pushed to object storage. Features that would improve Spark on Kubernetes include image volumes, flexible PV to PVC mappings, encrypted volumes, and clean deletion for compliance. The document provides an overview of Spark, Kubernetes benefits, and typical Spark deployments.

Webinar - DreamObjects/Ceph Case StudyCeph Community

This document summarizes DreamObjects, an object storage platform powered by Ceph. It discusses the hardware used in storage and support nodes, including Intel and AMD processors, RAM, disks, and networking components. The document also provides details on Ceph configuration including replication, CRUSH mapping, OSD configuration, and application tuning. Monitoring tools discussed include Chef, pdsh, Sensu, collectd, graphite, logstash, Jenkins and future plans.

Storage Requirements and Options for Running Spark on KubernetesDataWorks Summit

In a world of serverless computing users tend to be frugal when it comes to expenditure on compute, storage and other resources. Paying for the same when they aren’t in use becomes a significant factor. Offering Spark as service on cloud presents very unique challenges. Running Spark on Kubernetes presents a lot of challenges especially around storage and persistence. Spark workloads have very unique requirements of Storage for intermediate data, long time persistence, Share file system and requirements become very tight when it same need to be offered as a service for enterprise to mange GDPR and other compliance like ISO 27001 and HIPAA certifications. This talk covers challenges involved in providing Serverless Spark Clusters share the specific issues one can encounter when running large Kubernetes clusters in production especially covering the scenarios related to persistence. This talk will help people using Kubernetes or docker runtime in production and help them understand various storage options available and which is more suitable for running Spark workloads on Kubernetes and what more can be done

Azure Cafe Marketplace with Hortonworks March 31 2016Joan Novino

Apache drillMapR Technologies

Trend Micro Big Data Platform and Apache BigtopEvans Ye

Building a Scalable and Modern Infrastructure at CARFAXMongoDB

Deploy Apache Spark™ on Rackspace OnMetal™ for Cloud Big Data PlatformRackspace

Aspera For Microsoft Sharepoint June 2016Aspera, an IBM Company

HBaseCon 2013: General SessionCloudera, Inc.

Michael stack -the state of apache h basehdhappy001

Wasp2 - IoT and Streaming PlatformPaolo Platter

Liferay & Big Data Dev Con 2014Miguel Pastor

Apache hadoop technology : BeginnersShweta Patnaik

HPCC Systems Engineering Summit: Community Use Case: Because Who Has Time for...HPCC Systems

How companies use NoSQL and Couchbase - NoSQL Now 2013Dipti Borkar

Introduction to Big Data & Hadoop Architecture - Module 1Rohit Agrawal

Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineData Con LA

Spark volume requirements 2018Rachit Arora

Webinar - DreamObjects/Ceph Case StudyCeph Community

Storage Requirements and Options for Running Spark on KubernetesDataWorks Summit

Azure Cafe Marketplace with Hortonworks March 31 2016Joan Novino

Apache drillMapR Technologies

Trend Micro Big Data Platform and Apache BigtopEvans Ye

More from MongoDB (20)

MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB

This presentation discusses migrating data from other data stores to MongoDB Atlas. It begins by explaining why MongoDB and Atlas are good choices for data management. Several preparation steps are covered, including sizing the target Atlas cluster, increasing the source oplog, and testing connectivity. Live migration, mongomirror, and dump/restore options are presented for migrating between replicasets or sharded clusters. Post-migration steps like monitoring and backups are also discussed. Finally, migrating from other data stores like AWS DocumentDB, Azure CosmosDB, DynamoDB, and relational databases are briefly covered.

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB

MongoDB Kubernetes operator and MongoDB Open Service Broker are ready for production operations. Learn about how MongoDB can be used with the most popular container orchestration platform, Kubernetes, and bring self-service, persistent storage to your containerized applications. A demo will show you how easy it is to enable MongoDB clusters as an External Service using the Open Service Broker API for MongoDB

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB

Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: Common components of an IoT solution The challenges involved with managing time-series data in IoT applications Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

MongoDB SoCal 2020: MongoDB Atlas Jump StartMongoDB

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB

Our clients have unique use cases and data patterns that mandate the choice of a particular strategy. To implement these strategies, it is mandatory that we unlearn a lot of relational concepts while designing and rapidly developing efficient applications on NoSQL. In this session, we will talk about some of our client use cases, the strategies we have adopted, and the features of MongoDB that assisted in implementing these strategies.

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB

Encryption is not a new concept to MongoDB. Encryption may occur in-transit (with TLS) and at-rest (with the encrypted storage engine). But MongoDB 4.2 introduces support for Client Side Encryption, ensuring the most sensitive data is encrypted before ever leaving the client application. Even full access to your MongoDB servers is not enough to decrypt this data. And better yet, Client Side Encryption can be enabled at the "flick of a switch". This session covers using Client Side Encryption in your applications. This includes the necessary setup, how to encrypt data without sacrificing queryability, and what trade-offs to expect.

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB

When you need to model data, is your first instinct to start breaking it down into rows and columns? Mine used to be too. When you want to develop apps in a modern, agile way, NoSQL databases can be the best option. Come to this talk to learn how to take advantage of all that NoSQL databases have to offer and discover the benefits of changing your mindset from the legacy, tabular way of modeling data. We’ll compare and contrast the terms and concepts in SQL databases and MongoDB, explain the benefits of using MongoDB compared to SQL databases, and walk through data modeling basics so you feel confident as you begin using MongoDB.

MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB

The document discusses guidelines for ordering fields in compound indexes to optimize query performance. It recommends the E-S-R approach: placing equality fields first, followed by sort fields, and range fields last. This allows indexes to leverage equality matches, provide non-blocking sorts, and minimize scanning. Examples show how indexes ordered by these guidelines can support queries more efficiently by narrowing the search bounds.

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB

The document describes a methodology for data modeling with MongoDB. It begins by recognizing the differences between document and tabular databases, then outlines a three step methodology: 1) describe the workload by listing queries, 2) identify and model relationships between entities, and 3) apply relevant patterns when modeling for MongoDB. The document uses examples around modeling a coffee shop franchise to illustrate modeling approaches and techniques.

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB

Virtual assistants are becoming the new norm when it comes to daily life, with Amazon’s Alexa being the leader in the space. As a developer, not only do you need to make web and mobile compliant applications, but you need to be able to support virtual assistants like Alexa. However, the process isn’t quite the same between the platforms. How do you handle requests? Where do you store your data and work with it to create meaningful responses with little delay? How much of your code needs to change between platforms? In this session we’ll see how to design and develop applications known as Skills for Amazon Alexa powered devices using the Go programming language and MongoDB.

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB

Il n’a jamais été aussi facile de commander en ligne et de se faire livrer en moins de 48h très souvent gratuitement. Cette simplicité d’usage cache un marché complexe de plus de 8000 milliards de $. La data est bien connu du monde de la Supply Chain (itinéraires, informations sur les marchandises, douanes,…), mais la valeur de ces données opérationnelles reste peu exploitée. En alliant expertise métier et Data Science, Upply redéfinit les fondamentaux de la Supply Chain en proposant à chacun des acteurs de surmonter la volatilité et l’inefficacité du marché.