Elasticsearch meetup final_2014_04

Apr 28, 2014Download as PPTX, PDF0 likes4,586 views

This document summarizes a presentation about using Elasticsearch for analytics on customer communities. It discusses how Lithium uses Elasticsearch to analyze terabytes of daily social media data to understand customer participation and content that generates engagement. It also provides details about Lithium's large Elasticsearch cluster containing over 7 billion documents, and lessons learned around bulk loading, faceting, and settings for data centers.

Elasticsearch for reporting
analytics on communities
Elasticsearch Meetup - 23 April 2014
Marc Harrison

Lithium makes software that helps brands
better connect with their customers
Our social software helps companies respond on social networks and build
trusted content on a community they own.

Empower brands to distill terabytes of daily
data into understanding participation
▪ fast
▪ flexible
▪ scalable
What
products/services
are generating the
most conversations?
Who is authoring
content that
generates the most
kudos/likes?
Are customer posts
getting timely
replies?
What types of
content does this
audience segment
look for?

Cluster specs
▪ One of our clusters – elastic search 1.0
• 7+ billion documents/4.4+ TB and growing fast!
• 21 nodes (3 masters, 2 clients, 16 data)

Lessons learned
▪ Bulk loading
▪ Faceting

Bulk initial load / rebuild of data
Hadoop
mysql stream
Transform/
route
…
JSON Elasticsearch

Bulk loading
▪ Make sure ingest logic is robust
• Idempotent for bulk reply - ‘_id’
• Include revision based on processor/time
• Check cluster/index status to make sure ready to ingest
▪ Know the cache and thread pool sizes
• Bulk – fixed - # of processors - queue size 50
• Handle back off and retry
▪ How many docs?
• Like capacity - test with data –
• number of shards
• index.refresh_interval: 30s
• indices.memory.index_buffer_size: 5%
• indices.memory.*
• index.translog.*

Faceting
▪ Don't forget about memory!
• Strings - not_analyzed
• Numbers long vs int, double vs float, etc
• Do you need seconds/minutes when faceting?
• fielddata format - doc_values (1.0)
• Admin API’s allow checking field data size + evictions
• indices.cache.filter.size: 15%
• indices.fielddata.cache.size: 45%

Faceting II
▪ Accuracy
• shard_size
• Number of shards
• Cardinality
• Routing
▪ Great custom plugin
framework
• Uniques
• Array faceting

Impact
▪ Order of magnitude improvement
▪ Developers able to focus on improving insights
▪ community + elasticsearch + hadoop + horton works =
exciting

Select settings (data center)
• bootstrap.mlockall: true
• cluster.routing.allocation.disk.threshold_enabled: true
• http.compression: true
• transport.tcp.compress: true
• gateway.recover_after_data_nodes: 13
• gateway.recover_after_master_nodes: 2
• gateway.recover_after_time: 3m
• gateway.expected_nodes: 17
• indices.memory.index_buffer_size: 5%
• indices.cache.filter.size: 15%
• indices.fielddata.cache.size: 45%
• index.store.type: mmapfs
• index.translog.flush_threshold_ops: 10000
• action.auto_create_index: false
• action.disable_delete_all_indices: true
• cluster.routing.allocation.node_initial_primaries_recoveries: 4
• cluster.routing.allocation.node_concurrent_recoveries: 15
• indices.recovery.max_bytes_per_sec: 100mb
• indices.recovery.concurrent_streams: 5
• discovery.zen.minimum_master_nodes: 2
• index.search.slowlog.threshold.query.warn: 5s
• index.search.slowlog.threshold.query.info: 1s
• index.indexing.slowlog.threshold.index.warn: 5s
• plugin.mandatory: lithium-unique-facets

The document discusses CARFAX's transition from a proprietary key-value store to MongoDB. It describes CARFAX's production MongoDB environment including 12 sharded servers with 128GB RAM each. It details how CARFAX loads millions to billions of records per day via distributed processing. It also discusses implementing high availability reads through tagging data centers and replica sets to scale to millions of reports per day. The presentation emphasizes automating processes and sharding early and often for scaling their MongoDB deployment.

Elk - An introductionHossein Shemshadi

So, what is the ELK Stack? "ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Oss as a competitive advantageRegunath B

Exploration of multidimensional biomedical data in pub chem, Presented by Lia...Lucidworks (Archived)

The document discusses the development of a new search system for PubChem to allow for exploration of multidimensional biomedical data. The new system was needed to address the challenges of handling large and heterogeneous datasets with many relationships between data types in a way that allows for fast querying. The system leverages Apache SOLR to provide features like full text search, faceting, molecule structure searching and joining of related data. It includes backend components like SOLR, SQL and specialized search engines as well as web APIs and frontend interfaces like reusable widgets and a new search interface.

Your data layer - Choosing the right database solutions for the futureObjectRocket

This document discusses choosing the right database solutions and trends related to databases. It notes the explosion in the number of database options and the expansion of their use cases. It advocates a polyglot persistence approach of using different databases for different data storage needs within an application. Some key trends discussed are the rise of open source databases and issues around cloud and vendor portability. The document emphasizes finding the right balance between databases and not getting locked into any single technology or cloud provider.

Hadoop and friendsChandan Rajah

Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It provides reliable data storage, parallel processing, and fault tolerance. Key components of Hadoop include HDFS for distributed file storage, MapReduce for distributed processing, Hive for data warehousing, and HBase for NoSQL database access. Hadoop has seen widespread adoption for applications such as log analysis, data warehousing, and machine learning due to its scalability, low costs, and fault tolerance.

Big Data Overview Part 1William Simms

Big Data has become the new buzzword like “Agile” and “Cloud”. Like those two others, it’s a transformative technology. We’ll be discussing: •What is it? •Technology key words •HDFS •Hadoop •MapReduce This will be part 1 of 2 (at least). This first talk will not be overly technical. We’ll go over the concepts and terms you’ll encounter when considering a big data solution.

Visualizing Austin's data with Elasticsearch and KibanaObjectRocket

Session #2, tech session: Build realtime search by Sylvain Utard from AlgoliaSaaS Is Beautiful

This document provides an overview of building a real-time search engine. It discusses how search engines work by indexing documents to build an inverted index optimized for queries. When a query is received, the inverted index is used to quickly match and rank relevant documents. The document then describes moving from a mobile SDK to a hosted search as a service (SaaS) and the technical considerations for scaling the SaaS such as architecture, security, and operations.

Practical Use of a NoSQLIBM Cloud Data Services

In the world of NoSQL, each database has its own strengths and weaknesses. Understanding which open source database is "the right tool for the job" is half the battle if you want to start building better applications quickly. IBM developer advocate Glynn Bird explores practical examples of how two popular NoSQL databases - the Cloudant JSON document store and the Redis in-memory key-value store - can be used together to create performant and scalable Web applications. It also includes real world use cases you can try today, for free, using the IBM Cloud Data Services suite of fully managed NoSQL databases-as-a-service.

GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...Yann Cluchey

My talk from GOTO Aarhus, 30th September 2014. Cogenta is a retail intelligence company which tracks ecommerce web sites around the world to provide competitive monitoring and analysis services to retailers. Using its proprietary crawler technology, Lucene and SQL Server, a stream of 20 million raw product data entries is captured and processed each day. This case study looks at how Cogenta uses Elasticsearch to break the shackles imposed by the RDBMS (and a limited budget) to make the data available in real time to its customers. Cogenta uses SQL as its canonical store & for complex reporting, and Elasticsearch for real-time processing & to drive its SaaS web applications. Elasticsearch is easy to use, delivers the powerful features of Lucene and enables the data & platform cost to scale linearly. But… synchronising your existing data in two places presents some interesting challenges such as aggregation and concurrency control. This talk will take a detailed look at how Cogenta how overcame those challenges, with a perpetually changing and asynchronously updated dataset. http://gotocon.com/aarhus-2014/presentation/Cogenta%20-%20Making%20Enterprise%20Data%20Available%20in%20Real%20Time%20with%20Elasticsearch

Next Generation Data Platforms - Deon ThomasThoughtworks

MongoDB & Hadoop - Understanding Your Big DataMongoDB

Big Data is the evolution of supercomputing for commercial enterprise and governments. Originally the domain of companies operating at Internet scale, today Big Data connects organizations of all sizes with discovery about their patterns, and insights into their business. But understanding the differences between the plethora of new technologies can be daunting. Graph / columnar / key value store / document are all called NoSQL, but which is best? How does Hadoop play in this ecosystem - its low cost and high efficiency have made it very popular, but how does it fit? In this webinar, we will explore: The full spectrum of Big Data Hadoop and MongoDB: friends or frenemies? Differences between Systems of Record and Systems of Engagement MongoDB customer examples of Systems of Engagement

Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, Wal...Lucidworks

This document discusses how Walmart uses Apache Solr as a "not-so-evil twin" to complement their source of truth database and help scale their data infrastructure. It describes how Walmart abstracts the complexity of managing databases, caches, search queries, and messaging to provide scalable querying across database shards. The use of Solr has allowed Walmart to offload queries, recurring reads, analytics

Why Elastic? @ 50th Vinitaly 2016Christoph Wurm

MongoDB meetup at HikeBharvi Dixit

NoSQL for SQL UsersIBM Cloud Data Services

Practical Use of a NoSQL DatabaseIBM Cloud Data Services

Building tiered data stores using aesop to bridge sql and no sql systemsRegunath B

Redis & MongoDB: Stop Big Data Indigestion Before It StartsItamar Haber

Efficiently digesting data in large volumes can prove to be challenging for any database. The challenges are compounded when this influx must be analyzed on the fly, or "tasted", to satisfy the sophisticated palates of modern apps. Luckily, there are several proven remedies you can concoct with Redis to help with potential indigestion. The URLs from the presentation are also available at: https://gist.github.com/itamarhaber/325e515c1715a12ef132

Enterprise Presto PaaS offering in Google Cloud Ashish Tadose

hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and SparkMichael Stack

Basic Introduction to Crate @ ViennaDB MeetupJohannes Moser

Capacity Planning For Your Growing MongoDB ClusterMongoDB

This document discusses capacity planning for deploying MongoDB. It defines capacity planning as planning for requirements like availability, throughput, and responsiveness by determining necessary resources like CPU, memory, storage, and network capacity. It emphasizes starting capacity planning before launch to avoid downtime. Key aspects of capacity planning for MongoDB include estimating working memory set size, storage I/O needs based on data size and access patterns, using tools like IOStat and MongoDB Management Service for monitoring and automation, and conducting iterative testing and deployments. Failure occurs if planned resources cannot meet requirements.

Distributed Query Service Powered By Presto & Alluxio Across Clouds @Walmart...Ashish Tadose

Presto is used as a distributed query service across Walmart's clouds to provide simplified query access and better scalability. Alluxio caching is used to optimize query performance by avoiding unpredictable network I/O. The architectural components include authentication with LDAP, authorization using custom Presto Ranger plugins, query tuning using session property managers, and auto-scaling on Google Cloud Platform. Presto combined with Alluxio improves query response times by caching frequently used data and avoiding network hops for recurrent queries.

Aesop change data propagationRegunath B

1) Aesop is an open source change data capture and propagation tool that reliably captures changes from data sources and propagates them to other data stores and systems to enable eventual consistency across polyglot data platforms. 2) It uses log mining to capture changes from data sources like MySQL and propagates the change events to consumers like ElasticSearch and HBase through an enhanced relay component. 3) It provides utilities for bootstrapping consumers, monitoring and administering the system, and has been used in production at Flipkart for applications including payments, ETL, and data serving.

Webinar: MongoDB and Hadoop - Working Together to provide Business InsightsMongoDB

Join us for a webinar on how MongoDB and Hadoop can work together to solve Big Data problems in today's enterprises. We will take an in depth look at how the two technologies make real business intelligence accessible to end users. After a brief introduction to both technologies, this webinar will dive deep into the MongoDB+Hadoop Connector and how it is applied to enable new business insights. In this webinar you will learn: What information problems are a good fit for MongoDB and Hadoop How to integrate the two technologies using the MongoDB+Hadoop Connector Programming paradigms for tackling common problems

Cortana Analytics Workshop: Big Data @ MicrosoftMSAdvAnalytics

Fostering Community With Social Media - Midwest Newspaper Summit 2010Nathan Wright

How to Grow a Vibrant Community that Delivers Real Business ResultsCMX

More Related Content

What's hot (20)

Session #2, tech session: Build realtime search by Sylvain Utard from AlgoliaSaaS Is Beautiful

Practical Use of a NoSQLIBM Cloud Data Services

GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...Yann Cluchey

Next Generation Data Platforms - Deon ThomasThoughtworks

MongoDB & Hadoop - Understanding Your Big DataMongoDB

Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, Wal...Lucidworks

Why Elastic? @ 50th Vinitaly 2016Christoph Wurm

MongoDB meetup at HikeBharvi Dixit

NoSQL for SQL UsersIBM Cloud Data Services

Practical Use of a NoSQL DatabaseIBM Cloud Data Services

Building tiered data stores using aesop to bridge sql and no sql systemsRegunath B

Redis & MongoDB: Stop Big Data Indigestion Before It StartsItamar Haber

Enterprise Presto PaaS offering in Google Cloud Ashish Tadose

hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and SparkMichael Stack

Basic Introduction to Crate @ ViennaDB MeetupJohannes Moser

Capacity Planning For Your Growing MongoDB ClusterMongoDB

Distributed Query Service Powered By Presto & Alluxio Across Clouds @Walmart...Ashish Tadose

Aesop change data propagationRegunath B

Webinar: MongoDB and Hadoop - Working Together to provide Business InsightsMongoDB

Cortana Analytics Workshop: Big Data @ MicrosoftMSAdvAnalytics

Session #2, tech session: Build realtime search by Sylvain Utard from AlgoliaSaaS Is Beautiful

Practical Use of a NoSQLIBM Cloud Data Services

GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...Yann Cluchey

Next Generation Data Platforms - Deon ThomasThoughtworks

MongoDB & Hadoop - Understanding Your Big DataMongoDB

Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, Wal...Lucidworks

Why Elastic? @ 50th Vinitaly 2016Christoph Wurm

MongoDB meetup at HikeBharvi Dixit

NoSQL for SQL UsersIBM Cloud Data Services

Practical Use of a NoSQL DatabaseIBM Cloud Data Services

Building tiered data stores using aesop to bridge sql and no sql systemsRegunath B

Redis & MongoDB: Stop Big Data Indigestion Before It StartsItamar Haber

Enterprise Presto PaaS offering in Google Cloud Ashish Tadose

hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and SparkMichael Stack

Basic Introduction to Crate @ ViennaDB MeetupJohannes Moser

Capacity Planning For Your Growing MongoDB ClusterMongoDB

Distributed Query Service Powered By Presto & Alluxio Across Clouds @Walmart...Ashish Tadose

Aesop change data propagationRegunath B

Webinar: MongoDB and Hadoop - Working Together to provide Business InsightsMongoDB

Cortana Analytics Workshop: Big Data @ MicrosoftMSAdvAnalytics

Viewers also liked (8)

Fostering Community With Social Media - Midwest Newspaper Summit 2010Nathan Wright

How to Grow a Vibrant Community that Delivers Real Business ResultsCMX

Digital Innovation That Drives BusinessPhillip Parisis

Digital Engagement Journey Kapil Bhatia

Building, Growing & Sustaining Social Media Communities Keynote Presentation Pam Moore

Building, Growing & Sustaining Social Media Communities. Pam Moore, CEO /Founder Marketing Nutz, full service social brand, digital marketing, conversion optimization agency & consultancy used this presentation for her closing keynote presentation at the American For the Arts, National Art Project Conference in Portland, Oregon, November, 2013. Pam shares 10 tips to help business leaders birth, grow and sustain communities both on and offline. http://www.pammarketingnut.com/2013/11/building-growing-sustaining-social-communities-includes-slideshare-presentation

Social Media: Growing Brand AwarenessLiam Dempsey

Deep Social: The Next Phase of Social MediaOgilvy Consulting

This document discusses the concept of "Deep Social" as the next phase of social media. Deep Social involves three key shifts for brands: 1) Moving from a marketing to a publisher mindset by becoming culturally relevant content creators, 2) Transforming superficial content into great stories and experiences to excite customers, and 3) Shifting from aimless connections to meaningful conversations and advocacy through precision targeting and bonding with customers. Examples are provided of brands implementing Deep Social strategies through insights, inspiration, innovation, precision targeting, and meaningful conversations to achieve business goals like increased revenue, engagement, and customer loyalty.

Radical Candor: No BS, helping your team create better work.Digital Surgeons

Fostering Community With Social Media - Midwest Newspaper Summit 2010Nathan Wright

How to Grow a Vibrant Community that Delivers Real Business ResultsCMX

Digital Innovation That Drives BusinessPhillip Parisis

Digital Engagement Journey Kapil Bhatia

Building, Growing & Sustaining Social Media Communities Keynote Presentation Pam Moore

Social Media: Growing Brand AwarenessLiam Dempsey

Deep Social: The Next Phase of Social MediaOgilvy Consulting

Radical Candor: No BS, helping your team create better work.Digital Surgeons

Similar to Elasticsearch meetup final_2014_04 (20)

Social Media, Cloud Computing, Machine Learning, Open Source, and Big Data An...Open Analytics

This document discusses using social media, cloud computing, machine learning, open source, and big data analytics to analyze Twitter data. It describes how to collect tweets using the Twitter API, classify tweets in real-time using machine learning models on AWS, store classified tweets in MongoDB on AWS, and present results. Cost estimates for real-time classification of 1 million tweets per day are provided. Use cases described include tracking food poisoning reports and disease occurrence. Future directions discussed include developing turnkey services and linking to additional open data sources.

Open Data Summit Presentation by Joe OlsenChristopher Whitaker

Big Data Open Source Tools and Trends: Enable Real-Time Business Intelligence...Perficient, Inc.

This document discusses big data tools and trends that enable real-time business intelligence from machine logs. It provides an overview of Perficient, a leading IT consulting firm, and introduces the speakers Eric Roch and Ben Hahn. It then covers topics like what constitutes big data, how machine data is a source of big data, and how tools like Hadoop, Storm, Elasticsearch can be used to extract insights from machine data in real-time through open source solutions and functional programming approaches like MapReduce. It also demonstrates a sample data analytics workflow using these tools.

Nisha talagala keynote_inflow_2016Nisha Talagala

This document discusses new data applications like machine learning and deep learning and their implications for storage. It notes that these applications deal with large and diverse data types including time series, matrices, and graphs. They have relaxed requirements for data correctness and persistence compared to traditional transactions. Opportunities exist to optimize storage for these workloads through techniques like tiering across memory types, streamlining data access, and exploiting lineage metadata to cache intermediate results. Fundamental shifts may also be possible by integrating analytics optimizations into storage management.

Engineering patterns for implementing data science models on big data platformsHisham Arafat

Discussion of practically implementing data science models on big data platforms from engineering perspective. An eye opener on the engineering factors associated with designing and working solution. We use a simple text mining example on social media analytics for brand marketing. At the first while, it seems simple solution however if you go deeply and think on implementation aspects of even a simple analytics model, you can discover the degree of complexity at each part of the solution. An Abstraction of the Big Data key advantages would be very helpful to select appropriate Big Data technology components out of very large landscape. Two examples with reference are given for using Lambda Architecture and unusual way of image processing using Big Data abstraction provided.

Elasticsearch Introduction at BigData meetupEric Rodriguez (Hiring in Lex)

Elasticsearch - Scalability and MultitenancyBozhidar Bozhanov

Solving Office 365 Big Challenges using Cassandra + Spark Anubhav Kale

Share point 2013 enterprise search (public)Petter Skodvin-Hvammen

This document summarizes key learnings from a presentation about SharePoint 2013 and Enterprise Search. It discusses how to run a successful search project through planning, development, testing and deployment. It also covers infrastructure needs and capacity testing findings. Additionally, it provides examples of how to customize the user experience through display templates and Front search. Methods for crawling thousands of file shares and enriching indexed content are presented. The document concludes with discussions on relevancy, managing property weighting, changing ranking models, and tuning search results.

Solr Under the Hood at S&P Global- Sumit Vadhera, S&P Global Lucidworks

This document summarizes S&P Global's use of Solr for search capabilities across their large datasets. It discusses how S&P Global indexes over 50 million documents into Solr monthly and handles over 5 million queries per week. It outlines challenges faced with an on-premise Solr deployment and how migrating to Solr Cloud helped address issues like performance, availability, and scalability. Next steps discussed include improving relevancy through data science, continuing to leverage new Solr features, and exploring ways to integrate machine learning into search capabilities.

Apache drillMapR Technologies

Making Session Stores More IntelligentKyle Davis

MongoDB .local Houston 2019: Building an IoT Streaming Analytics Platform to ...MongoDB

Building high performance and scalable share point applicationsTalbott Crowell

10 Big Data Technologies you Didn't Know About Jesus Rodriguez

This document introduces several big data technologies that are less well known than traditional solutions like Hadoop and Spark. It discusses Apache Flink for stream processing, Apache Samza for processing real-time data from Kafka, Google Cloud Dataflow which provides a managed service for batch and stream data processing, and StreamSets Data Collector for collecting and processing data in real-time. It also covers machine learning technologies like TensorFlow for building dataflow graphs, and cognitive computing services from Microsoft. The document aims to think beyond traditional stacks and learn from companies building pipelines at scale.

RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...S. Diana Hu

Search engines have focused on solving the document retrieval problem, so their scoring functions do not handle naturally non-traditional IR data types, such as numerical or categorical. Therefore, on domains beyond traditional search, scores representing strengths of associations or matches may vary widely. As such, the original model doesn’t suffice, so relevance ranking is performed as a two-phase approach with 1) regular search 2) external model to re-rank the filtered items. Metrics such as click-through and conversion rates are associated with the users’ response to items served. The predicted selection rates that arise in real-time can be critical for optimal matching. For example, in recommender systems, predicted performance of a recommended item in a given context, also called response prediction, is often used in determining a set of recommendations to serve in relation to a given serving opportunity. Similar techniques are used in the advertising domain. To address this issue the authors have created ML-Scoring, an open source framework that tightly integrates machine learning models into a popular search engine (SOLR/Elasticsearch), replacing the default IR-based ranking function. A custom model is trained through either Weka or Spark and it is loaded as a plugin used at query time to compute custom scores.

RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...Joaquin Delgado PhD.

This tutorial gives an overview of how search engines and machine learning techniques can be tightly coupled to address the need for building scalable recommender or other prediction based systems. Typically, most of them architect retrieval and prediction in two phases. In Phase I, a search engine returns the top-k results based on constraints expressed as a query. In Phase II, the top-k results are re-ranked in another system according to an optimization function that uses a supervised trained model. However this approach presents several issues, such as the possibility of returning sub-optimal results due to the top-k limits during query, as well as the prescence of some inefficiencies in the system due to the decoupling of retrieval and ranking. To address this issue the authors created ML-Scoring, an open source framework that tightly integrates machine learning models into Elasticsearch, a popular search engine. ML-Scoring replaces the default information retrieval ranking function with a custom supervised model that is trained through Spark, Weka, or R that is loaded as a plugin in Elasticsearch. This tutorial will not only review basic methods in information retrieval and machine learning, but it will also walk through practical examples from loading a dataset into Elasticsearch to training a model in Spark, Weka, or R, to creating the ML-Scoring plugin for Elasticsearch. No prior experience is required in any system listed (Elasticsearch, Spark, Weka, R), though some programming experience is recommended.

ADV Slides: Trends in Streaming Analytics and Message-oriented MiddlewareDATAVERSITY

Streaming and real-time data has high business value, but that value can rapidly decay if not processed quickly. If the value of the data is not realized in a certain window of time, its value is lost and the decision or action that was needed as a result never occurs. Streaming data – whether from sensors, devices, applications, or events – needs special attention because a sudden price change, a critical threshold met, a sensor reading changing rapidly, or a blip in a log file can all be of immense value, but only if the alert is in time.

ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY

Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020. Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms. Data lakes will be built in cloud object storage. We’ll discuss the options there as well. Get this data point for your data lake journey.

Search on the fly: how to lighten your Big Data - Simona Russo, Auro Rolle - ...Codemotion

The talk presents a new technique of realtime single entity information extraction and investigation. The technique eliminates regular refresh and persistence of data within the search engine (ETL), providing real-time access to source data and improving response times using in-memory data techniques. The solution presented is a concrete solution with live customers, based upon real business needs. I will explain the architectural overview, the technology stack used based on Apache Lucene library, the accomplished results and how to scale out the solution.