Deep Learning using Spark and DL4J for fun and profit

Jul 11, 2016Download as PPTX, PDF4 likes1,833 views

This document discusses deep learning using Spark and DL4J. It introduces the speakers, Adam Gibson and Dhruv Kumar, and outlines the topics to be covered: an overview of deep learning, architectures, implementation and libraries for real-life applications, and a demonstration. Deep learning is described as one technique in data science that excels at tasks like image recognition, speech translation, and voice recognition by being loosely inspired by human brain models. The document then discusses using these techniques for enterprise use cases and realizing modern data applications in a Hadoop-centric world.

Page1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Deep Learning using Spark and DL4J for
fun and profit
Adam Gibson and Dhruv Kumar
2015
Version 1.0

Page2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Who are we?
Adam Gibson
- Co founder of Skymind
- Wrote DeepLearning4J, ND4J
Dhruv Kumar
- Sr Solutions Architect, HWX
- MS Umass, Mahout, ASF

Page3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
In this talk
- What’s Deep Learning?
- Architectures
- Implementation and Libraries in Real Life
- Demo!

Page4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Deep Learning
• One of the many pattern recognition techniques in Data
Science
• Excels at rich media applications:
• Image recognition
• Speech translation
• Voice recognition
• Loosely inspired by human brain models
• Synonymous with Artificial Neural Networks, Multi Layer
Networks

Page5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Enterprise use cases

Page6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Doing this in real life for enterprise

Page7 © Hortonworks Inc. 2011 – 2014. All Rights ReservedPage7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
HDP FOR DATA AT
REST
HDF FOR DATA IN
MOTION
ACTIONABLE
INTELLIGENCE
MODERN DATA APPS
Modern Data Applications
in Enterprise: Connected,
Fast, Intelligent
PERISHABLE
INSIGHTS
HISTORICAL
INSIGHTS
INTERNET
OF
ANYTHING

How do we realize MDA in a Hadoop Centric World?
HDF
Hadoop
HDFS
HBase Hive SOLR
YARN
Storm
Service
Management /
Workflow
SIEM
Spark
Raw Network Stream
Network Metadata Stream
Data Stores
Syslog
Raw Application Logs
Other Streaming Telemetry

www.hortonworks.com
NiFi 1
NiFi 2
Storm 1
Kafka 1
Storm 2
Kafka 2
Storm 3
Kafka 3
DataNode 1
HBase 1
Source 1
Source 2
Source 3
Source N
NiFi Nodes
Edge Nodes
Master NodesClients 1
Clients 2
DataNode 2
Hbase 2
DataNode 3
Hbase 3
DataNode 4
Hbase 4
DataNode 5
Hbase 5
DataNode 6
Hbase 6
DataNode 7
Hbase 7
DataNode 8
Hbase 8
DataNode 9 DataNode 10
DataNode 31 DataNode 32
Master 1
Master 2
Master 3
Master 4
Master 5
Worker Nodes
HDF
HDP
World Azure

Page10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Storm/Spark Streaming
Storm
Detailed Reference Architecture
HDF
Flume
Sink to
HDFS
Transform
Interactive
UI Framework
Hive
Hive
HDFS
HDFS
SOURCE DATA
Server logs
Application Logs
Firewall Logs
CRM/ERP
Sensor
Kafka
Kafka
Stream to
HDF
Forward to
Storm
Real Time Storage
Spark-ML
Pig
Alerts
Bolt to
HDFS
Dashboard
Silk
JMS
Alerts
Hive Server
HiveServer
Reporting
BI Tools
High Speed
Ingest
Real-Time
Batch Interactive
Machine Learning
Models
Spark
Pig
Alerts SQOOP
Flume
Iterative ML
Hbase/Pheonix
HBaseEvent Enrichment
Spark-Thrift
Pig

Page11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
For Model Building: Typical Workflow
11
1.Ingest training data and store it
2.Split data set into: training, testing and validation sets
3.Vectorize and extract features to go into next step
4.Architect multi layer network, initialize
5.Feed data and train
6.Test and Validate
7.Repeat steps 4 and 5 until desired
8.Store model
9.Put model in app, start generalizing on real data.

Page12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
So what do you get?
12
1.Ingest training data and store it using Nifi or other ingest tools
2.Split data set into: training, testing and validation sets
3.Vectorize and extract features to go into next step
4.Architect multi layer network, initialize
5.Feed data and train
6.Test and Validate
7.Repeat steps 4 and 5 until desired
8.Store model
9.Put model in app, start generalizing on real data.
Steps 2, 3, 4 and 5:
Use libraries such as
Deeplearning4j

Page13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Deeplearning4j Architecture
13

Page14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
DL4J: Canova for Vectorization and Ingest
• Canova uses an input/output format system (similar to
how Hadoop uses MapReduce)
• Supports all major types of input data (text, CSV, audio,
image and video)
• Can be extended for specialized input formats
• Connects to Kafka
14

Page15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
ND4J:
• N-dimensional vector library
• Scientific computing for JVM
• DL4J uses it to do linear algebra for backpropagation
• Supports GPUs via CUDA and Native via Jblas
• Deploys on Android
• DL4J code remains unchanged whether using GPU or
CPU
15

Page16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved 16
How to chose a
Neural Net in
DL4J core?

Page17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Demo!

Page18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Thank You
hortonworks.com

The document discusses deploying Hadoop in the cloud. Some key benefits of using Hadoop in the cloud include scalability, automated failover of replicated data, and cost efficiency through distributed processing and storage. Microsoft's Azure HDInsight offering provides a fully managed Hadoop and Spark service in the cloud that allows clusters to be provisioned in minutes and is optimized for analytics workloads. The Cortana Intelligence Suite integrates big data technologies like HDInsight with machine learning and data processing tools.

Evolving HDFS to a Generalized Storage SubsystemDataWorks Summit/Hadoop Summit

Sanjay Radia presents on evolving HDFS to support a generalized storage subsystem. HDFS currently scales well to large clusters and storage sizes but faces challenges with small files and blocks. The solution is to (1) only keep part of the namespace in memory to scale beyond memory limits and (2) use block containers of 2-16GB to reduce block metadata and improve scaling. This will generalize the storage layer to support containers for multiple use cases beyond HDFS blocks.

Hdfs 2016-hadoop-summit-san-jose-v4Chris Nauroth

The Hadoop Distributed File System is the foundational storage layer in typical Hadoop deployments. Performance and stability of HDFS are crucial to the correct functioning of applications at higher layers in the Hadoop stack. This session is a technical deep dive into recent enhancements committed to HDFS by the entire Apache contributor community. We describe real-world incidents that motivated these changes and how the enhancements prevent those problems from reoccurring. Attendees will leave this session with a deeper understanding of the implementation challenges in a distributed file system and identify helpful new metrics to monitor in their own clusters.

Improving Hadoop Resiliency and Operational Efficiency with EMC IsilonDataWorks Summit/Hadoop Summit

The document discusses how EMC Isilon scale-out NAS storage improves Hadoop resiliency and operational efficiency. It analyzes the impact of DataNode and TaskTracker failures on Hadoop jobs. EMC Isilon provides high availability, independent scalability of storage and compute, data protection features, and support for multiple Hadoop distributions and protocols like HDFS, NFS, SMB. This allows using existing data for analysis without replication and reduces time-to-results for Hadoop jobs.

Big Data in the Cloud - The What, Why and How from the ExpertsDataWorks Summit/Hadoop Summit

Hadoop has traditionally been an on-premises workload, with very few notable implementations on the cloud. With Organizations either having jumped on the cloud bandwagon or have started planning their expansion into the ecosystem, it is imperative for us to explore how Hadoop conforms to the cloud paradigm. With the coming off age of some very useful cloud paradigms and the nature of Big Data with high seasonality of workloads, this is becoming a very common ask from customers. Robust architectures, elastic scale, open platforms, OSS integrations, and addressing complex pain points will all be part of this lively talk. To be able to implement effective solutions for Big Data in the cloud it is imperative that you understand the core principles and grasp the design principles of how the cloud can enhance the benefits of parallelized analytics. Join this session to understand the nitty-gritties of implementing Big Data in the cloud and the various options therein. Big Data + Cloud is definitely a deadly combination.

End-to-End Security and Auditing in a Big Data as a Service DeploymentDataWorks Summit/Hadoop Summit

1. The document discusses security considerations for deploying big data as a service (BDaaS) across multiple tenants and applications. It focuses on maintaining a single user identity to prevent data duplication and enforce access policies consistently. 2. It describes using Apache Ranger to centrally define and enforce policies across Hadoop services like HDFS, HBase, Hive. Ranger integrates with LDAP/AD for authentication. 3. The key challenge is propagating user identities from the application layer to the data layer. This can be done by connecting HDFS directly via Kerberos or using a "super-user" that impersonates other users when accessing HDFS.

A New "Sparkitecture" for modernizing your data warehouseDataWorks Summit/Hadoop Summit

1) Enterprises struggle to manage big data with existing technologies due to more systems, complexity, and data to handle. 2) HPE proposes a new "Sparkitecture" called the HPE Elastic Platform for Analytics to address these issues. It uses a data-centric foundation to consolidate all data and applications on a single, elastic platform for analytics workloads. 3) The platform offers workload-optimized systems that provide better performance, scalability, and economics than traditional Hadoop architectures.

Scaling HDFS to Manage Billions of Files with Distributed Storage SchemesDataWorks Summit

The document discusses scaling HDFS to manage billions of files through distributed storage schemes. It outlines the current HDFS architecture and challenges with namespace and block scaling. It proposes a storage container architecture with distributed block maps and a storage container manager to address these challenges. This would allow HDFS to easily scale to manage trillions of blocks and billions of files across large clusters.

Backup and Disaster Recovery in Hadoop DataWorks Summit/Hadoop Summit

While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.

LLAP: Sub-Second Analytical Queries in HiveDataWorks Summit/Hadoop Summit

The document discusses LLAP (Live Long and Process), a new execution engine in Apache Hive 2.0 that enables sub-second analytical queries. LLAP keeps a small subset of frequently accessed data in memory to enable faster query processing times compared to traditional Hive architectures that rely on disk access. It works by running Hive query fragments concurrently in long-running daemon processes with an in-memory cache, rather than in short-lived YARN containers. This allows queries to retrieve data from memory rather than disk, providing significant performance improvements for interactive analytics workloads. The document provides details on how LLAP is implemented and evaluates its performance benefits based on benchmarks and customer case studies.

Dynamic DDL: Adding structure to streaming IoT data on the flyDataWorks Summit

At the end of day the only thing that data scientists want is one thing. They want tabular data for their analysis. They do not want to spend hours or days preparing data. How does a data engineer handle the massive amount of data that is being streamed at them from IoT devices and apps and at the same time add structure to it so that data scientists can focus on finding insights and not preparing data? By the way, you need to do this within minutes (sometimes seconds). Oh... and there are a bunch more data sources that you need to ingest and the current providers of data are changing their structure. At GoPro, we have massive amounts of heterogeneous data being streamed at us from our consumer devices and applications, and we have developed a concept of "dynamic DDL" to structure our streamed data on the fly using Spark Streaming, Kafka, HBase, Hive, and S3. The idea is simple. Add structure (schema) to the data as soon as possible. Allow the providers of the data to dictate the structure. And automatically create event-based and state-based tables (DDL) for all data sources to allow data scientists to access the data via their lingua franca, SQL, within minutes.

Accelerating Big Data InsightsDataWorks Summit

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARNDataWorks Summit

DeathStar is a system that runs HBase on YARN to provide easy, dynamic multi-tenant HBase clusters via YARN. It allows different applications to run HBase in separate application-specific clusters on a shared HDFS and YARN infrastructure. This provides strict isolation between applications and enables dynamic scaling of clusters as needed. Some key benefits are improved cluster utilization, easier capacity planning and configuration, and the ability to start new clusters on demand without lengthy provisioning times.

Big Data Simplified - Is all about Ab'strakSHeNDataWorks Summit

This document discusses designing a new big data platform to replace an existing complex and outdated one. It analyzes challenges with the current platform, including inability to keep up with business needs. The proposed new platform called Dredge would use abstraction layers to integrate big data tools in a loosely coupled and scalable way. This would simplify development and maintenance while supporting business goals. Key aspects of Dredge include declarative configuration, logical workflows, and plug-and-play integration of tools like HDFS, Hive, HBase, Kafka and Spark in a reusable and event-driven manner. The new platform aims to improve scalability, reduce costs and better support analytics needs over time.

Spark Uber Development KitDataWorks Summit/Hadoop Summit

The document discusses tools and techniques used by Uber's Hadoop team to make their Spark and Hadoop platforms more user-friendly and efficient. It introduces tools like SCBuilder to simplify Spark context creation, Kafka dispersal to distribute RDD results, and SparkPlug to provide templates for common jobs. It also describes a distributed log debugger called SparkChamber to help debug Spark jobs and techniques like building a spatial index to optimize geo-spatial joins. The goal is to abstract out infrastructure complexities and enforce best practices to make the platforms more self-service for users.

Hadoop crash course workshop at Hadoop SummitDataWorks Summit

This document provides an overview of Hadoop and its ecosystem. It discusses the evolution of Hadoop from version 1 which focused on batch processing using MapReduce, to version 2 which introduced YARN for distributed resource management and supported additional data processing engines beyond MapReduce. It also describes key Hadoop services like HDFS for distributed storage and the benefits of a Hadoop data platform for unlocking the value of large datasets.

The Unbearable Lightness of Ephemeral ProcessingDataWorks Summit

Ephemeral clusters can be launched quickly (minutes), are pre-configured for a specific processing purpose, and can be brought down quickly as soon as their usefulness has expired. The ability to launch Ephemeral clusters for on-demand processing, quickly and efficiently, is transforming how organizations design, deploy and Manage applications. The velocity and elasticity of fast cluster deployment enables seamless peak-demand provisioning, enables cost optimization by leveraging significantly lower cloud spot pricing, and maximizes utilization of existing compute capacity. Additionally, being able to launch bespoke clusters for specific compute needs in a repeatable fashion and within a shared infrastructure provides flexibility for special purpose processing needs. Organizations can leverage Ephemeral Clusters for parallel compute intensive applications which require short bursts of power but are short lived. In this session we will explore how to design Ephemeral clusters, how to launch, modify and bring them down, as well as application design considerations to maximize Ephemeral clusters usability.

Real-time Hadoop: The Ideal Messaging System for Hadoop DataWorks Summit/Hadoop Summit

Ted Dunning presents on streaming architectures and MapR Technologies' streaming capabilities. He discusses MapR Streams, which implements the Kafka API for high performance and scale. MapR provides a converged data platform with files, tables, and streams managed under common security and permissions. Dunning reviews several use cases and lessons learned around real-time data processing, microservices, and global data management requirements.

Hybrid Data Platform DataWorks Summit/Hadoop Summit

The document discusses the need for a hybrid data platform that combines on-premise and cloud-based infrastructure. A hybrid data platform can address issues like mixed workloads, long-tail data usage, and data movement costs. It also allows tapping into the scale of cloud while keeping some data and workloads on-premise. The proposed architecture shows data integration between on-premise Hadoop and cloud services like S3, Redshift, and EMR through private tunnels and security measures. Case studies demonstrate how a large commodities producer built a hybrid analytics environment for productivity, supply chain, and operations.

Apache Hadoop 3.0 What's new in YARN and MapReduceDataWorks Summit/Hadoop Summit

This document summarizes a presentation about new features in Apache Hadoop 3.0 related to YARN and MapReduce. It discusses major evolutions like the re-architecture of the YARN Timeline Service (ATS) to address scalability, usability, and reliability limitations. Other evolutions mentioned include improved support for long-running native services in YARN, simplified REST APIs, service discovery via DNS, scheduling enhancements, and making YARN more cloud-friendly with features like dynamic resource configuration and container resizing. The presentation estimates the timeline for Apache Hadoop 3.0 releases with alpha, beta, and general availability targeted throughout 2017.

Managing Hadoop, HBase and Storm Clusters at Yahoo ScaleDataWorks Summit/Hadoop Summit

The document discusses managing Hadoop, HBase and Storm clusters at Yahoo scale. It describes Yahoo's grid infrastructure which includes 3 data centers with over 45k nodes across 18 Hadoop clusters, 9 HBase clusters and 13 Storm clusters. It then provides details on the rolling upgrade processes for HDFS, YARN, HBase and Storm which involve minimizing downtime, upgrading components independently and verifying upgrades. CI/CD processes are used to automate software deployment and upgrades.

Disaster Recovery and Cloud Migration for your Apache Hive WarehouseDataWorks Summit

As Apache Hadoop clusters become central to an organization’s operations, they have clusters in more than one data center. Historically, this has been largely driven by requirements of business continuity planning or geo localization. It has also recently been gaining a lot of interest from a hybrid cloud perspective, i.e. wherein people are trying to augment their traditional on-prem setup with cloud-based additions as well. A robust replication solution is a fundamental requirement in such cases. The Apache Hive community has been working on new capabilities for efficient and fault tolerant replication of data in the Hive warehouse. In this talk, we will discuss these new capabilities, how it works, what replication at Hive-scale looks like, what challenges it poses, what we have done to solve those issues. We will also focus on what we need to be aware of in our use case that might make replication optimal. Speaker Sankar Hariappan, Senior Software Engineer, Hortonworks

From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...DataWorks Summit

Businesses often have to interact with different data sources to get a unified view of the business or to resolve discrepancies. These EDW data repositories are often large and complex, are business critical, and cannot afford downtime. This session will share best practices and lessons learned for building a Data Fabric on Spark / Hadoop / HIVE/ NoSQL that provides a unified view, enables a simplified access to the data repositories, resolves technical challenges and adds business value. Businesses often have to interact with different data sources to get a unified view of the business or to resolve discrepancies. These EDW data repositories are often large and complex, are business critical, and cannot afford downtime. This session will share best practices and lessons learned for building a Data Fabric on Spark / Hadoop / HIVE/ NoSQL that provides a unified view, enables a simplified access to the data repositories, resolves technical challenges and adds business value.

Hadoop from Hive with Stinger to TezJan Pieter Posthuma

Efficient Data Formats for Analytics with Parquet and ArrowDataWorks Summit/Hadoop Summit

Hadoop makes it relatively easy to store petabytes of data. However, storing data is not enough; columnar layouts for storage and in-memory execution allow the analysis of large amounts of data very quickly and efficiently. It provides the ability for multiple applications to share a common data representation and perform operations at full CPU throughput using SIMD and Vectorization. For interoperability, row based encodings (CSV, Thrift, Avro) combined with general purpose compression algorithms (GZip, LZO, Snappy) are common but inefficient. As discussed extensively in the database literature, a columnar layout with statistics and sorting provides vertical and horizontal partitioning, thus keeping IO to a minimum. Additionally a number of key big data technologies have or will soon have in-memory columnar capabilities. This includes Kudu, Ibis and Drill. Sharing a common in-memory columnar representation allows interoperability without the usual cost of serialization. Understanding modern CPU architecture is critical to maximizing processing throughput. We’ll discuss the advantages of columnar layouts in Parquet and Arrow for in-memory processing and data encodings used for storage (dictionary, bit-packing, prefix coding). We’ll dissect and explain the design choices that enable us to achieve all three goals of interoperability, space and query efficiency. In addition, we’ll provide an overview of what’s coming in Parquet and Arrow in the next year.

Fast SQL on Hadoop, Really?DataWorks Summit

The document discusses Apache Hive and Apache Druid for fast SQL on big data. It provides performance benchmarks showing Hive LLAP is faster than Presto and Spark SQL for TPC-DS queries. It describes features of Hive LLAP including in-memory caching, query result caching, and metadata caching. It also discusses new Hive 3 features like materialized views and optimizer improvements. The document then provides an overview of Apache Druid's capabilities for real-time ingestion and querying of streaming data before discussing how Hive and Druid can work together, with Hive able to push down queries to Druid.

Operationalizing YARN based Hadoop Clusters in the CloudDataWorks Summit/Hadoop Summit

Hadoop clusters are operated on an ephemeral basis in the cloud by Qubole, processing over 300 petabytes of data per month across over 100 customers. Qubole addresses challenges of ephemeral clusters through auto-scaling of resources using YARN, optimizing performance for cloud storage, and storing job history remotely. Volatile low-cost nodes are leveraged through policies that ensure data replication despite potential node failures.

Hortonworks Technical Workshop - Operational Best Practices WorkshopHortonworks

Brief introduction to Distributed Deep LearningAdam Gibson

Applied Deep Learning with Spark and Deeplearning4jDataWorks Summit

This document discusses deep learning and DL4J. It begins with an overview of deep learning, describing it as automated feature engineering through chained techniques like restricted Boltzmann machines. It then introduces DL4J, describing it as an enterprise-grade deep learning library for Java, Scala, and Python that supports parallelization on YARN and Spark as well as GPUs. The rest of the document discusses using DL4J with Spark for deep learning workflows on large datasets and provides an example of using the DL4J tool suite to perform vectorization, training, and evaluation on the Iris dataset.

More Related Content

What's hot (20)

Backup and Disaster Recovery in Hadoop DataWorks Summit/Hadoop Summit

LLAP: Sub-Second Analytical Queries in HiveDataWorks Summit/Hadoop Summit

Dynamic DDL: Adding structure to streaming IoT data on the flyDataWorks Summit

Accelerating Big Data InsightsDataWorks Summit

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARNDataWorks Summit

Big Data Simplified - Is all about Ab'strakSHeNDataWorks Summit

Spark Uber Development KitDataWorks Summit/Hadoop Summit

Hadoop crash course workshop at Hadoop SummitDataWorks Summit

The Unbearable Lightness of Ephemeral ProcessingDataWorks Summit

Real-time Hadoop: The Ideal Messaging System for Hadoop DataWorks Summit/Hadoop Summit

Hybrid Data Platform DataWorks Summit/Hadoop Summit

Apache Hadoop 3.0 What's new in YARN and MapReduceDataWorks Summit/Hadoop Summit

Managing Hadoop, HBase and Storm Clusters at Yahoo ScaleDataWorks Summit/Hadoop Summit

Disaster Recovery and Cloud Migration for your Apache Hive WarehouseDataWorks Summit

From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...DataWorks Summit

Hadoop from Hive with Stinger to TezJan Pieter Posthuma

Efficient Data Formats for Analytics with Parquet and ArrowDataWorks Summit/Hadoop Summit

Fast SQL on Hadoop, Really?DataWorks Summit

Operationalizing YARN based Hadoop Clusters in the CloudDataWorks Summit/Hadoop Summit

Hortonworks Technical Workshop - Operational Best Practices WorkshopHortonworks

Backup and Disaster Recovery in Hadoop DataWorks Summit/Hadoop Summit

LLAP: Sub-Second Analytical Queries in HiveDataWorks Summit/Hadoop Summit

Dynamic DDL: Adding structure to streaming IoT data on the flyDataWorks Summit

Accelerating Big Data InsightsDataWorks Summit

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARNDataWorks Summit

Big Data Simplified - Is all about Ab'strakSHeNDataWorks Summit

Spark Uber Development KitDataWorks Summit/Hadoop Summit

Hadoop crash course workshop at Hadoop SummitDataWorks Summit

The Unbearable Lightness of Ephemeral ProcessingDataWorks Summit

Real-time Hadoop: The Ideal Messaging System for Hadoop DataWorks Summit/Hadoop Summit

Hybrid Data Platform DataWorks Summit/Hadoop Summit

Apache Hadoop 3.0 What's new in YARN and MapReduceDataWorks Summit/Hadoop Summit

Managing Hadoop, HBase and Storm Clusters at Yahoo ScaleDataWorks Summit/Hadoop Summit

Disaster Recovery and Cloud Migration for your Apache Hive WarehouseDataWorks Summit

From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...DataWorks Summit

Hadoop from Hive with Stinger to TezJan Pieter Posthuma

Efficient Data Formats for Analytics with Parquet and ArrowDataWorks Summit/Hadoop Summit

Fast SQL on Hadoop, Really?DataWorks Summit

Operationalizing YARN based Hadoop Clusters in the CloudDataWorks Summit/Hadoop Summit

Hortonworks Technical Workshop - Operational Best Practices WorkshopHortonworks

Viewers also liked (20)

Brief introduction to Distributed Deep LearningAdam Gibson

Applied Deep Learning with Spark and Deeplearning4jDataWorks Summit

I'm being followed by dronesDataWorks Summit/Hadoop Summit

This document provides an overview of the emerging commercial drone market and opportunities for software applications development. It discusses what drones are and common uses like precision agriculture, infrastructure inspection, and real estate. An example of using drones for pole inspection is described. Regulations from the FAA are limiting but expected to evolve. Big data and internet of things technologies can be leveraged to build solutions for automated data collection, analysis and decision making. While public perception and some limitations still exist, the market is growing and those that act now will be well positioned for the future.

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...Yahoo Developer Network

Spark and Ignite are two of the most popular open source projects in the area of high-performance Big Data and Fast Data. But did you know that one of the best ways to boost performance for your next generation real-time applications is to use them together? In this session, Dmitriy Setrakyan, Apache Ignite Project Management Committee Chairman and co-founder and CPO at GridGain will explain in detail how IgniteRDD — an implementation of native Spark RDD and DataFrame APIs — shares the state of the RDD across other Spark jobs, applications and workers. Dmitriy will also demonstrate how IgniteRDD, with its advanced in-memory indexing capabilities, allows execution of SQL queries many times faster than native Spark RDDs or Data Frames. Don't miss this opportunity to learn from one of the experts how to use Spark and Ignite better together in your projects. Speakers: Dmitriy Setrakyan, is a founder and CPO at GridGain Systems. Dmitriy has been working with distributed architectures for over 15 years and has expertise in the development of various middleware platforms, financial trading systems, CRM applications and similar systems. Prior to GridGain, Dmitriy worked at eBay where he was responsible for the architecture of an add-serving system processing several billion hits a day. Currently Dmitriy also acts as PMC chair of Apache Ignite project.

Deep Learning for Java (DL4J)신동 강

Distributed deep rl on spark strata singaporeAdam Gibson

Deep Learning: DL4J and DataVecJosh Patterson

This document discusses using DL4J and DataVec to build deep learning workflows for modeling time series sensor data with recurrent neural networks. It provides an example of loading and transforming sensor data with DataVec, configuring an RNN with DL4J, and training the model both locally and distributed on Spark. The overall workflow involves extracting, transforming, and loading data with DataVec, vectorizing it, modeling with DL4J, evaluating performance, and deploying trained models for execution on Spark/Hadoop platforms.

Drones and the Internet of Things: realising the potential of airborne comput...Prayukth K V

This paper focuses on services and applications provided to mobile users using airborne computing infrastructure. Concepts such as drones-as-a-service and flyin,fly-out infrastructure, and note data management and system design issues that arise in these scenarios are discussed. Issues of Big Data arising from such applications, optimising the configuration of airborne and ground infrastructure to provide the best QoS and QoE, situation-awareness, scalability, reliability, scheduling for efficiency, interaction with users and drones using physical annotations are outlined.

Wrangleconf Big Data Malaysia 2016Adam Gibson

Drones Collaboration and IoT enabling digitalization of sensors - Angelo Fien...Codemotion

Drones can be relevant to Cisco by enabling the digitization of various sensors using technologies like the Internet of Things (IoT) and collaboration. There are many disparate sensors (IoE) and developers need a platform to make sensors accessible via APIs, provide data transport and routing, processing capabilities like fog computing, and a collaborative environment. Cisco's IoT, collaboration, connectivity, security, processing and storage capabilities combined with analytics APIs and algorithms can provide such a platform as the "IoT of everything". Drones equipped with specialized sensors and using targeted algorithms can help with applications in areas like precision agriculture, site surveys, 3D modeling, and more when integrated into this platform.

Big Data Analytics TokyoAdam Gibson

Network intrusion detection uses deep learning to analyze network traffic logs and detect anomalous activity that could indicate hackers. The logs are preprocessed and fed into a neural network to be analyzed in batches on a GPU cluster. The trained model can then detect intrusions in new incoming log data from multiple sources in real-time and help network administrators find malicious traffic on the network.

Recurrent nets and sensorsAdam Gibson

This document discusses recurrent neural networks and their ability to learn sequences through an internal memory component. It covers different recurrent architectures like RNNs, GRUs, and LSTMs. Recurrent nets can be used for applications involving sequences and prediction like generating text, forecasting, image captioning, and predictive maintenance in IoT. Their ability to model temporal data makes them well-suited for problems involving videos, sensors and predicting future events or states.

Hubba Deep LearningIvan Goloskokovic

The document discusses various topics related to deep learning including programming languages used for deep learning like Python, Java, and Lua, what deep learning is and is not, common machine learning algorithms, training deep learning models, convolutional neural networks, popular deep learning models from 1998 to present, pre-trained models, links to deep learning resources, and data used to train autonomous vehicles.

Introduction to TensorFlowMatthias Feys

TensorFlow is an open source software library for machine learning developed by Google. It provides primitives for defining functions on tensors and automatically computing their derivatives. TensorFlow represents computations as data flow graphs with nodes representing operations and edges representing tensors. It is widely used for neural networks and deep learning tasks like image classification, language processing, and speech recognition. TensorFlow is portable, scalable, and has a large community and support for deployment compared to other frameworks. It works by constructing a computational graph during modeling, and then executing operations by pushing data through the graph.

Business Disruption with Robots, Drones and Bots The IoT Tech Expo Oct 20 2016Sudha Jamthe

Deep learning with Hortonworks and Apache Spark - Hortonworks technical workshopHortonworks

Rich media is exploding all around us. From our personal usage to retailers monitoring store traffic for optimized associate placement, there is wide and growing application of rich media. Despite the pervasive usage, enterprises have had limited choice of generally available tools to analyze rich media. In this session we will look into leveraging deep learning algorithms for rich media analysis and provide practical hands on example of image recognition using Apache Hadoop and Spark.

large scale collaborative filtering using Apache GiraphDataWorks Summit

This document discusses large scale collaborative filtering using Apache Giraph. It describes neighborhood-based models and matrix factorization for collaborative filtering. It also details how these techniques were implemented and optimized in Giraph to provide recommendations for billions of Facebook users and ratings. Key optimizations included a rotational approach for matrix factorization to reduce network traffic and memory usage for skewed item degrees.

Future of ai on the jvmAdam Gibson

This document discusses the future of artificial intelligence on the Java Virtual Machine (JVM). It outlines how machine learning frameworks are currently monolithic and make assumptions about data. The document proposes a micro-services approach to machine learning that separates out concerns like data pipelines, scoring, model training, and evaluation. This would help reduce lock-in and allow greater flexibility. It also discusses how new hardware like GPUs are better suited for deep learning and the role frameworks like Spark and Akka could play in distributed, real-time machine learning applications on the JVM.

Skymind 深度学习 - T11 SummitShu Wei Goh

Deep Learning on Production with SparkShu Wei Goh

Brief introduction to Distributed Deep LearningAdam Gibson

Applied Deep Learning with Spark and Deeplearning4jDataWorks Summit

I'm being followed by dronesDataWorks Summit/Hadoop Summit

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...Yahoo Developer Network

Deep Learning for Java (DL4J)신동 강

Distributed deep rl on spark strata singaporeAdam Gibson

Deep Learning: DL4J and DataVecJosh Patterson

Drones and the Internet of Things: realising the potential of airborne comput...Prayukth K V

Wrangleconf Big Data Malaysia 2016Adam Gibson

Drones Collaboration and IoT enabling digitalization of sensors - Angelo Fien...Codemotion

Big Data Analytics TokyoAdam Gibson

Recurrent nets and sensorsAdam Gibson

Hubba Deep LearningIvan Goloskokovic

Introduction to TensorFlowMatthias Feys

Business Disruption with Robots, Drones and Bots The IoT Tech Expo Oct 20 2016Sudha Jamthe

Deep learning with Hortonworks and Apache Spark - Hortonworks technical workshopHortonworks

large scale collaborative filtering using Apache GiraphDataWorks Summit

Future of ai on the jvmAdam Gibson

Skymind 深度学习 - T11 SummitShu Wei Goh

Deep Learning on Production with SparkShu Wei Goh

Similar to Deep Learning using Spark and DL4J for fun and profit (20)

Hortonworks - What's Possible with a Modern Data Architecture?Hortonworks

Hortonworks and Red Hat Webinar - Part 2Hortonworks

[Hortonworks] Future Of Data: Madrid - HDF & Data in motionRaúl Marín

Discover.hdp2.2.storm and kafka.finalHortonworks

Introduction to the Hadoop EcoSystemShivaji Dutta

This document summarizes Hortonworks' Hadoop distribution called Hortonworks Data Platform (HDP). It discusses how HDP provides a comprehensive data management platform built around Apache Hadoop and YARN. HDP includes tools for storage, processing, security, operations and accessing data through batch, interactive and real-time methods. The document also outlines new capabilities in HDP 2.2 like improved engines for SQL, Spark and streaming and expanded deployment options.

Curb your insecurity with HDPDataWorks Summit/Hadoop Summit

This document discusses securing Hadoop and Spark clusters. It begins with an overview of Hadoop security in four steps: authentication, authorization, data protection, and audit. It then discusses specific Hadoop security components like Kerberos, Apache Ranger, HDFS encryption, Knox gateway, and data encryption in motion and at rest. For Spark security, it covers authentication using Kerberos, authorization with Ranger, and encrypting data channels. The document provides demos of HDFS encryption and discusses common gotchas with Spark security.

Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!Pardeep Kumar Mishra (Big Data / Hadoop Consultant)

You got your cluster installed and configured. You celebrate, until the party is ruined by your company's Security officer stamping a big "Deny" on your Hadoop cluster. And oops!! You cannot place any data onto the cluster until you can demonstrate it is secure In this session you would learn the tips and tricks to fully secure your cluster for data at rest, data in motion and all the apps including Spark. Your Security officer can then join your Hadoop revelry (unless you don't authorize him to, with your newly acquired admin rights)

Realtime analytics + hadoop 2.0Rommel Garcia

The document discusses Hadoop and its capabilities for real-time analytics. It describes how Hadoop provides massive storage and parallel processing. It also explains the key components of Hadoop including HDFS for storage and YARN as the data operating system. The document outlines use cases for real-time analytics in various industries and demonstrates a real-time analytics application for monitoring truck driver behavior.

Realtime Analytics in HadoopRommel Garcia

Supporting Financial Services with a More Flexible Approach to Big DataHortonworks

The document discusses how Hortonworks Data Platform (HDP) enables a modern data architecture with Apache Hadoop. HDP provides a common data set stored in HDFS that can be accessed through various applications for batch, interactive, and real-time processing. This allows organizations to store all their data in one place and access it simultaneously through multiple means. YARN is the architectural center of HDP and enables this modern data architecture. HDP also provides enterprise capabilities like security, governance, and operations to make Hadoop suitable for business use.

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...Innovative Management Services

Mr. Slim Baltagi is a Systems Architect at Hortonworks, with over 4 years of Hadoop experience working on 9 Big Data projects: Advanced Customer Analytics, Supply Chain Analytics, Medical Coverage Discovery, Payment Plan Recommender, Research Driven Call List for Sales, Prime Reporting Platform, Customer Hub, Telematics, Historical Data Platform; with Fortune 100 clients and global companies from Financial Services, Insurance, Healthcare and Retail. Mr. Slim Baltagi has worked in various architecture, design, development and consulting roles at. Accenture, CME Group, TransUnion, Syntel, Allstate, TransAmerica, Credit Suisse, Chicago Board Options Exchange, Federal Reserve Bank of Chicago, CNA, Sears, USG, ACNielsen, Deutshe Bahn. Mr. Baltagi has also over 14 years of IT experience with an emphasis on full life cycle development of Enterprise Web applications using Java and Open-Source software. He holds a master’s degree in mathematics and is an ABD in computer science from Université Laval, Québec, Canada. Languages: Java, Python, JRuby, JEE , PHP, SQL, HTML, XML, XSLT, XQuery, JavaScript, UML, JSON Databases: Oracle, MS SQL Server, MYSQL, PostreSQL Software: Eclipse, IBM RAD, JUnit, JMeter, YourKit, PVCS, CVS, UltraEdit, Toad, ClearCase, Maven, iText, Visio, Japser Reports, Alfresco, Yslow, Terracotta, Toad, SoapUI, Dozer, Sonar, Git Frameworks: Spring, Struts, AppFuse, SiteMesh, Tiles, Hibernate, Axis, Selenium RC, DWR Ajax , Xstream Distributed Computing/Big Data: Hadoop, MapReduce, HDFS, Hive, Pig, Sqoop, HBase, R, RHadoop, Cloudera CDH4, MapR M7, Hortonworks HDP 2.1

Cloud Austin Meetup - Hadoop like a championAmeet Paranjape

The document discusses how Hadoop can be used for interactive and real-time data analysis. It notes that the amount of digital data is growing exponentially and will reach 40 zettabytes by 2020. Traditional data systems are struggling to manage this new data. Hadoop provides a solution by tying together inexpensive servers to act as one large computer for processing big data using various Apache projects for data access, governance, security and operations. Examples show how Hadoop can be used to analyze real-time streaming data from sensors on trucks to monitor routes, vehicles and drivers.

Hortonworks tech workshop in-memory processing with sparkHortonworks

Apache Spark offers unique in-memory capabilities and is well suited to a wide variety of data processing workloads including machine learning and micro-batch processing. With HDP 2.2, Apache Spark is a fully supported component of the Hortonworks Data Platform. In this session we will cover the key fundamentals of Apache Spark and operational best practices for executing Spark jobs along with the rest of Big Data workloads. We will also provide a working example to showcase micro-batch and machine learning processing using Apache Spark.

Webinar: Selecting the Right SQL-on-Hadoop SolutionMapR Technologies

Storm Demo Talk - Colorado Springs May 2015Mac Moore

The document discusses real-time processing capabilities in Hadoop and Hortonworks Data Platform (HDP). It begins with an introduction to Hortonworks and an overview of real-time streaming architectures on HDP. It then demonstrates streaming capabilities with and without predictive analytics additions. The document highlights how HDP provides a centralized architecture and open data platform to enable real-time and batch processing of any type of data for analytics applications.

Apache NiFi + Tensorflow + Hadoop：Big Data AI サンドイッチの作り方HortonworksJapan

This document discusses using Apache NiFi, TensorFlow, and Hadoop together for big data AI applications. It provides an overview of each technology and how they can be combined. Specifically, it describes how NiFi can be used to ingest and transport data to Hadoop, TensorFlow can be used to build and deploy machine learning models for real-time insights, and both can leverage Hadoop for storage and processing of historical data. Examples of building basic and professional "AI sandwiches" by layering these components are also presented.

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...Hortonworks

Discover hdp 2.2 hdfs - finalHortonworks

Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUGskumpf

The document discusses real-time processing in Hadoop using the Hortonworks Data Platform (HDP). It provides an overview of using HDP for real-time streaming analytics in a logistics scenario. Example applications and architectures are presented, including using Kafka for ingesting sensor data, Storm for stream processing, and HBase for real-time querying. Demos will also illustrate integrating predictive analytics into streaming scenarios.

Discover.hdp2.2.ambari.final[1]Hortonworks

Apache Ambari is a single framework for IT administrators to provision, manage and monitor a Hadoop cluster. Apache Ambari 1.7.0 is included with Hortonworks Data Platform 2.2. In this 30-minute webinar, Hortonworks Product Manager Jeff Sposetti and Apache Ambari committer Mahadev Konar discussed new capabilities including: Improvements to Ambari core - such as support for ResourceManager HA Extensions to Ambari platform - introducing Ambari Administration and Ambari Views Enhancements to Ambari Stacks - dynamic configuration recommendations and validations via a "Stack Advisor"

Hortonworks - What's Possible with a Modern Data Architecture?Hortonworks

Hortonworks and Red Hat Webinar - Part 2Hortonworks

[Hortonworks] Future Of Data: Madrid - HDF & Data in motionRaúl Marín

Discover.hdp2.2.storm and kafka.finalHortonworks

Introduction to the Hadoop EcoSystemShivaji Dutta

Curb your insecurity with HDPDataWorks Summit/Hadoop Summit

Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!Pardeep Kumar Mishra (Big Data / Hadoop Consultant)

Realtime analytics + hadoop 2.0Rommel Garcia

Realtime Analytics in HadoopRommel Garcia

Supporting Financial Services with a More Flexible Approach to Big DataHortonworks

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...Innovative Management Services

Cloud Austin Meetup - Hadoop like a championAmeet Paranjape

Hortonworks tech workshop in-memory processing with sparkHortonworks

Webinar: Selecting the Right SQL-on-Hadoop SolutionMapR Technologies

Storm Demo Talk - Colorado Springs May 2015Mac Moore

Apache NiFi + Tensorflow + Hadoop：Big Data AI サンドイッチの作り方HortonworksJapan

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...Hortonworks

Discover hdp 2.2 hdfs - finalHortonworks

Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUGskumpf

Discover.hdp2.2.ambari.final[1]Hortonworks

More from DataWorks Summit/Hadoop Summit (20)

Running Apache Spark & Apache Zeppelin in ProductionDataWorks Summit/Hadoop Summit

This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.

State of Security: Apache Spark & Apache ZeppelinDataWorks Summit/Hadoop Summit

Unleashing the Power of Apache Atlas with Apache RangerDataWorks Summit/Hadoop Summit

The document discusses the Virtual Data Connector project which aims to leverage Apache Atlas and Apache Ranger to provide unified metadata and access governance across data sources. Key points include: - The project aims to address challenges of understanding, governing, and controlling access to distributed data through a centralized metadata catalog and policies. - Apache Atlas provides a scalable metadata repository while Apache Ranger enables centralized access governance. The project will integrate these using a virtualization layer. - Enhancements to Atlas and Ranger are proposed to better support the project's goals around a unified open metadata platform and metadata-driven governance. - An initial minimum viable product will be built this year with the goal of an open, collaborative ecosystem around shared

Enabling Digital Diagnostics with a Data Science PlatformDataWorks Summit/Hadoop Summit

This document discusses using a data science platform to enable digital diagnostics in healthcare. It provides an overview of healthcare data sources and Yale/YNHH's data science platform. It then describes the data science journey process using a clinical laboratory use case as an example. The goal is to use big data and machine learning to improve diagnostic reproducibility, throughput, turnaround time, and accuracy for laboratory testing by developing a machine learning algorithm and real-time data processing pipeline.

Revolutionize Text Mining with Spark and ZeppelinDataWorks Summit/Hadoop Summit

This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.

Double Your Hadoop Performance with Hortonworks SmartSenseDataWorks Summit/Hadoop Summit

This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.

Hadoop Crash CourseDataWorks Summit/Hadoop Summit

The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.

Data Science Crash CourseDataWorks Summit/Hadoop Summit

This document provides an overview of data science and machine learning. It discusses what data science and machine learning are, including extracting insights from data and computers learning without being explicitly programmed. It also covers Apache Spark, which is an open source framework for large-scale data processing. Finally, it discusses common machine learning algorithms like regression, classification, clustering, and dimensionality reduction.

Apache Spark Crash CourseDataWorks Summit/Hadoop Summit

This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.

Dataflow with Apache NiFiDataWorks Summit/Hadoop Summit

This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.

Schema Registry - Set you Data FreeDataWorks Summit/Hadoop Summit

Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats. SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc. In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit

There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time. The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit

DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.

Mool - Automated Log Analysis using Data Science and MLDataWorks Summit/Hadoop Summit

QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful. At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.

How Hadoop Makes the Natixis Pack More Efficient DataWorks Summit/Hadoop Summit

Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together. This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear: • How and why the business and IT requirements originated • How we leverage the platform to fulfill security and production requirements • How we organize a community to: o Guard all the players, no one gets left on the ground! o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead) • What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match! DETAILS This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.

HBase in Practice DataWorks Summit/Hadoop Summit

HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.

The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit

There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases. In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.

Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopDataWorks Summit/Hadoop Summit

In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit

In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs. Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.

Scaling HDFS to Manage Billions of Files with Distributed Storage SchemesDataWorks Summit/Hadoop Summit

Hadoop Distributed File System (HDFS) evolves from a MapReduce-centric storage system to a generic, cost-effective storage infrastructure where HDFS stores all data of inside the organizations. The new use case presents a new sets of challenges to the original HDFS architecture. One challenge is to scale the storage management of HDFS - the centralized scheme within NameNode becomes a main bottleneck which limits the total number of files stored. Although a typical large HDFS cluster is able to store several hundred petabytes of data, it is inefficient to handle large amounts of small files under the current architecture. In this talk, we introduce our new design and in-progress work that re-architects HDFS to attack this limitation. The storage management is enhanced to a distributed scheme. A new concept of storage container is introduced for storing objects. HDFS blocks are stored and managed as objects in the storage containers instead of being tracked only by NameNode. Storage containers are replicated across DataNodes using a newly-developed high-throughput protocol based on the Raft consensus algorithm. Our current prototype shows that under the new architecture the storage management of HDFS scales 10x better, demonstrating that HDFS is capable of storing billions of files.

Running Apache Spark & Apache Zeppelin in ProductionDataWorks Summit/Hadoop Summit

State of Security: Apache Spark & Apache ZeppelinDataWorks Summit/Hadoop Summit

Unleashing the Power of Apache Atlas with Apache RangerDataWorks Summit/Hadoop Summit

Enabling Digital Diagnostics with a Data Science PlatformDataWorks Summit/Hadoop Summit

Revolutionize Text Mining with Spark and ZeppelinDataWorks Summit/Hadoop Summit

Double Your Hadoop Performance with Hortonworks SmartSenseDataWorks Summit/Hadoop Summit

Hadoop Crash CourseDataWorks Summit/Hadoop Summit

Data Science Crash CourseDataWorks Summit/Hadoop Summit

Apache Spark Crash CourseDataWorks Summit/Hadoop Summit

Dataflow with Apache NiFiDataWorks Summit/Hadoop Summit

Schema Registry - Set you Data FreeDataWorks Summit/Hadoop Summit

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit

Mool - Automated Log Analysis using Data Science and MLDataWorks Summit/Hadoop Summit

How Hadoop Makes the Natixis Pack More Efficient DataWorks Summit/Hadoop Summit

HBase in Practice DataWorks Summit/Hadoop Summit

The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit

Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopDataWorks Summit/Hadoop Summit

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit

Scaling HDFS to Manage Billions of Files with Distributed Storage SchemesDataWorks Summit/Hadoop Summit

Recently uploaded (20)

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

Splunk Security Update | Public Sector Summit Germany 2025Splunk

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how: • Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules. • Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance. • Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity. • Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications. • Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market. With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications. Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Procurement Insights Cost To Value Guide.pptxJon Hansen

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Want to learn practical tips for designing systems that can scale efficiently without compromising speed? Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development. As you explore key principles of designing low-latency systems with Rust, you will learn how to: - Create and compile a real-world app with Rust - Connect the application to ScyllaDB (NoSQL data store) - Negotiate tradeoffs related to data modeling and querying - Manage and monitor the database for consistently low latencies

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn