LEGO: Data Driven Growth Hacking Powered by Big Data
Download as PPTX, PDF3 likes1,600 views
This document describes LEGO's data-driven growth hacking powered by big data. It discusses how LEGO uses a data lake and log processing engine to process hundreds of billions of log lines from various sources at scale. The data is then profiled, cleansed, transformed and loaded into data warehouses to enable self-service analysis and predictive data apps for executives, analysts and product managers.
1 of 14
Downloaded 33 times
![11
Salesforce Confidential
Data Profiler
Dataset Field Type, Total, Min, Max, Avg, # Nulls, # Distinct, Median, 99th %tile, Top
N
lego_feat browser STR 2.3B 7 63 25 1M 50 34 38 [.....]
lego_feat url STR 2.3B 20 223 50 0 5M 70 90
[.....]
. . . . . . . . . .
.
. . . . . . . . . .
.
. . . . . . . . . .
.
Datasets
across
platform
HCatalog
MapReduce
Datasets Dataset Profile
An Example
Monitoring & alerting](https://image.slidesharecdn.com/june28500salesforcegokhaleduggireddy-160710170858/85/LEGO-Data-Driven-Growth-Hacking-Powered-by-Big-Data-11-320.jpg)
Recommended
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionDataWorks Summit/Hadoop Summit This document discusses Hadoop integration with cloud storage. It describes the Hadoop-compatible file system architecture, which allows Hadoop applications to work with both HDFS and cloud storage transparently. Recent enhancements to the S3A file system connector for Amazon S3 are discussed, including performance improvements and support for encryption. Benchmark results show significant performance gains for Hive queries with S3A compared to earlier versions. Upcoming work on output committers, object store abstraction, and consistency are outlined.
Visualizing Big Data in Realtime
Visualizing Big Data in RealtimeDataWorks Summit This document provides an overview of Apache Apex and real-time data visualization. Apache Apex is a platform for developing scalable streaming applications that can process billions of events per second with millisecond latency. It uses YARN for resource management and includes connectors, compute operators, and integrations. The document discusses using Apache Apex to build real-time dashboards and widgets using the App Data Framework, which exposes application data sources via topics. It also covers exporting and packaging dashboards to include in Apache Apex application packages.
Solr + Hadoop: Interactive Search for Hadoop
Solr + Hadoop: Interactive Search for Hadoopgregchanan This document discusses Cloudera Search, which integrates Apache Solr with Cloudera's distribution of Apache Hadoop (CDH) to provide interactive search capabilities. It describes the architecture of Cloudera Search, including components like Solr, SolrCloud, and Morphlines for extraction and transformation. Methods for indexing data in real-time using Flume or batch using MapReduce are presented. The document also covers querying, security features like Kerberos authentication and collection-level authorization using Sentry, and concludes by describing how to obtain Cloudera Search.
IOT, Streaming Analytics and Machine Learning 
IOT, Streaming Analytics and Machine Learning DataWorks Summit/Hadoop Summit This document discusses integrating Internet of Things (IOT) data, streaming analytics, and machine learning using Apache NiFi and SAS Event Stream Processing. It describes how SAS ESP can be used to build real-time analytics models using a drag-and-drop interface to detect patterns in streaming data. It also outlines how SAS ESP can integrate with Hortonworks Data Flow (NiFi) to enable rapid prototyping of machine learning models on streaming data within an open framework. Finally, it provides an overview of how SAS ESP connectors and adapters allow flexibility and integration with other data sources.
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...Alex Zeltov Introduction to Big Data Analytics using Apache Spark on HDInsights on Azure (SaaS) and/or HDP on Azure(PaaS)
This workshop will provide an introduction to Big Data Analytics using Apache Spark using the HDInsights on Azure (SaaS) and/or HDP deployment on Azure(PaaS) . There will be a short lecture that includes an introduction to Spark, the Spark components.
Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes.
The lecture will be followed by demo . There will be a short lecture on Hadoop and how Spark and Hadoop interact and compliment each other. You will learn how to move data into HDFS using Spark APIs, create Hive table, explore the data with Spark and SQL, transform the data and then issue some SQL queries. We will be using Scala and/or PySpark for labs.
Spark Uber Development Kit
Spark Uber Development KitDataWorks Summit/Hadoop Summit The document discusses tools and techniques used by Uber's Hadoop team to make their Spark and Hadoop platforms more user-friendly and efficient. It introduces tools like SCBuilder to simplify Spark context creation, Kafka dispersal to distribute RDD results, and SparkPlug to provide templates for common jobs. It also describes a distributed log debugger called SparkChamber to help debug Spark jobs and techniques like building a spatial index to optimize geo-spatial joins. The goal is to abstract out infrastructure complexities and enforce best practices to make the platforms more self-service for users.
Creating the Internet of Your Things
Creating the Internet of Your ThingsDataWorks Summit/Hadoop Summit The document discusses Microsoft's Azure IoT platform for connecting, managing, and analyzing Internet of Things devices and data. It provides an overview of the key components of Azure IoT including Azure IoT Hub for device connectivity and management, analytics services like Azure Machine Learning and Stream Analytics, and connectivity to other Azure services. It also highlights aspects of Azure IoT like its open ecosystem, support for open standards, and global infrastructure running on Microsoft's Azure cloud.
Apache Hive 2.0: SQL, Speed, Scale
Apache Hive 2.0: SQL, Speed, ScaleDataWorks Summit/Hadoop Summit This document discusses the new features of Apache Hive 2.0, including:
1) The addition of procedural SQL capabilities through HPLSQL to add features like cursors and loops.
2) Performance improvements for interactive queries through LLAP which uses in-memory caching and persistent daemons.
3) Using HBase as the metastore to speed up query planning by reducing metadata access times.
4) Enhancements to Hive on Spark such as dynamic partition pruning and vectorized joins.
5) Improvements to the cost-based optimizer including better statistics collection.
Securing Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise ContextDataWorks Summit/Hadoop Summit Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
Innovation in the Enterprise Rent-A-Car Data Warehouse
Innovation in the Enterprise Rent-A-Car Data WarehouseDataWorks Summit Big Data adoption is a journey. Depending on the business the process can take weeks, months, or even years. With any transformative technology the challenges have less to do with the technology and more to do with how a company adapts itself to a new way of thinking about data. Building a Center of Excellence is one way for IT to help drive success.
This talk will explore Enterprise Holdings Inc. (which operates the Enterprise Rent-A-Car, National Car Rental and Alamo Rent A Car) and their experience with Big Data. EHI’s journey started in 2013 with Hadoop as a POC and today are working to create the next generation data warehouse in Microsoft’s Azure cloud utilizing a lambda architecture.
We’ll discuss the Center of Excellence, the roles in the new world, share the things which worked well, and rant about those which didn’t.
No deep Hadoop knowledge is necessary, architect or executive level.
Realizing the Promise of Portable Data Processing with Apache Beam
Realizing the Promise of Portable Data Processing with Apache BeamDataWorks Summit The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the Big Data ecosystem together; it enables users to "run-anything-anywhere".
This talk will briefly cover the capabilities of the Beam model for data processing, as well as the current state of the Beam ecosystem. We'll discuss Beam architecture and dive into the portability layer. We'll offer a technical analysis of the Beam's powerful primitive operations that enable true and reliable portability across diverse environments. Finally, we'll demonstrate a complex pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Spark on Amazon Web Services, Apache Flink on Google Cloud, Apache Apex on-premise), and give a glimpse at some of the challenges Beam aims to address in the future.
Speaker
Davor Bonaci, Senior Software Engineer, Google
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
Druid: Sub-Second OLAP queries over Petabytes of Streaming DataDataWorks Summit When interacting with analytics dashboards in order to achieve a smooth user experience, two major key requirements are sub-second response time and data freshness. Cluster computing frameworks such as Hadoop or Hive/Hbase work well for storing large volumes of data, although they are not optimized for ingesting streaming data and making it available for queries in realtime. Also, long query latencies make these systems sub-optimal choices for powering interactive dashboards and BI use-cases.
In this talk we will present Druid as a complementary solution to existing hadoop based technologies. Druid is an open-source analytics data store, designed from scratch, for OLAP and business intelligence queries over massive data streams. It provides low latency realtime data ingestion and fast sub-second adhoc flexible data exploration queries.
Many large companies are switching to Druid for analytics, and we will cover how druid is able to handle massive data streams and why it is a good fit for BI use cases.
Agenda -
1) Introduction and Ideal Use cases for Druid
2) Data Architecture
3) Streaming Ingestion with Kafka
4) Demo using Druid, Kafka and Superset.
5) Recent Improvements in Druid moving from lambda architecture to Exactly once Ingestion
6) Future Work
Cloudy with a chance of Hadoop - real world considerations
Cloudy with a chance of Hadoop - real world considerationsDataWorks Summit Over the last eighteen months, we have seen significant adoption of Hadoop eco-system centric big data processing in Microsoft Azure and Amazon AWS. In this talk we present some of the lessons learned and architectural considerations for cloud-based deployments including security, fault tolerance and auto-scaling.
We look at how Hortonworks Data Cloud and Cloudbreak can automate that scaling of Hadoop clusters, showing how it can react dynamically to workloads, and what that can deliver in cost-effective Hadoop-in-cloud deployments.
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...DataWorks Summit There have been many voices discussing how to architect streaming
applications on Hadoop. Before now, there have been very few worked
examples existing within the open source. Apache Metron (Incubating) is a
streaming advanced analytics cybersecurity application which utilizes
the components within the Hadoop stack as its platform.
We will attempt to go beyond theoretical discussions of Kappa vs Lambda
architectures and describe the nuts and bolts of a streaming
architecture that enables advanced analytics in Hadoop. We will discuss
the componentry that we had to build and what we could utilize. We will
discuss why we made the architectural decisions that we made and how
they fit together to knit together a coherent application on top of many
different Hadoop ecosystem projects.
We will also discuss the domain specific language that we created out of
necessity to enable a pluggable layer to enable user defined enrichments.
We will discuss how this helped make Metron less rigid and easier to
use. We will also candidly discuss mistakes that we made early on.
Securing Spark Applications
Securing Spark ApplicationsDataWorks Summit/Hadoop Summit This document discusses securing Spark applications. It covers encryption to protect data in transit and at rest, authentication using Kerberos to identify users, and authorization for access control through tools like Sentry and a proposed RecordService. While Spark can be secured today by leveraging Hadoop security, continued work is needed for easier encryption, improved Kerberos support for long-running jobs, and row/column-level authorization beyond file permissions.
Embeddable data transformation for real time streams
Embeddable data transformation for real time streamsJoey Echeverria This document summarizes Joey Echeverria's presentation on embeddable data transformation for real-time streams. Some key points include:
- Stream processing requires the ability to perform common data transformations like filtering, extracting, projecting, and aggregating on streaming data.
- Tools like Apache Storm, Spark, and Flink can be used to build stream processing topologies and jobs, but also have limitations for embedding transformations.
- Rocana Transform provides a library and DSL for defining reusable data transformation configurations that can be run within different stream processing systems or in batch jobs.
- The library supports common transformations as well as custom actions defined through Java. Configurations can extract metrics, parse logs, and perform
Enabling Modern Application Architecture using Data.gov open government data
Enabling Modern Application Architecture using Data.gov open government dataDataWorks Summit Big Data and the Internet of Things (IoT) have forced businesses and the Federal Government to reevaluate their existing data strategies and adopt a more modern data architecture. With the advent of the connected data platform, migrating or building data-driven applications that take advantage of data-in-motion and data-at-rest can be a daunting journey to undertake. Scaling, reusability, and achieving operational agility are just some of the common pitfalls associated with existing software architectures. How do we embrace this paradigm shift? Adopting agile methodologies and emerging development practices such as Microservices and DevOps offer greater agility and operational efficiency enabling the government to rapidly build modern data-driven applications.
During this talk and demonstration, we will show how the federal government can unleash the true power of the connected data platform with modern data-driven applications.
Connected Data Platform:
• Hortonworks DataFlow
o Using Apache NiFi for capturing data at the edge of the data lake & managing the flow of data to the data platform
o Apache Storm for complex event processing and stream processing
• Hortonworks Data Platform
o Apache Accumulo for scalability and cell-level security
o Apache YARN for resource management
• Modern Data-Driven Applications
o Microservices: a software architecture practice for designing software applications as suites of independently deployable services, promoting componentization, single responsibility & scalability. Adopting a Microservices mindset enables the government to be technology agnostic: using the best tool or programming language for the job.
♣ Demoed REST API’s on-top of Apache Accumulo. (Spark-Java, AngularJS/Typescript)
o DevOps: A culture and practice that breaks down the silos found between development and operations teams in traditional software practices.
♣ CI / CD pipelines, automated build kick-offs using containers (Docker, Jenkins)
This talk will lay out a basic environment for promoting greater agility and operational efficiency for the federal government while taking advantage of a connected data platform.
Boost Performance with Scala – Learn From Those Who’ve Done It! 
Boost Performance with Scala – Learn From Those Who’ve Done It! Cécile Poyet Scalding is a scala DSL for Cascading. Run on Hadoop, it’s a concise, functional, and very efficient way to build big data applications. One significant benefit of Scalding is that it allows easy porting of Scalding apps from MapReduce to newer, faster execution fabrics.
In this webinar, Cyrille Chépélov, of Transparency Rights Management, will share how his organization boosted the performance of their Scalding apps by over 50% by moving away from MapReduce to Cascading 3.0 on Apache Tez. Dhruv Kumar, Hortonworks Partner Solution Engineer, will then explain how you can interact with data on HDP using Scala and leverage Scala as a programming language to develop Big Data applications.
Spark in the Enterprise - 2 Years Later by Alan Saldich
Spark in the Enterprise - 2 Years Later by Alan SaldichSpark Summit Over the past 2 years, Cloudera has focused on improving and supporting Apache Spark. They have integrated Spark with Hadoop components like YARN, HBase, and Kafka. Cloudera engineers have also contributed security, monitoring, and governance features to Spark. More than 200 customers now use Spark for tasks like ETL, machine learning, and streaming analytics. Customers want Spark to have security comparable to databases, high performance, and simplicity. Cloudera is developing technologies like Sentry and Kudu to meet these needs and make Spark more powerful and useful for enterprises.
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit Kerberos is the ubiquitous authentication mechanism when it comes to secure any Hadoop Services. With recent updates in Hadoop core and various Apache Hadoop components, inherent Kerberos support has matured and has come a long way.
Understanding & configuring Kerberos is still a challenge but even more painful & frustrating is troubleshooting a Kerberos issue. There are lot of things (small & big) that can go wrong (and will go wrong!). This talk covers the Kerberos debugging part in detail and discusses the tools & tricks that can be used to narrow down any Kerberos issue.
Rather than discussing the issues and their resolution, we will focus on how to approach a Kerberos problem and do's / dont's in Kerberos scene. This talk will provide a step by step guide that will equip the audience for troubleshooting future Kerberos problems.
Agenda is to discuss:
- Systematic approach to Kerberos troubleshooting
- Kerberos Tools available in Hadoop arsenal
- Tips & Tricks to narrow down Kerberos issues quickly
- Some nasty Kerberos issues from Support trenches
Some prior knowledge on Kerberos basics will be appreciated but is not a prerequisite.
Speaker:
Vipin Rathor, Sr. Product Specialist (HDP Security), Hortonworks
High-Scale Entity Resolution in Hadoop
High-Scale Entity Resolution in HadoopDataWorks Summit/Hadoop Summit eBay maintains hundreds of millions of accounts across its properties that are unstructured and in different formats. Identifying which accounts belong to the same person enables eBay to personalize customer experiences, provide customer service, and fight fraud. MapReduce provides a robust design pattern to simplify high-scale entity resolution through parallelized modular operations, including linking accounts pairwise, identifying connected components through iterative MapReduce jobs, and validating the results.
Built-In Security for the Cloud
Built-In Security for the CloudDataWorks Summit Today enterprises desire to move more and more of their data lakes to the cloud to help them execute faster, increase productivity, drive innovation while leveraging the scale and flexibility of the cloud. However, such gains come with risks and challenges in the areas of data security, privacy, and governance. In this talk we cover how enterprises can overcome governance and security obstacles to leverage these new advances that the cloud can provide to ease the management of their data lakes in the cloud. We will also show how the enterprise can have consistent governance and security controls in the cloud for their ephemeral analytic workloads in a multi-cluster cloud environment without sacrificing any of the data security and privacy/compliance needs that their business context demands. Additionally, we will outline some use cases and patterns as well as best practices to rationally manage such a multi-cluster data lake infrastructure in the cloud.
Speaker:
Jeff Sposetti, Product Management, Hortonworks
Real time fraud detection at 1+M scale on hadoop stack
Real time fraud detection at 1+M scale on hadoop stackDataWorks Summit/Hadoop Summit Rocketfuel processes over 120 billion ad auctions per day and needs to detect fraud in real time to prevent losses. They developed Helios, which ingests event data from Kafka and HDFS into Storm in real time, joins the streams in HBase, then runs MapReduce jobs hourly to populate an OLAP cube for analyzing feature vectors and detecting fraud patterns. This architecture on Hadoop allows them to easily scale real-time processing and experiment with different configurations to quickly react to fraud.
Real Time Machine Learning Visualization with Spark
Real Time Machine Learning Visualization with SparkDataWorks Summit/Hadoop Summit This document discusses enabling real-time machine learning visualization with Spark. It presents a callback interface for Spark ML algorithms to send messages during training and a task channel to deliver messages from the Spark driver to a client. The messages are pushed to a browser using server-sent events and HTTP chunked responses. This allows visualizing training metrics, determining early stopping, and monitoring algorithm convergence in real time.
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionDataWorks Summit/Hadoop Summit The document discusses Hadoop integration with cloud storage. It describes the Hadoop-compatible file system architecture, which allows applications to work with different storage systems transparently. Recent enhancements to the S3A connector for Amazon S3 are discussed, including performance improvements and support for encryption. Benchmark results show significant performance gains for Hive queries running on S3A compared to earlier versions. Upcoming work on consistency, output committers, and abstraction layers is outlined to further improve object store integration.
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies 
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies DataWorks Summit/Hadoop Summit Apache Atlas provides metadata services and a centralized metadata repository for Hadoop platforms. It aims to enable data governance across structured and unstructured data through hierarchical taxonomies. Upcoming features include expanded dataset lineage tracking and integration with Apache Kafka and Ranger for dynamic access policy management. Challenges of big data management include scaling traditional tools to handle large volumes of entities and metadata, and Atlas addresses this through its decentralized and metadata-driven approach.
Accelerate Your Big Data Analytics Efforts with SAS and Hadoop
Accelerate Your Big Data Analytics Efforts with SAS and HadoopDataWorks Summit Analytics and machine learning continue to be the top use cases for deploying big data platforms such as Hadoop. SAS recognised the potential and power of Hadoop platform early on and has been integrating analytical solutions with Hadoop to leverage the power and flexibility of Hadoop for analytical workloads. The combination of SAS and Hadoop offers developers and organisations an approach that can accelerate the development and deployment of big data analytics applications that are mature, proven and scalable. Furthermore, by giving developers and analysts analytical applications that are rich, proven and collaborative, SAS allows more users across different skill levels to unleash the value of data stored in big data platform more easily and quickly.
In this session, we will cover common big data analytics use cases, the depth and breadth of SAS analytical capabilities on Hadoop, and how SAS solutions are integrated into the Hadoop ecosystem via technologies such as Hive, YARN and Spark.
Speaker
Felix Liao, SAS Institute Australia & New Zealand
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...Hortonworks As Hadoop becomes the defacto big data platform, enterprises deploy HDP across wide range of physical and virtual environments spanning private and public clouds. This session will cover key considerations for cloud deployment and showcase Cloudbreak for simple and consistent deployment across cloud providers of choice.
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage SubsystemDataWorks Summit/Hadoop Summit Sanjay Radia presents on evolving HDFS to support a generalized storage subsystem. HDFS currently scales well to large clusters and storage sizes but faces challenges with small files and blocks. The solution is to (1) only keep part of the namespace in memory to scale beyond memory limits and (2) use block containers of 2-16GB to reduce block metadata and improve scaling. This will generalize the storage layer to support containers for multiple use cases beyond HDFS blocks.
NLP Structured Data Investigation on Non-Text
NLP Structured Data Investigation on Non-TextDataWorks Summit/Hadoop Summit Casey Stella presents on using natural language processing (NLP) techniques like word2vec to analyze structured medical data. Clinical encounters can be viewed as "sentences" with vital signs, labs, procedures, diagnoses and drugs as "words". Word2vec is applied to a dataset of 197,340 clinical records to learn relationships between these clinical concepts without domain expertise. The results are explored using Spark and IPython Notebook on Hadoop.
More Related Content
What's hot (20)
Securing Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise ContextDataWorks Summit/Hadoop Summit Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
Innovation in the Enterprise Rent-A-Car Data Warehouse
Innovation in the Enterprise Rent-A-Car Data WarehouseDataWorks Summit Big Data adoption is a journey. Depending on the business the process can take weeks, months, or even years. With any transformative technology the challenges have less to do with the technology and more to do with how a company adapts itself to a new way of thinking about data. Building a Center of Excellence is one way for IT to help drive success.
This talk will explore Enterprise Holdings Inc. (which operates the Enterprise Rent-A-Car, National Car Rental and Alamo Rent A Car) and their experience with Big Data. EHI’s journey started in 2013 with Hadoop as a POC and today are working to create the next generation data warehouse in Microsoft’s Azure cloud utilizing a lambda architecture.
We’ll discuss the Center of Excellence, the roles in the new world, share the things which worked well, and rant about those which didn’t.
No deep Hadoop knowledge is necessary, architect or executive level.
Realizing the Promise of Portable Data Processing with Apache Beam
Realizing the Promise of Portable Data Processing with Apache BeamDataWorks Summit The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the Big Data ecosystem together; it enables users to "run-anything-anywhere".
This talk will briefly cover the capabilities of the Beam model for data processing, as well as the current state of the Beam ecosystem. We'll discuss Beam architecture and dive into the portability layer. We'll offer a technical analysis of the Beam's powerful primitive operations that enable true and reliable portability across diverse environments. Finally, we'll demonstrate a complex pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Spark on Amazon Web Services, Apache Flink on Google Cloud, Apache Apex on-premise), and give a glimpse at some of the challenges Beam aims to address in the future.
Speaker
Davor Bonaci, Senior Software Engineer, Google
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
Druid: Sub-Second OLAP queries over Petabytes of Streaming DataDataWorks Summit When interacting with analytics dashboards in order to achieve a smooth user experience, two major key requirements are sub-second response time and data freshness. Cluster computing frameworks such as Hadoop or Hive/Hbase work well for storing large volumes of data, although they are not optimized for ingesting streaming data and making it available for queries in realtime. Also, long query latencies make these systems sub-optimal choices for powering interactive dashboards and BI use-cases.
In this talk we will present Druid as a complementary solution to existing hadoop based technologies. Druid is an open-source analytics data store, designed from scratch, for OLAP and business intelligence queries over massive data streams. It provides low latency realtime data ingestion and fast sub-second adhoc flexible data exploration queries.
Many large companies are switching to Druid for analytics, and we will cover how druid is able to handle massive data streams and why it is a good fit for BI use cases.
Agenda -
1) Introduction and Ideal Use cases for Druid
2) Data Architecture
3) Streaming Ingestion with Kafka
4) Demo using Druid, Kafka and Superset.
5) Recent Improvements in Druid moving from lambda architecture to Exactly once Ingestion
6) Future Work
Cloudy with a chance of Hadoop - real world considerations
Cloudy with a chance of Hadoop - real world considerationsDataWorks Summit Over the last eighteen months, we have seen significant adoption of Hadoop eco-system centric big data processing in Microsoft Azure and Amazon AWS. In this talk we present some of the lessons learned and architectural considerations for cloud-based deployments including security, fault tolerance and auto-scaling.
We look at how Hortonworks Data Cloud and Cloudbreak can automate that scaling of Hadoop clusters, showing how it can react dynamically to workloads, and what that can deliver in cost-effective Hadoop-in-cloud deployments.
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...DataWorks Summit There have been many voices discussing how to architect streaming
applications on Hadoop. Before now, there have been very few worked
examples existing within the open source. Apache Metron (Incubating) is a
streaming advanced analytics cybersecurity application which utilizes
the components within the Hadoop stack as its platform.
We will attempt to go beyond theoretical discussions of Kappa vs Lambda
architectures and describe the nuts and bolts of a streaming
architecture that enables advanced analytics in Hadoop. We will discuss
the componentry that we had to build and what we could utilize. We will
discuss why we made the architectural decisions that we made and how
they fit together to knit together a coherent application on top of many
different Hadoop ecosystem projects.
We will also discuss the domain specific language that we created out of
necessity to enable a pluggable layer to enable user defined enrichments.
We will discuss how this helped make Metron less rigid and easier to
use. We will also candidly discuss mistakes that we made early on.
Securing Spark Applications
Securing Spark ApplicationsDataWorks Summit/Hadoop Summit This document discusses securing Spark applications. It covers encryption to protect data in transit and at rest, authentication using Kerberos to identify users, and authorization for access control through tools like Sentry and a proposed RecordService. While Spark can be secured today by leveraging Hadoop security, continued work is needed for easier encryption, improved Kerberos support for long-running jobs, and row/column-level authorization beyond file permissions.
Embeddable data transformation for real time streams
Embeddable data transformation for real time streamsJoey Echeverria This document summarizes Joey Echeverria's presentation on embeddable data transformation for real-time streams. Some key points include:
- Stream processing requires the ability to perform common data transformations like filtering, extracting, projecting, and aggregating on streaming data.
- Tools like Apache Storm, Spark, and Flink can be used to build stream processing topologies and jobs, but also have limitations for embedding transformations.
- Rocana Transform provides a library and DSL for defining reusable data transformation configurations that can be run within different stream processing systems or in batch jobs.
- The library supports common transformations as well as custom actions defined through Java. Configurations can extract metrics, parse logs, and perform
Enabling Modern Application Architecture using Data.gov open government data
Enabling Modern Application Architecture using Data.gov open government dataDataWorks Summit Big Data and the Internet of Things (IoT) have forced businesses and the Federal Government to reevaluate their existing data strategies and adopt a more modern data architecture. With the advent of the connected data platform, migrating or building data-driven applications that take advantage of data-in-motion and data-at-rest can be a daunting journey to undertake. Scaling, reusability, and achieving operational agility are just some of the common pitfalls associated with existing software architectures. How do we embrace this paradigm shift? Adopting agile methodologies and emerging development practices such as Microservices and DevOps offer greater agility and operational efficiency enabling the government to rapidly build modern data-driven applications.
During this talk and demonstration, we will show how the federal government can unleash the true power of the connected data platform with modern data-driven applications.
Connected Data Platform:
• Hortonworks DataFlow
o Using Apache NiFi for capturing data at the edge of the data lake & managing the flow of data to the data platform
o Apache Storm for complex event processing and stream processing
• Hortonworks Data Platform
o Apache Accumulo for scalability and cell-level security
o Apache YARN for resource management
• Modern Data-Driven Applications
o Microservices: a software architecture practice for designing software applications as suites of independently deployable services, promoting componentization, single responsibility & scalability. Adopting a Microservices mindset enables the government to be technology agnostic: using the best tool or programming language for the job.
♣ Demoed REST API’s on-top of Apache Accumulo. (Spark-Java, AngularJS/Typescript)
o DevOps: A culture and practice that breaks down the silos found between development and operations teams in traditional software practices.
♣ CI / CD pipelines, automated build kick-offs using containers (Docker, Jenkins)
This talk will lay out a basic environment for promoting greater agility and operational efficiency for the federal government while taking advantage of a connected data platform.
Boost Performance with Scala – Learn From Those Who’ve Done It!
Boost Performance with Scala – Learn From Those Who’ve Done It! Cécile Poyet Scalding is a scala DSL for Cascading. Run on Hadoop, it’s a concise, functional, and very efficient way to build big data applications. One significant benefit of Scalding is that it allows easy porting of Scalding apps from MapReduce to newer, faster execution fabrics.
In this webinar, Cyrille Chépélov, of Transparency Rights Management, will share how his organization boosted the performance of their Scalding apps by over 50% by moving away from MapReduce to Cascading 3.0 on Apache Tez. Dhruv Kumar, Hortonworks Partner Solution Engineer, will then explain how you can interact with data on HDP using Scala and leverage Scala as a programming language to develop Big Data applications.
Spark in the Enterprise - 2 Years Later by Alan Saldich
Spark in the Enterprise - 2 Years Later by Alan SaldichSpark Summit Over the past 2 years, Cloudera has focused on improving and supporting Apache Spark. They have integrated Spark with Hadoop components like YARN, HBase, and Kafka. Cloudera engineers have also contributed security, monitoring, and governance features to Spark. More than 200 customers now use Spark for tasks like ETL, machine learning, and streaming analytics. Customers want Spark to have security comparable to databases, high performance, and simplicity. Cloudera is developing technologies like Sentry and Kudu to meet these needs and make Spark more powerful and useful for enterprises.
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit Kerberos is the ubiquitous authentication mechanism when it comes to secure any Hadoop Services. With recent updates in Hadoop core and various Apache Hadoop components, inherent Kerberos support has matured and has come a long way.
Understanding & configuring Kerberos is still a challenge but even more painful & frustrating is troubleshooting a Kerberos issue. There are lot of things (small & big) that can go wrong (and will go wrong!). This talk covers the Kerberos debugging part in detail and discusses the tools & tricks that can be used to narrow down any Kerberos issue.
Rather than discussing the issues and their resolution, we will focus on how to approach a Kerberos problem and do's / dont's in Kerberos scene. This talk will provide a step by step guide that will equip the audience for troubleshooting future Kerberos problems.
Agenda is to discuss:
- Systematic approach to Kerberos troubleshooting
- Kerberos Tools available in Hadoop arsenal
- Tips & Tricks to narrow down Kerberos issues quickly
- Some nasty Kerberos issues from Support trenches
Some prior knowledge on Kerberos basics will be appreciated but is not a prerequisite.
Speaker:
Vipin Rathor, Sr. Product Specialist (HDP Security), Hortonworks
High-Scale Entity Resolution in Hadoop
High-Scale Entity Resolution in HadoopDataWorks Summit/Hadoop Summit eBay maintains hundreds of millions of accounts across its properties that are unstructured and in different formats. Identifying which accounts belong to the same person enables eBay to personalize customer experiences, provide customer service, and fight fraud. MapReduce provides a robust design pattern to simplify high-scale entity resolution through parallelized modular operations, including linking accounts pairwise, identifying connected components through iterative MapReduce jobs, and validating the results.
Built-In Security for the Cloud
Built-In Security for the CloudDataWorks Summit Today enterprises desire to move more and more of their data lakes to the cloud to help them execute faster, increase productivity, drive innovation while leveraging the scale and flexibility of the cloud. However, such gains come with risks and challenges in the areas of data security, privacy, and governance. In this talk we cover how enterprises can overcome governance and security obstacles to leverage these new advances that the cloud can provide to ease the management of their data lakes in the cloud. We will also show how the enterprise can have consistent governance and security controls in the cloud for their ephemeral analytic workloads in a multi-cluster cloud environment without sacrificing any of the data security and privacy/compliance needs that their business context demands. Additionally, we will outline some use cases and patterns as well as best practices to rationally manage such a multi-cluster data lake infrastructure in the cloud.
Speaker:
Jeff Sposetti, Product Management, Hortonworks
Real time fraud detection at 1+M scale on hadoop stack
Real time fraud detection at 1+M scale on hadoop stackDataWorks Summit/Hadoop Summit Rocketfuel processes over 120 billion ad auctions per day and needs to detect fraud in real time to prevent losses. They developed Helios, which ingests event data from Kafka and HDFS into Storm in real time, joins the streams in HBase, then runs MapReduce jobs hourly to populate an OLAP cube for analyzing feature vectors and detecting fraud patterns. This architecture on Hadoop allows them to easily scale real-time processing and experiment with different configurations to quickly react to fraud.
Real Time Machine Learning Visualization with Spark
Real Time Machine Learning Visualization with SparkDataWorks Summit/Hadoop Summit This document discusses enabling real-time machine learning visualization with Spark. It presents a callback interface for Spark ML algorithms to send messages during training and a task channel to deliver messages from the Spark driver to a client. The messages are pushed to a browser using server-sent events and HTTP chunked responses. This allows visualizing training metrics, determining early stopping, and monitoring algorithm convergence in real time.
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionDataWorks Summit/Hadoop Summit The document discusses Hadoop integration with cloud storage. It describes the Hadoop-compatible file system architecture, which allows applications to work with different storage systems transparently. Recent enhancements to the S3A connector for Amazon S3 are discussed, including performance improvements and support for encryption. Benchmark results show significant performance gains for Hive queries running on S3A compared to earlier versions. Upcoming work on consistency, output committers, and abstraction layers is outlined to further improve object store integration.
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies DataWorks Summit/Hadoop Summit Apache Atlas provides metadata services and a centralized metadata repository for Hadoop platforms. It aims to enable data governance across structured and unstructured data through hierarchical taxonomies. Upcoming features include expanded dataset lineage tracking and integration with Apache Kafka and Ranger for dynamic access policy management. Challenges of big data management include scaling traditional tools to handle large volumes of entities and metadata, and Atlas addresses this through its decentralized and metadata-driven approach.
Accelerate Your Big Data Analytics Efforts with SAS and Hadoop
Accelerate Your Big Data Analytics Efforts with SAS and HadoopDataWorks Summit Analytics and machine learning continue to be the top use cases for deploying big data platforms such as Hadoop. SAS recognised the potential and power of Hadoop platform early on and has been integrating analytical solutions with Hadoop to leverage the power and flexibility of Hadoop for analytical workloads. The combination of SAS and Hadoop offers developers and organisations an approach that can accelerate the development and deployment of big data analytics applications that are mature, proven and scalable. Furthermore, by giving developers and analysts analytical applications that are rich, proven and collaborative, SAS allows more users across different skill levels to unleash the value of data stored in big data platform more easily and quickly.
In this session, we will cover common big data analytics use cases, the depth and breadth of SAS analytical capabilities on Hadoop, and how SAS solutions are integrated into the Hadoop ecosystem via technologies such as Hive, YARN and Spark.
Speaker
Felix Liao, SAS Institute Australia & New Zealand
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...Hortonworks As Hadoop becomes the defacto big data platform, enterprises deploy HDP across wide range of physical and virtual environments spanning private and public clouds. This session will cover key considerations for cloud deployment and showcase Cloudbreak for simple and consistent deployment across cloud providers of choice.
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies DataWorks Summit/Hadoop Summit
Viewers also liked (20)
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage SubsystemDataWorks Summit/Hadoop Summit Sanjay Radia presents on evolving HDFS to support a generalized storage subsystem. HDFS currently scales well to large clusters and storage sizes but faces challenges with small files and blocks. The solution is to (1) only keep part of the namespace in memory to scale beyond memory limits and (2) use block containers of 2-16GB to reduce block metadata and improve scaling. This will generalize the storage layer to support containers for multiple use cases beyond HDFS blocks.
NLP Structured Data Investigation on Non-Text
NLP Structured Data Investigation on Non-TextDataWorks Summit/Hadoop Summit Casey Stella presents on using natural language processing (NLP) techniques like word2vec to analyze structured medical data. Clinical encounters can be viewed as "sentences" with vital signs, labs, procedures, diagnoses and drugs as "words". Word2vec is applied to a dataset of 197,340 clinical records to learn relationships between these clinical concepts without domain expertise. The results are explored using Spark and IPython Notebook on Hadoop.
Parcours « bâtiment et travaux publics » au Salesforce World Tour Paris
Parcours « bâtiment et travaux publics » au Salesforce World Tour ParisSalesforce France Le Salesforce World Tour Paris est le plus grand évènement annuel gratuit dédié au CRM, au cloud et à la transformation de l'entreprise réunissant plus de 5 000 professionnels, experts, partenaires, développeurs et entrepreneurs.
Lego Social Media Analysis Q4 2015
Lego Social Media Analysis Q4 2015Unmetric Take a deep dive into the social media habits of one of the most popular toy brands in the world. Lego uses the launch of various new products to keep its global audience hyper-engaged.
Salesforce - Client Strategy Presentation
Salesforce - Client Strategy PresentationSophia Myers I prepared and delivered this presentation to describe our strategy to the client and get buy-in from other stakeholders. It includes our vision for the client and the value it would bring to their organization.
Data Preparation of Data Science
Data Preparation of Data ScienceDataWorks Summit/Hadoop Summit This document discusses the importance of data preparation and understanding for data science projects. It notes that most of the work in any data project involves cleaning and preparing the data. The document emphasizes gaining both syntactic and semantic understanding of data through techniques like determining true data types, analyzing data density and distributions, and identifying equivalences. It also discusses how data understanding can reveal problems over time. Finally, it stresses the need for data science teams to be involved in the hands-on work of data transformation and understanding.
Lego
LegoRehan Fazal Lego began as a wooden toy company in Denmark in the 1890s and started producing plastic toys in 1947. It is best known for its interlocking plastic bricks which were invented in 1949. Since then, Lego has expanded to include thousands of sets across various themes. The precision engineering of the bricks allows them to stick together firmly while being easily separated. Lego pieces are manufactured from ABS plastic in plants around the world. Technological advancements include Lego Mindstorms robotics sets in 1998 and Lego-themed amusement parks which showcase how far the simple plastic bricks have come.
Lambda-less Stream Processing @Scale in LinkedIn 
Lambda-less Stream Processing @Scale in LinkedIn DataWorks Summit/Hadoop Summit Lambda-less Stream Processing @Scale in LinkedIn
The document discusses challenges with stream processing including data accuracy and reprocessing. It proposes a "lambda-less" approach using windowed computations and handling late and out-of-order events to produce eventually correct results. Samza is used in LinkedIn's implementation to store streaming data locally using RocksDB for processing within configurable windows. The approach avoids code duplication compared to traditional lambda architectures while still supporting reprocessing through resetting offsets. Challenges remain in merging online and reprocessed results at large scale.
Apache Hive 2.0: SQL, Speed, Scale
Apache Hive 2.0: SQL, Speed, ScaleDataWorks Summit/Hadoop Summit Apache Hive 2.0 provides major new features for SQL on Hadoop such as:
- HPLSQL which adds procedural SQL capabilities like loops and branches.
- LLAP which enables sub-second queries through persistent daemons and in-memory caching.
- Using HBase as the metastore which speeds up query planning times for queries involving thousands of partitions.
- Improvements to Hive on Spark and the cost-based optimizer.
- Many bug fixes and under-the-hood improvements were also made while maintaining backwards compatibility where possible.
The Stream is the Database - Revolutionizing Healthcare Data Architecture
The Stream is the Database - Revolutionizing Healthcare Data ArchitectureDataWorks Summit/Hadoop Summit This document discusses using event streams as the system of record for data, rather than traditional databases. It argues that streams can serve as the single source of truth for data, providing benefits like data lineage, auditing, and integrity. It also describes how healthcare company Liaison uses a streaming platform from MapR to power their data integration platform, gaining the advantages of streams while meeting various compliance requirements.
LEGO is full of WIN
LEGO is full of WINRoo Reynolds My presentation at Interesting 2008. 21st June 2008. 30 slides in exactly 3 minutes.
http://rooreynolds.com/2008/06/22/lego-is-full-of-win-my-interesting-2008-talk/
YARN Federation 
YARN Federation DataWorks Summit/Hadoop Summit This document discusses YARN federation, which allows multiple YARN clusters to be connected together. It summarizes:
- YARN is used at Microsoft for resource management but faces challenges of large scale and diverse workloads. Federation aims to address this.
- The federation architecture connects multiple independent YARN clusters through centralized services for routing, policies, and state. Applications are unaware and can seamlessly run across clusters.
- Federation policies determine how work is routed and scheduled across clusters, balancing objectives like load balancing, scaling, fairness, and isolation. A spectrum of policy options is discussed from full partitioning to full replication to dynamic partial replication.
- A demo is presented showing a job running across
LEGO
LEGOVaibhavi Dalvi The document discusses the product life cycle of Lego toys from their introduction in the 1940s through periods of growth, maturity, and decline. In the late 1990s, Lego began facing challenges as it entered the maturity stage, with increasing competition and changing consumer behavior. A series of missteps in the early 2000s, including over-expanding product lines and venturing into unrelated businesses, led to major financial losses. However, under new CEO Jorgen Vig Knudstorp in 2004, Lego implemented a successful turnaround plan focused on profitability, simplifying operations, and refocusing on the core brick-building toy business. These changes helped stabilize the company's decline and set it back on a path of growth.
The Role of Analytics in Smart Cities
The Role of Analytics in Smart CitiesManthan Analytics can be used to improve living efficiency and provide better urban governance in smart cities.
BOOSTING SALES AND OPERATIONS WITH SALESFORCE SALES CLOUD
BOOSTING SALES AND OPERATIONS WITH SALESFORCE SALES CLOUDDazeworks A company specializing in complete turn-key hospitality solutions with a presence in over 50 states were in need of a platform which could scale and manage the pace of their business growth.
HDFS Erasure Coding in Action 
HDFS Erasure Coding in Action DataWorks Summit/Hadoop Summit This document discusses HDFS Erasure Coding and its usage at Yahoo Japan. It begins with an overview of erasure coding, how it is implemented in HDFS, and compares it to replication. Test results show the write performance is lower for erasure coding while read performance is similar. Yahoo Japan uses erasure coding for cold weblog data, reducing storage costs by 65% compared to replication. Future plans include supporting additional codecs and features to provide more usability.
The Elephant in the Clouds
The Elephant in the CloudsDataWorks Summit/Hadoop Summit The document discusses key considerations for running Hadoop in the cloud. It notes that running Hadoop in the cloud provides unlimited elastic scale, ephemeral and long-running workloads, no upfront hardware costs, and IT and business agility. It outlines some of the major cloud Hadoop solutions according to a Forrester Wave report and discusses architectural considerations like shared data and storage, on-demand ephemeral workloads, elastic resource management, and shared metadata, security, and governance.
Dataflow with Apache NiFi - Crash Course - HS16SJ
Dataflow with Apache NiFi - Crash Course - HS16SJDataWorks Summit/Hadoop Summit This document provides an overview of Apache NiFi, a dataflow management software. It begins with an introduction to dataflow and challenges in moving data effectively. It then discusses key features of Apache NiFi like guaranteed delivery, data buffering, and data provenance. The document outlines NiFi's architecture including repositories and extension points. It also advertises an upcoming Birds of a Feather session on streaming, dataflow and cybersecurity. Finally, it encourages learning more about NiFi and getting involved in the community.
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit The document discusses bringing multi-tenancy to Apache Zeppelin through the use of Apache Livy. Livy is an open-source REST interface that allows interacting with Spark from anywhere and enables features like multi-user sessions and security. It improves on previous versions of interactive analysis in Zeppelin by allowing custom user sessions through Livy and improving security and isolation between users through mechanisms like SPNEGO and impersonation. The integration of Livy provides multi-tenancy, security, and isolation for interactive analysis in Zeppelin.
LEGO-Case
LEGO-CaseOliver Kuhn LEGO engages in brand co-creation with consumers and other stakeholders through dialogue, access, transparency, and risk management. As the main stakeholder, consumers provide ideas and feedback that help develop new LEGO products. Other stakeholders like employees, business partners, and educational institutions also contribute to co-creation. While co-creation brings benefits, it also carries risks like loss of control or reputation damage that companies must navigate.
The Stream is the Database - Revolutionizing Healthcare Data Architecture
The Stream is the Database - Revolutionizing Healthcare Data ArchitectureDataWorks Summit/Hadoop Summit
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
Similar to LEGO: Data Driven Growth Hacking Powered by Big Data (20)
SAP and Salesforce Integration
SAP and Salesforce IntegrationGlenn Johnson This document provides an overview of integrating Salesforce with SAP ERP using Magic xpi Integration Platform. It discusses how the platform can automate processes running between Salesforce and ERP like opportunity to order conversion, inventory availability, and more. It then covers considerations for integrating SAP and Salesforce like integration methods, objects, metadata, bulk jobs, and replication. The document concludes with an overview of Magic xpi's features like visual development, orchestration, monitoring, security and its use for mobile integration.
Dynamics AX and Salesforce Integration
Dynamics AX and Salesforce IntegrationGlenn Johnson Integrate Your Salesforce to Dynamics ERP. Learn how-to automate business processes running between Salesforce.com CRM and Dynamics AX ERP systems.
Salesforce & SAP Integration
Salesforce & SAP IntegrationRaymond Gao Cloud to onpremise integration with Salesforce & SAP technologies
see: https://github.com/raygao/RaysCruiserDemo
Limitless Data, Rapid Discovery, Powerful Insight: How to Connect Cloudera to...
Limitless Data, Rapid Discovery, Powerful Insight: How to Connect Cloudera to...Cloudera, Inc. What if…
…your data stores were limitless and accessible?
…data discovery was fast… really fast?
…connectivity was so seamless you could almost take it for granted?
And what if you could do all this with your preferred BI tool?
Learn how to integrate Cloudera Enterprise with SAP Lumira via embedded connectivity from Simba Technologies.
In this interactive webinar, experts from Cloudera, SAP, and Simba Technologies will introduce strategies for overcoming current data-discovery challenges, show you how to achieve powerful analytical insight, and demonstrate how to integrate Cloudera Enterprise with SAP Lumira.
How does Microsoft solve Big Data?
How does Microsoft solve Big Data?James Serra So you got a handle on what Big Data is and how you can use it to find business value in your data. Now you need an understanding of the Microsoft products that can be used to create a Big Data solution. Microsoft has many pieces of the puzzle and in this presentation I will show how they fit together. How does Microsoft enhance and add value to Big Data? From collecting data, transforming it, storing it, to visualizing it, I will show you Microsoft’s solutions for every step of the way
Microsoft Data Science Technologies 201608
Microsoft Data Science Technologies 201608Mark Tabladillo Delivered to SQL Saturday Columbus, GA
Microsoft provides several technologies which can be used for casual to serious data science. This presentation provides an authoritative overview of two major categories: products and services. The products include: SQL Server Analysis Services, Excel Add-in for SSAS, Semantic Search, SQL Server R Services, Microsoft R Technologies, and F#. The services include Cortana Intelligence and Bing Predicts. These technologies have been used by the presenter in various companies and industries, and he will be speaking toward how Microsoft uses these technologies today for its largest Azure customers.
Fusion - IBANK
Fusion - IBANKibankuk The document discusses Oracle's Fusion Middleware strategy and products. It describes Oracle's service-oriented architecture called Fusion Architecture, which aims to provide flexible business applications and processes through a unified multi-channel portal and service bus. It also discusses Oracle's products for application development, integration, business intelligence, collaboration, and voice access that are part of its Fusion Middleware platform.
Sf big analytics_2018_04_18: Evolution of the GoPro's data platform
Sf big analytics_2018_04_18: Evolution of the GoPro's data platformChester Chen Talk 1 : Evolution of the GoPro's data platform
In this talk, we will share GoPro’s experiences in building Data Analytics Cluster in Cloud. We will discuss: evolution of data platform from fixed-size Hadoop clusters to Cloud-based Spark Cluster with Centralized Hive Metastore +S3: Cost Benefits and DevOp Impact; Configurable, spark-based batch Ingestion/ETL framework;
Migration Streaming framework to Cloud + S3;
Analytics metrics delivery with Slack integration;
BedRock: Data Platform Management, Visualization & Self-Service Portal
Visualizing Machine learning Features via Google Facets + Spark
Speakers: Chester Chen
Chester Chen is the Head of Data Science & Engineering, GoPro. Previously, he was the Director of Engineering at Alpine Data Lab.
David Winters
David is an Architect in the Data Science and Engineering team at GoPro and the creator of their Spark-Kafka data ingestion pipeline. Previously He worked at Apple & Splice Machines.
Hao Zou
Hao is a Senior big data engineer at Data Science and Engineering team. Previously He worked as Alpine Data Labs and Pivotal
TestGuild and QuerySurge Presentation -DevOps for Data Testing
TestGuild and QuerySurge Presentation -DevOps for Data TestingRTTS This slide deck is from one of our 4 webinars in our half-day series in conjunction with Test Guild.
Chris Thompson and Mike Calabrese, Senior Solution Architects and QuerySurge experts, provide great information, a demo and lots of humor in this webinar on how to implement DevOps for Data in your DataOps pipeline.
This webinar was performed in conjunction with Test Guild.
To watch the video, go to:
https://youtu.be/1ihuRPgY_rs
Architect day 20181128 - Afternoon Session
Architect day 20181128 - Afternoon SessionSalesforce - Sweden, Denmark, Norway The document discusses integration solutions from Salesforce and MuleSoft. It promotes the MuleSoft Anypoint Platform as a complete integration platform that can connect any application, data, or device using API-led connectivity. It positions the platform as future-proof and able to empower organizations to innovate faster. Specific capabilities and use cases mentioned include building a complete customer view, modernizing legacy systems, creating omnichannel experiences, and connecting e-commerce.
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...Big Data Spain http://www.bigdataspain.org/2014/conference/state-of-play-data-science-on-hadoop-in-2015-keynote
Machine Learning is not new. Big Machine Learning is qualitatively different: More data beats algorithm improvement, scale trumps noise and sample size effects, can brute-force manual tasks.
Session presented at Big Data Spain 2014 Conference
18th Nov 2014
Kinépolis Madrid
http://www.bigdataspain.org
Event promoted by: http://www.paradigmatecnologico.com
Slides: https://speakerdeck.com/bigdataspain/state-of-play-data-science-on-hadoop-in-2015-by-sean-owen-at-big-data-spain-2014
DRM Webinar Series, PART 3: Will DRM Integrate With Our Applications?
DRM Webinar Series, PART 3: Will DRM Integrate With Our Applications?US-Analytics In the third part of the series, we'll debunk myths around integrating DRM:
“It can’t automate or integrate with my non-Oracle products like SAP, Salesforce, Workday, or ServiceNow.”
“DRM doesn’t support a SaaS-based cloud architecture.”
“It doesn’t have delivered support for maintaining Oracle EPM products, like Essbase, Planning, HFM, and PBCS."
Ashish_Maheshwari_Data_Analyst
Ashish_Maheshwari_Data_AnalystAshish Maheshwari Ashish Maheshwari has over 8 years of experience in data modeling, integration, analytics, mining, ETL processes, BI applications, and data warehousing. He currently works as a Technical Lead at Ness Technologies, Bangalore and has experience working on projects for clients such as FGL, Marks, and Target. His skills include Oracle SQL, PL/SQL, Datastage, data warehousing concepts, R programming, and Agile development processes.
IBM Cloud pak for data brochure
IBM Cloud pak for data brochureSimon Harrison ACMA CGMA IBM Cloud Pak for Data is a unified platform that simplifies data collection, organization, and analysis through an integrated cloud-native architecture. It allows enterprises to turn data into insights by unifying various data sources and providing a catalog of microservices for additional functionality. The platform addresses challenges organizations face in leveraging data due to legacy systems, regulatory constraints, and time spent preparing data. It provides a single interface for data teams to collaborate and access over 45 integrated services to more efficiently gain insights from data.
(Updated) SharePoint & jQuery Guide
(Updated) SharePoint & jQuery GuideMark Rackley This document provides a summary of a presentation on using jQuery with SharePoint. It discusses:
1) Why jQuery is useful for SharePoint - it allows dynamic updates without custom code, improves visuals and usability, and can work around limitations like the list view threshold.
2) The basics of using jQuery with SharePoint, including common methods to interact with elements, attributes, and SharePoint list data via APIs.
3) Best practices for jQuery development, such as putting code in document ready functions, debugging techniques, and chaining methods to concisely select and update elements.
Designing big data analytics solutions on azure
Designing big data analytics solutions on azureMohamed Tawfik This document discusses designing big data analytics solutions on Azure. It provides an overview of Azure's data landscape and common architectural patterns and scenarios for building analytics solutions using various Azure data and analytics services. These include Azure SQL Data Warehouse, Azure Data Lake Store, Azure Data Factory, Azure Machine Learning, and Power BI for reporting and visualization. The document also discusses using these services to build solutions for scenarios like data warehousing, data lakes, ETL/ELT, machine learning, streaming analytics and more.
Bringing the Power of Big Data Computation to Salesforce
Bringing the Power of Big Data Computation to SalesforceSalesforce Developers Apache Hadoop and Spark are best-of-breed technologies for distributed processing and storage of very large data sets: Big Data. Join us as we explain how to integrate Salesforce with off-the-shelf big data tools to build flexible applications. You'll also learn how Force.com is evolving in this area and how Big Objects and Data Pipelines will provide Big Data capability within the platform.
Rivivi il Data in Motion Tour Milano 2024
Rivivi il Data in Motion Tour Milano 2024mtabrea • Creare i tuoi Data Product più velocemente con il nostro set avanzato di connettori e strumenti per la gestione dei flussi.
• Connettere i tuoi ambienti con pipeline di dati robuste e scalabili.
• Scoprire il ruolo chiave del Data Streaming nelle moderne architetture basate sull'intelligenza artificiale.
• Ascoltare le ultime innovazioni in fatto di Stream Governance con Apache Flink®.
Overview on Azure Machine Learning
Overview on Azure Machine LearningJames Serra Machine learning allows us to build predictive analytics solutions of tomorrow - these solutions allow us to better diagnose and treat patients, correctly recommend interesting books or movies, and even make the self-driving car a reality. Microsoft Azure Machine Learning (Azure ML) is a fully-managed Platform-as-a-Service (PaaS) for building these predictive analytics solutions. It is very easy to build solutions with it, helping to overcome the challenges most businesses have in deploying and using machine learning. In this presentation, we will take a look at how to create ML models with Azure ML Studio and deploy those models to production in minutes.
MDS ap_OEM Product Portfolio Intorduction to the DT & Analytics
MDS ap_OEM Product Portfolio Intorduction to the DT & AnalyticsMDS ap The document provides an overview of SAP's data management and analytics portfolio, including products such as SAP HANA, SAP SQL Anywhere, SAP ASE, SAP IQ, SAP Data Services, SAP Data Hub, SAP Information Steward, SAP PowerDesigner, SAP BusinessObjects BI suite, SAP Crystal Reports, SAP Web Intelligence, SAP Lumira, SAP Predictive Analytics, and SAP Enable Now. It describes the key capabilities and uses of each product.
More from DataWorks Summit/Hadoop Summit (20)
Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in ProductionDataWorks Summit/Hadoop Summit This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
State of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache ZeppelinDataWorks Summit/Hadoop Summit This document discusses Spark security and provides an overview of authentication, authorization, encryption, and auditing in Spark. It describes how Spark leverages Kerberos for authentication and uses services like Ranger and Sentry for authorization. It also outlines how communication channels in Spark are encrypted and some common issues to watch out for related to Spark security.
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache RangerDataWorks Summit/Hadoop Summit The document discusses the Virtual Data Connector project which aims to leverage Apache Atlas and Apache Ranger to provide unified metadata and access governance across data sources. Key points include:
- The project aims to address challenges of understanding, governing, and controlling access to distributed data through a centralized metadata catalog and policies.
- Apache Atlas provides a scalable metadata repository while Apache Ranger enables centralized access governance. The project will integrate these using a virtualization layer.
- Enhancements to Atlas and Ranger are proposed to better support the project's goals around a unified open metadata platform and metadata-driven governance.
- An initial minimum viable product will be built this year with the goal of an open, collaborative ecosystem around shared
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science PlatformDataWorks Summit/Hadoop Summit This document discusses using a data science platform to enable digital diagnostics in healthcare. It provides an overview of healthcare data sources and Yale/YNHH's data science platform. It then describes the data science journey process using a clinical laboratory use case as an example. The goal is to use big data and machine learning to improve diagnostic reproducibility, throughput, turnaround time, and accuracy for laboratory testing by developing a machine learning algorithm and real-time data processing pipeline.
Revolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and ZeppelinDataWorks Summit/Hadoop Summit This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSenseDataWorks Summit/Hadoop Summit This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.
Hadoop Crash Course
Hadoop Crash CourseDataWorks Summit/Hadoop Summit The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.
Data Science Crash Course
Data Science Crash CourseDataWorks Summit/Hadoop Summit This document provides an overview of data science and machine learning. It discusses what data science and machine learning are, including extracting insights from data and computers learning without being explicitly programmed. It also covers Apache Spark, which is an open source framework for large-scale data processing. Finally, it discusses common machine learning algorithms like regression, classification, clustering, and dimensionality reduction.
Apache Spark Crash Course
Apache Spark Crash CourseDataWorks Summit/Hadoop Summit This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataWorks Summit/Hadoop Summit This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
Schema Registry - Set you Data Free
Schema Registry - Set you Data FreeDataWorks Summit/Hadoop Summit Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats.
SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc.
In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time.
The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and MLDataWorks Summit/Hadoop Summit QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful.
At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.
How Hadoop Makes the Natixis Pack More Efficient 
How Hadoop Makes the Natixis Pack More Efficient DataWorks Summit/Hadoop Summit Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together.
This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear:
• How and why the business and IT requirements originated
• How we leverage the platform to fulfill security and production requirements
• How we organize a community to:
o Guard all the players, no one gets left on the ground!
o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead)
• What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community
We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match!
DETAILS
This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.
HBase in Practice 
HBase in Practice DataWorks Summit/Hadoop Summit HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases.
In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopDataWorks Summit/Hadoop Summit In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs.
Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.
Backup and Disaster Recovery in Hadoop 
Backup and Disaster Recovery in Hadoop DataWorks Summit/Hadoop Summit While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit
Recently uploaded (20)
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/
HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar.
Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten.
In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich
- Zugriff auf die Konsole
- Auffinden und Interpretieren von Protokolldateien
- Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS)
- Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien
- Nutzung der Client Clocking-Funktion
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts!
📕 Agenda
Welcome & Introductions
Orchestrator API Overview
Exploring the Swagger Interface
Test Manager API Highlights
Streamlining Automation & Testing with APIs (Demo)
Q&A and Open Discussion
Perfect for developers, testers, and automation enthusiasts!
👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/
This session streamed live on April 29, 2025, 18:00 CET.
Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB Want to learn practical tips for designing systems that can scale efficiently without compromising speed?
Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development.
As you explore key principles of designing low-latency systems with Rust, you will learn how to:
- Create and compile a real-world app with Rust
- Connect the application to ScyllaDB (NoSQL data store)
- Negotiate tradeoffs related to data modeling and querying
- Manage and monitor the database for consistently low latencies
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies In late April 2025, a significant portion of Europe, particularly Spain, Portugal, and parts of southern France, experienced widespread, rolling power outages that continue to affect millions of residents, businesses, and infrastructure systems.
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3 The latest updates on Manifest's pre-seed stage progress.
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how:
• Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules.
• Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance.
• Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity.
• Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications.
• Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market.
With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications.
Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market
For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john Analyze the growth of meme coins from mere online jokes to potential assets in the digital economy. Explore the community, culture, and utility as they elevate themselves to a new era in cryptocurrency.
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru Introduction to LPIC-1 Exam - overview, exam details, price and job opportunities
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell With expertise in data architecture, performance tracking, and revenue forecasting, Andrew Marnell plays a vital role in aligning business strategies with data insights. Andrew Marnell’s ability to lead cross-functional teams ensures businesses achieve sustainable growth and operational excellence.
How analogue intelligence complements AI
How analogue intelligence complements AIPaul Rowe
Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results.
News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsInData Labs In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy.
This infographic contains:
-AI and data privacy: Key findings
-Statistics on AI data privacy in the today’s world
-Tips on how to overcome data privacy challenges
-Benefits of AI data security investments.
Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.
Generative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma My talk for the Indian School of Business (ISB) Emerging Leaders Program Cohort 9. In this talk, I discussed key issues around adoption of GenAI in business - benefits, opportunities and limitations. I also discussed how my research on Theory of Cognitive Chasms helps address some of these issues
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBrian McKeiver May 2nd, 2025 talk at StirTrek 2025 Conference.
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1 Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark?
At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍
Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat
LEGO: Data Driven Growth Hacking Powered by Big Data
- 1. 1 Salesforce ConfidentialSalesforce Confidential LEGO: Data Driven Growth Hacking Powered by Big Data June 2016 Kamal Duggireddy Prashant Gokhale
- 2. 2 Salesforce Confidential Kamal Duggireddy Kamal Duggireddy currently leads Data Engineering, Product Data Science Team at Salesforce.com Prior to this, he served as Director - Big Data Architecture at American Express. Combining deep technical skills along with business knowledge and strong execution experience, Kamal developed reference architectures and new enterprise-level capabilities with the Hadoop stack. Prashant Gokhale Prashant is currently working on solving big data problems at Salesforce.com using Hadoop and its ecosystem components. Prior to this he held several critical engineering positions at Yahoo, Cloudera & Lookout. About Us
- 3. 3 Salesforce Confidential The Use Case | Overview Executives Analysts Product Managers
- 4. 4 Salesforce Confidential The Use Case | Flow Ad-Hoc Requests Predictive Data Apps Data Engineering & Curation Smart Data Dashboards (Salesforce Wave) Advanced Analysis Instrumentation 150+ Loglines Hadoop Data Processing Traditional Data Warehouses Dimensions
- 5. 5 Salesforce Confidential The Journey | How it all started
- 6. 6 Salesforce Confidential Milestones | Along the way </> <> Reusability Declarative Data Lake Data Dictionary Self service Automation Security Visualization Governance
- 7. 7 Salesforce Confidential The Framework | Finally! Datasets (Variousgrain) Data Lake Log Processing Metadata Flow Engine WebApp Self Service LogSourcesCloudMetrics Data Profiler Data Science Kafka Splunk Files Warehouse Objects Hadoop Cubes (Customgrain)
- 8. 8 Salesforce Confidential Goals Scalable Process hundreds of billions of log lines. Flexible Handle thousands of log schemas. Support variable grain and transformations using custom code. Data Quality Automated data profiling, monitoring and alerting. Self Service Enable ad-hoc analysis
- 9. 9 Salesforce Confidential Log Processing Engine •Declaratively define features and flows. •Normalize data across multiple log lines. •Custom code injection for data transformation. Data Profiler •Profile data at scale to detect anomalies. Web App •Interface to manage features and flows. Job Automation engine •End to end automation from features/flows to curated data sets in Wave. Key Building Blocks
- 10. 10 Salesforce Confidential Log Processing Engine logType==’X’ and event==’Create Event’ and page==’Home Landing’,”Feat 1”,”eval_code(event.toUpperCase())”,page,….. logType==’ABC’ and event==’Create Event’ and page==’Home’,”Feat 2”,”eval_code(event.substring(5))”,event,….. usage Log Files Feature definitions Hive tables Data Normalization Data Cleansing Data Transformation +
- 11. 11 Salesforce Confidential Data Profiler Dataset Field Type, Total, Min, Max, Avg, # Nulls, # Distinct, Median, 99th %tile, Top N lego_feat browser STR 2.3B 7 63 25 1M 50 34 38 [.....] lego_feat url STR 2.3B 20 223 50 0 5M 70 90 [.....] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Datasets across platform HCatalog MapReduce Datasets Dataset Profile An Example Monitoring & alerting
- 12. 12 Salesforce Confidential Everything put together Datasets (Variousgrain) Data Lake Log Processing Metadata Flow Engine WebApp Self Service LogSourcesCloudMetrics Data Profiler Data Science Kafka Splunk Files Warehouse Objects Hadoop Cubes (Customgrain)
- 13. 13 Salesforce Confidential Data Volumetrics TOTAL Avg. Volume of App Logs processed (Compressed) 100’s TB/mon Avg. Number of Jobs 6000+ /mon Avg. Log Size volume growth rate A lot! Number of Log Record Types 1,000s Number of fields 10s of 1,000s 200+ B Events / Day 500+ Features
- 14. 14 Salesforce Confidential thank y u 14 We are hiring!! www.salesforce.com/comapany/careers