Risk Management Framework Using Intel FPGA, Apache Spark, and Persistent RDDs with Reza Sadri and Mutema Pittman
1 like746 views
Performing analytics for risk management purposes is applied in many fields, especially in financial services. We present a framework for accelerated risk analytics and show a large-scale financial sector application where this framework is used to run backtesting algorithms on risk-based securities such as options. These applications require highly computationally-intensive operations on extremely large data sets with objects numbering in the tens of billions. Intel FPGA and FinLib library for financial applications are used to offload the computation; however, another challenging problem (that we have resolved) is how to feed the data to the FPGA at the optimal speed without having to do customized coding. A combination of Apache Spark along with Levyx’s persistent dataframes are used to address this problem. These dataframes allow absorbing the computation from Spark and offloading it to Finlib in an automated way. This example can be expanded to many other areas of Risk Management such as Insurance and Cybersecurity.
1 of 12
Downloaded 38 times
Ad
Recommended
Building Real-Time Data Pipeline for Diabetes Medication Recommender System U...
Building Real-Time Data Pipeline for Diabetes Medication Recommender System U...Databricks American Diabetic association states that 29.1 million Americans and 300+ million people all over the world have diabetes. Diabetic medication management is always challenging. Based on doctor’s prescription, patients take insulin dosage one hour before breakfast, lunch or dinner. But the real world scenario insulin intake can be changed based on the blood glucose level, calorie intake on a specific day, etc.
This talk explains how a real-time Big Data pipeline recommendation engine can be used to suggest insulin intake for diabetic patients in near real time. Based on calorie intake and blood glucose level from patients as well as generated dataset, insulin dosage can be recommended which will help patients to avoid over/under dosage. Designing medication recommender system is a need for the Healthcare industry. There is a growing trend for the applications to help doctors by recommending medication based on patient’s historic data. This also helps facilitate a doctor friendly and hospital free atmosphere for all users all over the world.
This talk would delve into a Diabetes medication recommender system using Databricks and Spark. Databricks supports HIPAA compliant deployment for processing PHI data. This talk would cover building a secure pipeline with encrypted data and the end-to-end recommendation system using Structured streaming and IoT data flowing from sensor.
Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Das...
Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Das...Spark Summit 1) Financial institutions must report asset trades within a short regulatory window based on changing rules.
2) A leading financial institution executes around 6 million trades daily and must report qualifying trades in a specified format within 15 minutes.
3) The institution transforms, enriches, de-duplicates trades and applies business rules to identify reportable trades for generation within 35 seconds for 30,000 records using Spark on a 10 node cluster.
Deploying Enterprise Scale Deep Learning in Actuarial Modeling at Nationwide
Deploying Enterprise Scale Deep Learning in Actuarial Modeling at NationwideDatabricks The traditional approach to insurance pricing involves fitting a generalized linear model (GLM) to data collected on historical claims payments and premiums received. The explosive growth in data availability and increasing competitiveness in the marketplace are challenging actuaries to find new insights in their data and make predictions with more granularity, improved speed and efficiency, and with tighter integration among business units to support strategic decisions.
In this session we will share our experience implementing deep hierarchical neural networks using TensorFlow and PySpark on Databricks. We will discuss the benefits of the ML Runtime, our experience using the goofys mount, our process for hyperparameter tuning, specific considerations for the large dataset size and extreme volatility present in insurance data, among other topics.
Authors: Bryn Clark, Krish Rajaram
Building Robust Production Data Pipelines with Databricks Delta
Building Robust Production Data Pipelines with Databricks DeltaDatabricks "Most data practitioners grapple with data quality issues and data pipeline complexities—it's the bane of their existence. Data engineers, in particular, strive to design and deploy robust data pipelines that serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets.
Databricks Delta, part of Databricks Runtime, is a next-generation unified analytics engine built on top of Apache Spark. Built on open standards, Delta employs co-designed compute and storage and is compatible with Spark API’s. It powers high data reliability and query performance to support big data use cases, from batch and streaming ingests, fast interactive queries to machine learning. In this tutorial we will discuss the requirements of modern data pipelines, the challenges data engineers face when it comes to data reliability and performance and how Delta can help. Through presentation, code examples and notebooks, we will explain pipeline challenges and the use of Delta to address them. You will walk away with an understanding of how you can apply this innovation to your data architecture and the benefits you can gain.
This tutorial will be both instructor-led and hands-on interactive session. Instructions in how to get tutorial materials will be covered in class. WHAT
YOU’LL LEARN:
– Understand the key data reliability and performance data pipelines challenges
– How Databricks Delta helps build robust pipelines at scale
– Understand how Delta fits within an Apache Spark™ environment – How to use Delta to realize data reliability improvements
– How to deliver performance gains using Delta
PREREQUISITES:
– A fully-charged laptop (8-16GB memory) with Chrome or Firefox
– Pre-register for Databricks Community Edition"
Speakers: Steven Yu, Burak Yavuz
Log I am your father
Log I am your fatherDataWorks Summit/Hadoop Summit The document discusses how machine data from various sources such as IoT devices, industrial systems, mobile devices, and other systems can be collected and analyzed using Splunk software. Splunk provides capabilities for data ingestion, indexing, searching, analyzing, and visualizing large amounts of machine data. It also discusses how Splunk has been used by companies in various industries to gain insights from their machine data to improve operations, security, customer experience, and business outcomes. Specific use cases highlighted include predictive maintenance, anomaly detection, supply chain optimization, and understanding customer behavior.
Spark Summit Europe 2016 Keynote - Databricks CEO 
Spark Summit Europe 2016 Keynote - Databricks CEO Databricks This document discusses democratizing AI using Apache Spark on Databricks. It first discusses how AI is changing the world through advances like AlphaGo but that AI hasn't been fully democratized. It then discusses how Databricks uses Apache Spark to close gaps in managing big data infrastructure, establishing production-ready applications, and empowering teams. Specifically, Databricks provides an integrated workspace, just-in-time data platform, and automated Spark management to accelerate developing and deploying AI applications. The document concludes by discussing how Databricks enables faster and easier deep learning through features like TensorFlow, TensorFrames, GPU support, and a full stack for data ingestion, model training, and productionization.
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks A complex real-time data workflow implementation is very challenging. This session will describe the architecture of a data platform that provides a single, secure, high-performance system that can be deployed in a hybrid cloud architectures. We will present how to support simultaneous, consistent and high-performance access through multiple industry open source and cloud compatible standards of streaming, table, TSDB, object, and file APIs. A new serverless technology is also used in the architecture to support a dynamic and flexible implementations. The presenter will also outline how the platform was integrated with the Spark eco-system, including AI and ML tools, to simplify the development process
The rise of big data governance: insight on this emerging trend from active o...
The rise of big data governance: insight on this emerging trend from active o...DataWorks Summit Each of today’s most forward-thinking enterprises have been forced to face similar data challenges: the reliance on real-time data to better serve their customers and, subsequently, the requirement of complying with regulations to protect that data – one example being the General Data Protection Regulation (GDPR).
The solution to this emerging challenge is a tricky one – for companies like ING, this data governance challenge has been met with metadata, a consistent view across a large heterogeneous ecosystem and collaboration with an active open source community.
This joint presentation, John Mertic – Director of ODPi – and Ferd Scheepers – Global Chief Information Architect of ING – will address the benefits of a vendor-neutral approach to data governance, the need for an open metadata standard, along with insight around how companies ING, IBM, Hortonworks and more are delivering solutions to this challenge as an open source initiative.
Audience Takeaways include:
Understand the role of metadata;
Understand the need for a cross technology view on metadata;
Understand the role of Apache Atlas as a reference implementation; and
Understand the role of ODPi in offering value-added services including certification.
Speaker
John Mertic, Director of Program Management for ODPi, R Consortium, and Open Mainframe Project, The Linux Foundation
LinkedIn2
LinkedIn2DataWorks Summit/Hadoop Summit The document summarizes LinkedIn's experience scaling their Hadoop infrastructure from 1 cluster with 20 nodes and 10 users in 2008 to over 10 clusters with over 10,000 nodes and 1,000 users now. It discusses the challenges of scaling hardware and human infrastructure. It then introduces Dali, LinkedIn's system for managing metadata and data access that aims to make analytics infrastructure invisible through concepts like datasets, views, and lineage tracking. Key aspects of Dali include separating logical and physical concerns, versioning APIs and views, and expressing producer/consumer contracts as constraints.
Spark in the Hadoop Ecosystem-(Mike Olson, Cloudera)
Spark in the Hadoop Ecosystem-(Mike Olson, Cloudera)Spark Summit Spark fits into the Hadoop ecosystem alongside other frameworks like MapReduce, Hive, and Pig. It provides faster processing capabilities than MapReduce for interactive queries and stream processing. Spark also benefits from sharing components with other frameworks in Hadoop, including security, data governance, and operations.
Migrating legacy ERP data into Hadoop
Migrating legacy ERP data into HadoopDataWorks Summit Legacy ERP architecture offers an incredibly, efficient means of operational resource management, but a real challenge comes from extracting business insights from them. Over the past 30 years, ERP data system, such as SAP, can be hard to interact with especially at the source database level. Whether initial translation of business logic and hierarchies create significant customizations, as well as, merging those changes into analytical applications. Overall, the entire process of designing self-service reporting with business level context can be quite cumbersome, looking at an example platform like, SAP, which contain pre-packaged modules (MM, SD, PP, etc), integrating these systems into a series of pre-built analytics.
The orchestration and integration over a wide range of open source technology solutions with some commercial CDC and reporting solutions into a reference solution that mimics several real customer scenarios today, living on relational platforms. Key considerations of extracting from the operational system of record, especially the merging of multiple systems in different time zones, will be addressed. Furthermore, the integration concerns of an analytics Hadoop platform, using HIVE Acid and Merge, as well as, flattening techniques for dimensional models. Many times a customer is temporarily limited in the range of data their ERP can contain, and older data is often offloaded to secondary systems or cold archiving entities. That goes away, but the opportunities expanded with real-time reporting across all of history, and the expanded use cases with advanced machine learnings methods.
Speakers
Jordan Martz, Director of Tech Solutions, Attunity
David Freriks, Technology Evangelist, Qlik
The key to unlocking the Value in the IoT? Managing the Data!
The key to unlocking the Value in the IoT? Managing the Data!DataWorks Summit/Hadoop Summit The document discusses how managing data is key to unlocking value from the Internet of Things. It emphasizes that variety, not size, is most important with big data. Example use cases mentioned include predictive maintenance, search and root cause analysis. The technology landscape is changing with new architectures like data lakes and new patterns such as event histories and timelines. Managing data is also changing with schema on read, loosely coupled schemas, and increased importance of metadata. The document concludes that data management patterns and practices are foundational to effective analytics with IoT data.
Real-Time Robot Predictive Maintenance in Action
Real-Time Robot Predictive Maintenance in ActionDataWorks Summit This document describes a predictive maintenance system for robots using real-time sensor data. A team of 4 engineers built a solution in 2 months using standard open source software like H2O and MapR. Sensors on a robot collected accelerometer, gyroscope and other data. This raw data was analyzed using anomaly detection algorithms in H2O to build a machine learning model that identified normal vs abnormal robot states. The model was deployed as a microservice to make real-time predictions on new sensor data and detect potential failures. The solution was able to analyze data from hundreds of robots and identify anomalies within 3 seconds, demonstrating an effective low-cost predictive maintenance system.
Druid Overview by Rachel Pedreschi
Druid Overview by Rachel PedreschiBrian Olsen This document provides an introduction to Apache Druid, describing it as a real-time OLAP database for data-driven applications. It outlines the evolution from first generation on-premises data warehouses to modern data lakes and data rivers. Druid is presented as a high performance analytics database designed for event-driven data at large scales with low latency. Example use cases include digital advertising, user analytics, and IoT. The document encourages readers to learn more and get involved with the Druid open source community.
Webinar: The Modern Streaming Data Stack with Kinetica & StreamSets
Webinar: The Modern Streaming Data Stack with Kinetica & StreamSetsKinetica Enterprises are now faced with wrangling massive volumes of complex, streaming data from a variety of different sources, a new paradigm known as extreme data. However, the traditional data integration model that’s based on structured batch data and stable data movement patterns makes it difficult to analyze extreme data in real-time. Join Matt Hawkins, Principal Solutions Architect at Kinetica and Mark Brooks, Solution Engineer at StreamSets as they share how innovative organizations are modernizing their data stacks with StreamSets and Kinetica to enable faster data movement and analysis.In this webinar we’ll explore:
The modern data architecture required for dealing with extreme data
How StreamSets enables continuous data movement and transformation across the enterprise
How Kinetica harnesses the power of GPUs to accelerate analytics on streaming data
A live demo of StreamSets and Kinetica connector to enable high speed data ingestion, queries and data visualization
Shaping a Digital Vision
Shaping a Digital VisionDataWorks Summit/Hadoop Summit Undertaking a digital journey starts with clearly articulating the success factors for the entire digital journey, and our experience from the field has shown it to be an Achilles heel for most CXOs, across Fortune 500 organizations. Our findings were corroborated when a Mckinsey study reported that only 15% of the organizations are able to calculate the ROI of a digital initiative.
In this talk we will deliberate on demonstrated examples from multi-billion dollar businesses around proven methodologies to measure the value of a digital enterprise. The panel will share experiences as well as provide actionable advice for immediate next steps around the following:
Successful metrics for measuring the value for Digital / IoT / AI/ Machine learning engagements
How can 'Digital Traction Metrics' help with actionable insights even before the Financial Metrics have been reported
What are the best in-class organizational constructs and futuristic employee engagement methods to facilitate the digital revolution
Panelists for this session include:
• Christian Bilien - Head of Global Data at Societe Generale
• Pierre Alexandre Pautrat – Head of Big Data at BPCE/Nattixis
• Ronny Fehling – VP , Airbus
• Juergen Urbanski – Silicon Valley Data Science
• Abhas Ricky - EMEA Lead, Innovation & Strategy, Hortonworks
The Life of an Internet of Things Electron
The Life of an Internet of Things ElectronDataWorks Summit/Hadoop Summit The document discusses an "Infinity architecture" that leverages tagging, data processing, storage, and querying to collect data from various digital properties and IoT devices, identify and enrich visitor/object sessions in real-time, and provide a unified platform to access and analyze granular, unsampled visitor data across a globally distributed network. The architecture includes components for tagging, event processing, data storage, querying, and integrations with other systems like CRM, DMPs, and DSPs.
The Power of Data
The Power of DataDataWorks Summit This presentation starts off by discussing powerful examples of The Power of Data and the benefits of Data Driven architectures. A Data Governance program is important for the success of Data Driven architectures. We then discuss the challenges of implementing a Data Governance framework on a Big Data Data Lake with open source software including DataPlane, Apache Atlas and Apache Ranger. And finally, we discuss the importance of the democratization of data and the switching to a speed of thought framework with Hive LLAP.
Graph representation learning to prevent payment collusion fraud
Graph representation learning to prevent payment collusion fraudDataWorks Summit PayPal is at the forefront of applying large-scale graph processing and machine learning algorithms to keep fraudsters at bay. While deep learning algorithms have found enormous success in the application to unstructured data, their application to structured data, such as graphs, is in the nascent stage of research. One of the emerging research area is graph-based representation learning where features are learned automatically from network structure.
In this talk, I'll present how this emerging area can help to automatically learn features from a large scale payment network. Learned features are used to build machine learning models that are used to prevent complex fraudster activities where they collude with several other actors to commit fraud. I'll elaborate on specific challenges in applying this state-of-the-art machine learning algorithm to large scale payment network. Results from experiments conducted on a very large graph data set containing billions of edges and vertices will be presented.
Speaker
Venkatesh Ramanathan, Data Scientist, Paypal
Security, ETL, BI & Analytics, and Software Integration
Security, ETL, BI & Analytics, and Software IntegrationDataWorks Summit This document discusses how Liberty Mutual Insurance is using a Hadoop data lake to power analytics. It provides examples of how the data lake is used to integrate data from various sources and support various use cases. Specifically, it discusses how the data lake enables:
- Storage of structured and unstructured data from multiple sources in a centralized and secure location
- Analytics and machine learning by data scientists and analysts accessing the stored data
- Integrations with tools like Elasticsearch, Spark, and PowerBI for querying, analyzing and visualizing the data
- Archiving of log and sensor data from systems into hot, warm and cold storage tiers based on age and access frequency
Adding structure to your streaming pipelines: moving from Spark streaming to ...
Adding structure to your streaming pipelines: moving from Spark streaming to ...DataWorks Summit How do you go from a strictly typed object-based streaming pipeline with simple operations to a structured streaming pipeline with higher order complex relational operations? This is what the Data Engineering team did at GoPro to scale up the development of streaming pipelines for the rapidly growing number of devices and applications.
When big data frameworks such as Hadoop first came to exist, developers were happy because we could finally process large amounts of data without writing complex multi-threaded code or worse yet writing complicated distributed code. Unfortunately, only very simple operations were available such as map and reduce. Almost immediately, higher level operations were desired similar to relational operations. And so Hive and dozens (hundreds?) of SQL-based big data tools became available for more developer-efficient batch processing of massive amounts of data.
In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world, so that nearly every streaming framework now supports higher level relational operations.
In this talk, we will discuss in a very hands-on manner how the streaming data pipelines for GoPro devices and apps have moved from the original Spark streaming with its simple RDD-based operations in Spark 1.x to Spark's structured streaming with its higher level relational operations in Spark 2.x. We will talk about the differences, advantages, and necessary pain points that must be addressed in order to scale relational-based streaming pipelines for massive IoT streams. We will also talk about moving from “hand built” Hadoop/Spark clusters running in the cloud to using a Spark-based cloud service. DAVID WINTERS, Big Data Architect, GoPro and HAO ZOU, Senior Software Engineer, GoPro
Leveraging Spark to Democratize Data for Omni-Commerce with Shafaq Abdullah
Leveraging Spark to Democratize Data for Omni-Commerce with Shafaq AbdullahDatabricks Insnap, a hyper-personalized ML-based platform acquired by The Honest Company, has been used to build a real-time data platform based on Apache Spark, Cassandra and Redshift. Users’ behavioral and transactional data have been used to build data models and ML models, and to drive use cases for marketing, growth, finance and operations.
Learn how Honest Company has used Spark as a workhorse for 1) collecting, ETL and storing data from various sources including mysql, mongo, jde, Google analytics, Facebook, Localytics and REST API; 2) building data models and aggregating and generating reports of revenue, order fulfillment tracking, data pipeline monitoring and subscriptions; 3) Using ML to build model for user acquisitions, LTV and recommendations use cases. Spark replaced the monolithic codebase with flexible, scalable and robust pipelines. Databricks helped The Honest Company to focus on data instead of maintaining infrastructure. While Honest users got delightful recommendations to improve experience, data users at Honest understood users much better in terms of segmenting with behavioral information and advanced ML models, leading to increased revenue and retention.
Spark Summit EU talk by Pat Patterson
Spark Summit EU talk by Pat PattersonSpark Summit This document discusses building data pipelines with Spark and StreamSets. It describes how StreamSets Data Collector can be used to build pipelines that run on Spark today by leveraging Kafka RDDs and containers on Spark. It also outlines future directions for deeper Spark integration, including running pipelines on Databricks and developing a standalone Spark processor. The document concludes with a demo of StreamSets Data Collector capabilities.
A Stock Prediction System using Open-Source Software
A Stock Prediction System using Open-Source SoftwareFred Melo Presentation delivered at Apache Con 2015 conference. Talks about the architecture and implementation of a real-time prediction system based on live streaming for stock quotes using Spring XD, .
A demo was also featured.
Building Sessionization Pipeline at Scale with Databricks Delta
Building Sessionization Pipeline at Scale with Databricks DeltaDatabricks "Comcast has made a concerted effort to transform itself from a cable/ISP company to a technology company. Data-driven decision making is at the heart of this transformation, and we use data to understand how customers interact with our products, and we see data as the most truthful representation of the voice of our customer. My team, Product Analytics & behavior science (PABS) team plays the role as interpreter, transforming data into consumable insights.
The X1 entertainment operating system, is one of the largest video streaming platforms in the world, and our customers consume more than a billion hours of content a week on X1. Our team consumes X1 telemetry at a rate of more than 25TBs of data per day and uses this data to inform our product teams members about the performance of and engagement with the platform. We also use this data to research customer behaviors to help better inform our product team members about areas of opportunity in our products, which range from fixing bugs to creating new features.
To power these insights, we need to have a reliable real-time data pipelines to deliver these insights, and we need our data scientists and data engineers to be able to quickly and efficiently be able to develop and commit new code to ensure we can measure new features the product teams are developing. To do this in an environment at this scale, we have been using Databricks, and Databricks delta to gain operational efficiencies, optimization and cost savings.
Some of the features from delta that we took advantage of to achieve the desired levels of efficiencies, optimization and cost savings are:
· Distributed writes to s3 (essentially eliminating 500 errors)
· s3 log with fast reads and ACID transactions (massive increases in s3 scans/reads, and enabling consistent views of the bucket/table)
· Vacuum
· Pptimize (which has allowed us to reduce a 640 node job to 40, and massively increase efficiencies of our clusters as well as our DS/DE’s)"
Data Driven Decisions at Scale
Data Driven Decisions at ScaleDatabricks Comcast is the largest cable and internet provider in the US, reaching more than 30 million customers, and continues to grow its presence in the EU with the acquisition of Sky. Over the last couple years, Comcast has shifted focus to the customer experience.
Building an Enterprise Data Platform with Azure Databricks to Enable Machine ...
Building an Enterprise Data Platform with Azure Databricks to Enable Machine ...Databricks This document summarizes Walmart's transition to building an enterprise data platform on Azure Databricks to enable machine learning and data science at scale. Previously, Walmart had a complex and slow legacy technology stack. The new platform goals were to centralize data in the cloud, increase productivity with data science tools, and reduce costs. Key aspects of the new platform included using Azure and Databricks for data processing and machine learning, Airflow for orchestration, and building several machine learning models for applications like fraud detection and product recommendations. Challenges in the transition included optimizing performance and managing resources across the platforms.
AI on Spark for Malware Analysis and Anomalous Threat Detection
AI on Spark for Malware Analysis and Anomalous Threat DetectionDatabricks At Avast, we believe everyone has the right to be safe. We are dedicated to creating a world that provides safety and privacy for all, not matter where you are, who you are, or how you connect. With over 1.5 billion attacks stopped and 30 million new executable files monthly, big data pipelines are crucial for the security of our customers. At Avast we are leveraging Apache Spark machine learning libraries and TensorflowOnSpark for a variety of tasks ranging from marketing and advertisement, through network security to malware detection. This talk will cover our main cybersecurity usecases of Spark. After describing our cluster environment we will first demonstrate anomaly detection on time series of threats. Having thousands of types of attacks and malware, AI helps human analysts select and focus on most urgent or dire threats. We will walk through our setup for distributed training of deep neural networks with Tensorflow to deploying and monitoring of a streaming anomaly detection application with trained model. Next we will show how we use Spark for analysis and clustering of malicious files and large scale experimentation to automatically process and handle changes in malware. In the end, we will give comparison to other tools we used for solving those problems.
Learn How Financial Services Organizations Can Use Big Data to Mitigate Risks
Learn How Financial Services Organizations Can Use Big Data to Mitigate RisksMapR Technologies Risk comes in a variety of forms including uncertainty in financial markets, legal liabilities, operational risk, fraud, and protection against external and internal attacks. Models are becoming increasingly granular and improving risk modeling is a high priority.
Review this presentation from Splunk and MapR to learn how you can study months’ or years’ worth of raw data from disparate sources, without sampling, to understand and reduce risk.
The new dominant companies are running on data 
The new dominant companies are running on data SnapLogic The cost of Digital Transformation is dropping rapidly. The technologies and methodologies are evolving to open up new opportunities for new and established corporations to drive business. We will examine specific examples of how and why a combination of robust infrastructure, cloud first and machine learning can take your company to the next level of value and efficiency.
Rich Dill, SnapLogic's enterprise solutions architect, at Big Data LDN 2017.
Ad
More Related Content
What's hot (20)
LinkedIn2
LinkedIn2DataWorks Summit/Hadoop Summit The document summarizes LinkedIn's experience scaling their Hadoop infrastructure from 1 cluster with 20 nodes and 10 users in 2008 to over 10 clusters with over 10,000 nodes and 1,000 users now. It discusses the challenges of scaling hardware and human infrastructure. It then introduces Dali, LinkedIn's system for managing metadata and data access that aims to make analytics infrastructure invisible through concepts like datasets, views, and lineage tracking. Key aspects of Dali include separating logical and physical concerns, versioning APIs and views, and expressing producer/consumer contracts as constraints.
Spark in the Hadoop Ecosystem-(Mike Olson, Cloudera)
Spark in the Hadoop Ecosystem-(Mike Olson, Cloudera)Spark Summit Spark fits into the Hadoop ecosystem alongside other frameworks like MapReduce, Hive, and Pig. It provides faster processing capabilities than MapReduce for interactive queries and stream processing. Spark also benefits from sharing components with other frameworks in Hadoop, including security, data governance, and operations.
Migrating legacy ERP data into Hadoop
Migrating legacy ERP data into HadoopDataWorks Summit Legacy ERP architecture offers an incredibly, efficient means of operational resource management, but a real challenge comes from extracting business insights from them. Over the past 30 years, ERP data system, such as SAP, can be hard to interact with especially at the source database level. Whether initial translation of business logic and hierarchies create significant customizations, as well as, merging those changes into analytical applications. Overall, the entire process of designing self-service reporting with business level context can be quite cumbersome, looking at an example platform like, SAP, which contain pre-packaged modules (MM, SD, PP, etc), integrating these systems into a series of pre-built analytics.
The orchestration and integration over a wide range of open source technology solutions with some commercial CDC and reporting solutions into a reference solution that mimics several real customer scenarios today, living on relational platforms. Key considerations of extracting from the operational system of record, especially the merging of multiple systems in different time zones, will be addressed. Furthermore, the integration concerns of an analytics Hadoop platform, using HIVE Acid and Merge, as well as, flattening techniques for dimensional models. Many times a customer is temporarily limited in the range of data their ERP can contain, and older data is often offloaded to secondary systems or cold archiving entities. That goes away, but the opportunities expanded with real-time reporting across all of history, and the expanded use cases with advanced machine learnings methods.
Speakers
Jordan Martz, Director of Tech Solutions, Attunity
David Freriks, Technology Evangelist, Qlik
The key to unlocking the Value in the IoT? Managing the Data!
The key to unlocking the Value in the IoT? Managing the Data!DataWorks Summit/Hadoop Summit The document discusses how managing data is key to unlocking value from the Internet of Things. It emphasizes that variety, not size, is most important with big data. Example use cases mentioned include predictive maintenance, search and root cause analysis. The technology landscape is changing with new architectures like data lakes and new patterns such as event histories and timelines. Managing data is also changing with schema on read, loosely coupled schemas, and increased importance of metadata. The document concludes that data management patterns and practices are foundational to effective analytics with IoT data.
Real-Time Robot Predictive Maintenance in Action
Real-Time Robot Predictive Maintenance in ActionDataWorks Summit This document describes a predictive maintenance system for robots using real-time sensor data. A team of 4 engineers built a solution in 2 months using standard open source software like H2O and MapR. Sensors on a robot collected accelerometer, gyroscope and other data. This raw data was analyzed using anomaly detection algorithms in H2O to build a machine learning model that identified normal vs abnormal robot states. The model was deployed as a microservice to make real-time predictions on new sensor data and detect potential failures. The solution was able to analyze data from hundreds of robots and identify anomalies within 3 seconds, demonstrating an effective low-cost predictive maintenance system.
Druid Overview by Rachel Pedreschi
Druid Overview by Rachel PedreschiBrian Olsen This document provides an introduction to Apache Druid, describing it as a real-time OLAP database for data-driven applications. It outlines the evolution from first generation on-premises data warehouses to modern data lakes and data rivers. Druid is presented as a high performance analytics database designed for event-driven data at large scales with low latency. Example use cases include digital advertising, user analytics, and IoT. The document encourages readers to learn more and get involved with the Druid open source community.
Webinar: The Modern Streaming Data Stack with Kinetica & StreamSets
Webinar: The Modern Streaming Data Stack with Kinetica & StreamSetsKinetica Enterprises are now faced with wrangling massive volumes of complex, streaming data from a variety of different sources, a new paradigm known as extreme data. However, the traditional data integration model that’s based on structured batch data and stable data movement patterns makes it difficult to analyze extreme data in real-time. Join Matt Hawkins, Principal Solutions Architect at Kinetica and Mark Brooks, Solution Engineer at StreamSets as they share how innovative organizations are modernizing their data stacks with StreamSets and Kinetica to enable faster data movement and analysis.In this webinar we’ll explore:
The modern data architecture required for dealing with extreme data
How StreamSets enables continuous data movement and transformation across the enterprise
How Kinetica harnesses the power of GPUs to accelerate analytics on streaming data
A live demo of StreamSets and Kinetica connector to enable high speed data ingestion, queries and data visualization
Shaping a Digital Vision
Shaping a Digital VisionDataWorks Summit/Hadoop Summit Undertaking a digital journey starts with clearly articulating the success factors for the entire digital journey, and our experience from the field has shown it to be an Achilles heel for most CXOs, across Fortune 500 organizations. Our findings were corroborated when a Mckinsey study reported that only 15% of the organizations are able to calculate the ROI of a digital initiative.
In this talk we will deliberate on demonstrated examples from multi-billion dollar businesses around proven methodologies to measure the value of a digital enterprise. The panel will share experiences as well as provide actionable advice for immediate next steps around the following:
Successful metrics for measuring the value for Digital / IoT / AI/ Machine learning engagements
How can 'Digital Traction Metrics' help with actionable insights even before the Financial Metrics have been reported
What are the best in-class organizational constructs and futuristic employee engagement methods to facilitate the digital revolution
Panelists for this session include:
• Christian Bilien - Head of Global Data at Societe Generale
• Pierre Alexandre Pautrat – Head of Big Data at BPCE/Nattixis
• Ronny Fehling – VP , Airbus
• Juergen Urbanski – Silicon Valley Data Science
• Abhas Ricky - EMEA Lead, Innovation & Strategy, Hortonworks
The Life of an Internet of Things Electron
The Life of an Internet of Things ElectronDataWorks Summit/Hadoop Summit The document discusses an "Infinity architecture" that leverages tagging, data processing, storage, and querying to collect data from various digital properties and IoT devices, identify and enrich visitor/object sessions in real-time, and provide a unified platform to access and analyze granular, unsampled visitor data across a globally distributed network. The architecture includes components for tagging, event processing, data storage, querying, and integrations with other systems like CRM, DMPs, and DSPs.
The Power of Data
The Power of DataDataWorks Summit This presentation starts off by discussing powerful examples of The Power of Data and the benefits of Data Driven architectures. A Data Governance program is important for the success of Data Driven architectures. We then discuss the challenges of implementing a Data Governance framework on a Big Data Data Lake with open source software including DataPlane, Apache Atlas and Apache Ranger. And finally, we discuss the importance of the democratization of data and the switching to a speed of thought framework with Hive LLAP.
Graph representation learning to prevent payment collusion fraud
Graph representation learning to prevent payment collusion fraudDataWorks Summit PayPal is at the forefront of applying large-scale graph processing and machine learning algorithms to keep fraudsters at bay. While deep learning algorithms have found enormous success in the application to unstructured data, their application to structured data, such as graphs, is in the nascent stage of research. One of the emerging research area is graph-based representation learning where features are learned automatically from network structure.
In this talk, I'll present how this emerging area can help to automatically learn features from a large scale payment network. Learned features are used to build machine learning models that are used to prevent complex fraudster activities where they collude with several other actors to commit fraud. I'll elaborate on specific challenges in applying this state-of-the-art machine learning algorithm to large scale payment network. Results from experiments conducted on a very large graph data set containing billions of edges and vertices will be presented.
Speaker
Venkatesh Ramanathan, Data Scientist, Paypal
Security, ETL, BI & Analytics, and Software Integration
Security, ETL, BI & Analytics, and Software IntegrationDataWorks Summit This document discusses how Liberty Mutual Insurance is using a Hadoop data lake to power analytics. It provides examples of how the data lake is used to integrate data from various sources and support various use cases. Specifically, it discusses how the data lake enables:
- Storage of structured and unstructured data from multiple sources in a centralized and secure location
- Analytics and machine learning by data scientists and analysts accessing the stored data
- Integrations with tools like Elasticsearch, Spark, and PowerBI for querying, analyzing and visualizing the data
- Archiving of log and sensor data from systems into hot, warm and cold storage tiers based on age and access frequency
Adding structure to your streaming pipelines: moving from Spark streaming to ...
Adding structure to your streaming pipelines: moving from Spark streaming to ...DataWorks Summit How do you go from a strictly typed object-based streaming pipeline with simple operations to a structured streaming pipeline with higher order complex relational operations? This is what the Data Engineering team did at GoPro to scale up the development of streaming pipelines for the rapidly growing number of devices and applications.
When big data frameworks such as Hadoop first came to exist, developers were happy because we could finally process large amounts of data without writing complex multi-threaded code or worse yet writing complicated distributed code. Unfortunately, only very simple operations were available such as map and reduce. Almost immediately, higher level operations were desired similar to relational operations. And so Hive and dozens (hundreds?) of SQL-based big data tools became available for more developer-efficient batch processing of massive amounts of data.
In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world, so that nearly every streaming framework now supports higher level relational operations.
In this talk, we will discuss in a very hands-on manner how the streaming data pipelines for GoPro devices and apps have moved from the original Spark streaming with its simple RDD-based operations in Spark 1.x to Spark's structured streaming with its higher level relational operations in Spark 2.x. We will talk about the differences, advantages, and necessary pain points that must be addressed in order to scale relational-based streaming pipelines for massive IoT streams. We will also talk about moving from “hand built” Hadoop/Spark clusters running in the cloud to using a Spark-based cloud service. DAVID WINTERS, Big Data Architect, GoPro and HAO ZOU, Senior Software Engineer, GoPro
Leveraging Spark to Democratize Data for Omni-Commerce with Shafaq Abdullah
Leveraging Spark to Democratize Data for Omni-Commerce with Shafaq AbdullahDatabricks Insnap, a hyper-personalized ML-based platform acquired by The Honest Company, has been used to build a real-time data platform based on Apache Spark, Cassandra and Redshift. Users’ behavioral and transactional data have been used to build data models and ML models, and to drive use cases for marketing, growth, finance and operations.
Learn how Honest Company has used Spark as a workhorse for 1) collecting, ETL and storing data from various sources including mysql, mongo, jde, Google analytics, Facebook, Localytics and REST API; 2) building data models and aggregating and generating reports of revenue, order fulfillment tracking, data pipeline monitoring and subscriptions; 3) Using ML to build model for user acquisitions, LTV and recommendations use cases. Spark replaced the monolithic codebase with flexible, scalable and robust pipelines. Databricks helped The Honest Company to focus on data instead of maintaining infrastructure. While Honest users got delightful recommendations to improve experience, data users at Honest understood users much better in terms of segmenting with behavioral information and advanced ML models, leading to increased revenue and retention.
Spark Summit EU talk by Pat Patterson
Spark Summit EU talk by Pat PattersonSpark Summit This document discusses building data pipelines with Spark and StreamSets. It describes how StreamSets Data Collector can be used to build pipelines that run on Spark today by leveraging Kafka RDDs and containers on Spark. It also outlines future directions for deeper Spark integration, including running pipelines on Databricks and developing a standalone Spark processor. The document concludes with a demo of StreamSets Data Collector capabilities.
A Stock Prediction System using Open-Source Software
A Stock Prediction System using Open-Source SoftwareFred Melo Presentation delivered at Apache Con 2015 conference. Talks about the architecture and implementation of a real-time prediction system based on live streaming for stock quotes using Spring XD, .
A demo was also featured.
Building Sessionization Pipeline at Scale with Databricks Delta
Building Sessionization Pipeline at Scale with Databricks DeltaDatabricks "Comcast has made a concerted effort to transform itself from a cable/ISP company to a technology company. Data-driven decision making is at the heart of this transformation, and we use data to understand how customers interact with our products, and we see data as the most truthful representation of the voice of our customer. My team, Product Analytics & behavior science (PABS) team plays the role as interpreter, transforming data into consumable insights.
The X1 entertainment operating system, is one of the largest video streaming platforms in the world, and our customers consume more than a billion hours of content a week on X1. Our team consumes X1 telemetry at a rate of more than 25TBs of data per day and uses this data to inform our product teams members about the performance of and engagement with the platform. We also use this data to research customer behaviors to help better inform our product team members about areas of opportunity in our products, which range from fixing bugs to creating new features.
To power these insights, we need to have a reliable real-time data pipelines to deliver these insights, and we need our data scientists and data engineers to be able to quickly and efficiently be able to develop and commit new code to ensure we can measure new features the product teams are developing. To do this in an environment at this scale, we have been using Databricks, and Databricks delta to gain operational efficiencies, optimization and cost savings.
Some of the features from delta that we took advantage of to achieve the desired levels of efficiencies, optimization and cost savings are:
· Distributed writes to s3 (essentially eliminating 500 errors)
· s3 log with fast reads and ACID transactions (massive increases in s3 scans/reads, and enabling consistent views of the bucket/table)
· Vacuum
· Pptimize (which has allowed us to reduce a 640 node job to 40, and massively increase efficiencies of our clusters as well as our DS/DE’s)"
Data Driven Decisions at Scale
Data Driven Decisions at ScaleDatabricks Comcast is the largest cable and internet provider in the US, reaching more than 30 million customers, and continues to grow its presence in the EU with the acquisition of Sky. Over the last couple years, Comcast has shifted focus to the customer experience.
Building an Enterprise Data Platform with Azure Databricks to Enable Machine ...
Building an Enterprise Data Platform with Azure Databricks to Enable Machine ...Databricks This document summarizes Walmart's transition to building an enterprise data platform on Azure Databricks to enable machine learning and data science at scale. Previously, Walmart had a complex and slow legacy technology stack. The new platform goals were to centralize data in the cloud, increase productivity with data science tools, and reduce costs. Key aspects of the new platform included using Azure and Databricks for data processing and machine learning, Airflow for orchestration, and building several machine learning models for applications like fraud detection and product recommendations. Challenges in the transition included optimizing performance and managing resources across the platforms.
AI on Spark for Malware Analysis and Anomalous Threat Detection
AI on Spark for Malware Analysis and Anomalous Threat DetectionDatabricks At Avast, we believe everyone has the right to be safe. We are dedicated to creating a world that provides safety and privacy for all, not matter where you are, who you are, or how you connect. With over 1.5 billion attacks stopped and 30 million new executable files monthly, big data pipelines are crucial for the security of our customers. At Avast we are leveraging Apache Spark machine learning libraries and TensorflowOnSpark for a variety of tasks ranging from marketing and advertisement, through network security to malware detection. This talk will cover our main cybersecurity usecases of Spark. After describing our cluster environment we will first demonstrate anomaly detection on time series of threats. Having thousands of types of attacks and malware, AI helps human analysts select and focus on most urgent or dire threats. We will walk through our setup for distributed training of deep neural networks with Tensorflow to deploying and monitoring of a streaming anomaly detection application with trained model. Next we will show how we use Spark for analysis and clustering of malicious files and large scale experimentation to automatically process and handle changes in malware. In the end, we will give comparison to other tools we used for solving those problems.
Similar to Risk Management Framework Using Intel FPGA, Apache Spark, and Persistent RDDs with Reza Sadri and Mutema Pittman (20)
Learn How Financial Services Organizations Can Use Big Data to Mitigate Risks
Learn How Financial Services Organizations Can Use Big Data to Mitigate RisksMapR Technologies Risk comes in a variety of forms including uncertainty in financial markets, legal liabilities, operational risk, fraud, and protection against external and internal attacks. Models are becoming increasingly granular and improving risk modeling is a high priority.
Review this presentation from Splunk and MapR to learn how you can study months’ or years’ worth of raw data from disparate sources, without sampling, to understand and reduce risk.
The new dominant companies are running on data
The new dominant companies are running on data SnapLogic The cost of Digital Transformation is dropping rapidly. The technologies and methodologies are evolving to open up new opportunities for new and established corporations to drive business. We will examine specific examples of how and why a combination of robust infrastructure, cloud first and machine learning can take your company to the next level of value and efficiency.
Rich Dill, SnapLogic's enterprise solutions architect, at Big Data LDN 2017.
Big Data LDN 2017: The New Dominant Companies Are Running on Data
Big Data LDN 2017: The New Dominant Companies Are Running on DataMatt Stubbs Date: 16th November 2017
Location: Self-Service Analytics Theatre
Time: 12:30 - 13:00
Speaker: Rich Dill
Organisation: SnapLogic
Big Data LDN 2017: The New Dominant Companies Are Running on Data
Big Data LDN 2017: The New Dominant Companies Are Running on DataMatt Stubbs The document discusses solutions for deriving value from data through data integration and analytics. It describes three approaches companies have taken: 1) Building a custom machine learning platform like Uber's Michelangelo. 2) Developing custom integrations for a large multinational corporation with many technologies. 3) Implementing a cloud-first enterprise data stack for a 360-degree view of customers. The cloud-first approach provides benefits like scalability, collaboration, and reduced maintenance costs.
Digital Reinvention by NRB
Digital Reinvention by NRBWilliam Poos How to reinvent your organization in an iterative and pragmatic way? This is the result of using our digital toolbox. It allows you to transform your business model, expand your ecosystem by setting up your digital platform. This reinvention is also supported by the adaptation of your governance allowing you to innovate while guaranteeing the performance of your organization. For any information / suggestion / collaboration - [email protected]
Comment réinventer votre organisation de manière itérative et pragmatique ? C'est le résultat de l'utilisation de notre boîte à outils digitale. Elle vous permet de transformer votre modèle métier, d'étendre votre écosystème en mettant en place votre plateforme digitale. Cette réinvention est également supportée par l'adaptation de votre gouvernance vous permettant d'innover tout en garantissant la performance de votre organisation. Pour toute information / suggestion / collaboration - [email protected]
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughtonSynerzip Making AI real-time to meet mission-critical system demands put a new spin on your architecture. To deliver AI-based applications that will scale as your data grows takes a new approach where the data doesn’t become the bottleneck. We all know that the deeper the data the better the results and the lower the risk. However, doing thousands of computations on big data requires new data structures and messaging to be used together to deliver real-time AI. During this session will look at real reference architectures and review the new techniques that were needed to make AI Real-Time.
Sydney: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud 
Sydney: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud Certus Solutions Snowflake is a cloud data platform company that was founded in 2012. It has over 640 employees, 1500+ customers, and has raised $923 million in funding. Snowflake provides an elastic data warehouse that allows customers to instantly scale compute and storage resources. It offers a fully managed service with no infrastructure to manage and allows customers to consolidate siloed datasets and analyze data across multiple cloud regions and accounts.
Are your Cloud Services Secure and Compliant today?
Are your Cloud Services Secure and Compliant today?Sridhar Karnam The cyber threat landscape is evolving faster than security teams can manage without dramatically increasing headcount. As IT organizations seek to achieve new levels of IT efficiency and value for the hybrid cloud, both security and compliance headaches increase in severity as well. See how HP is delivering advanced, data-driven security technologies designed to empower security operations to run more efficiently for the Hybrid Cloud.
Register for this webinar to learn how you can benefit from a new style of IT through the combined wealth of information assimilated from multiple sources to provide you valuable insights that impact your business. In addition you will learn how you can enjoy the use of secure compliant cloud services, that can be consolidated in one view and automated to the click of a button.
Evolution of Big Data at Intel - Crawl, Walk and Run Approach
Evolution of Big Data at Intel - Crawl, Walk and Run ApproachDataWorks Summit Intel's big data journey began in 2011 with an evaluation of Hadoop. Since then, Intel has expanded its use of Hadoop and Cloudera across multiple environments. Intel's 3-year roadmap focuses on evolving its Hadoop platform to support more advanced analytics, real-time capabilities, and integrating with traditional BI tools. Key strategies include designing for scalability, following an iterative approach to understand data, and leveraging open source technologies.
Exclusive Verizon Employee Webinar: Getting More From Your CDR Data
Exclusive Verizon Employee Webinar: Getting More From Your CDR DataPentaho This document discusses a project between Pentaho and Verizon to leverage big data analytics. Verizon generates vast amounts of call detail record (CDR) data from mobile networks that is currently stored in a data warehouse for 2 years and then archived to tape. Pentaho's platform will help optimize the data warehouse by using Hadoop to store all CDR data history. This will free up data warehouse capacity for high value data and allow analysis of the full 10 years of CDR data. Pentaho tools will ingest raw CDR data into Hadoop, execute MapReduce jobs to enrich the data, load results into Hive, and enable analyzing the data to understand calling patterns by geography over time.
Webinar: Improve Splunk Analytics and Automate Processes with SnapLogic
Webinar: Improve Splunk Analytics and Automate Processes with SnapLogicSnapLogic Last week SnapLogic sponsored partner event Splunk Worldwide Users' Conference in Las Vegas. The theme of the conference was "Your Data, No Limits." In keeping with this theme, SnapLogic helps Splunk customers access more comprehensive analytics by integrating as much data as possible from as many sources as possible, and by streamlining the business process of loading data in Splunk, detecting problems, and facilitating actions that result in a prompt resolution.
To learn more, visit: http://www.snaplogic.com/.
Oil and gas big data edition
Oil and gas big data editionMark Kerzner The document discusses Informatica's data integration platform and its capabilities for big data and analytics projects. Some key points:
- Informatica is a leading data integration vendor with over 5,000 customers including over 70% of the Global 500.
- The Informatica platform provides capabilities across the entire data lifecycle from ingestion to delivery including data quality, master data management, integration, and analytics.
- It supports a variety of data sources including structured, unstructured, cloud, and big data and can run on-premises or in the cloud.
- Customers report the Informatica platform improves agility, scalability, and operational confidence for data integration projects compared to
Big Data for Product Managers
Big Data for Product ManagersPentaho Check out this presentation from Pentaho and ESRG to learn why product managers should understand Big Data and hear about real-life products that have been elevated with these innovative technologies.
Learn more in the brief that inspired the presentation, Product Innovation with Big Data: http://www.pentaho.com/resources/whitepaper/product-innovation-big-data
Fighting cyber fraud with hadoop v2
Fighting cyber fraud with hadoop v2Niel Dunnage This document discusses using Hadoop to fight cyber fraud by analyzing big data. It explains that big data technologies provide powerful tools for services but also enable malicious cyber attacks by sophisticated attackers. Hadoop allows analyzing large datasets to detect fraud and security threats through techniques like machine learning, anomaly detection, and predicting real-time and historical patterns. The document advocates asking bigger questions to innovate solutions and gain operational and business advantages from big data analytics.
QCon 2018 | Gimel | PayPal's Analytic Platform
QCon 2018 | Gimel | PayPal's Analytic PlatformDeepak Chandramouli Site | https://www.infoq.com/qconai2018/
Youtube | https://www.youtube.com/watch?v=2h0biIli2F4&t=19s
At PayPal, data engineers, analysts and data scientists work with a variety of datasources (Messaging, NoSQL, RDBMS, Documents, TSDB), compute engines (Spark, Flink, Beam, Hive), languages (Scala, Python, SQL) and execution models (stream, batch, interactive).
Due to this complex matrix of technologies and thousands of datasets, engineers spend considerable time learning about different data sources, formats, programming models, APIs, optimizations, etc. which impacts time-to-market (TTM). To solve this problem and to make product development more effective, PayPal Data Platform developed "Gimel", a unified analytics data platform which provides access to any storage through a single unified data API and SQL, that are powered by a centralized data catalog.
In this session, we will introduce you to the various components of Gimel - Compute Platform, Data API, PCatalog, GSQL and Notebooks. We will provide a demo depicting how Gimel reduces TTM by helping our engineers write a single line of code to access any storage without knowing the complexity behind the scenes.
Analyst Webinar: Best Practices In Enabling Data-Driven Decision Making
Analyst Webinar: Best Practices In Enabling Data-Driven Decision MakingDenodo Watch full webinar here: https://bit.ly/37YkgN4
This presentation looks at the trends that are emerging from companies on their journeys to becoming data-driven enterprises.
These trends are taken from a survey of 500 companies and highlight critical success factors, what companies are doing, their progress so far and their plans going forward. It also looks at the role that data virtualization has within the data driven enterprise.
During the session we'll address:
- What is a data-driven enterprise?
- What are the critical success factors?
- What are companies doing to create a data-driven enterprise and why?
- What progress are they making?
- What are the plans on people, process and technologies?
- Why is data virtualization central to provisioning and accessing data in a data-driven enterprise?
- How should you get started?
Replatform your Teradata to a Next-Gen Cloud Data Platform in Weeks, Not Years
Replatform your Teradata to a Next-Gen Cloud Data Platform in Weeks, Not YearsVMware Tanzu This document discusses Datometry, a virtualization platform that allows enterprises to replatform their legacy data warehousing workloads to the cloud at a fraction of the typical cost, time, and risk of conventional replatforming methods. It notes that Datometry keeps existing applications and logic, allowing businesses to move rapidly to the cloud while preserving years of infrastructure investment. Charts show that Datometry reduces both the upfront switching costs and ongoing annual costs of replatforming projects compared to conventional approaches. The document promotes Datometry as a way to redefine the economics of modernizing data management applications and proprietary systems like Teradata.
NRB - BE MAINFRAME DAY 2017 - Data spark and the data federation 
NRB - BE MAINFRAME DAY 2017 - Data spark and the data federation NRB Frank Van der Wal - Technical Lead IBM Z BENELUX Digital Transformation Specialist
Leif Pedersen - IBM Analytics for IBM Z Specialist at IBM
Mainframe Innovation Tour (API enconomy, Hybrid Cloud, Enterprise Linux, Machine learning, Spark)
NRB - LUXEMBOURG MAINFRAME DAY 2017 - Data Spark and the Data Federation
NRB - LUXEMBOURG MAINFRAME DAY 2017 - Data Spark and the Data FederationNRB Leif Pedersen - Executive IT Specialist, z Analytics, Europe
Data: Spark and the Data Federation
Successful AI/ML Projects with End-to-End Cloud Data Engineering
Successful AI/ML Projects with End-to-End Cloud Data EngineeringDatabricks Trusted, high-quality data and efficient use of data engineers’ time are critical success factors for AI/ML projects. Enterprise data is complex—it comes from several sources, in a variety of formats, and at varied speeds. For your machine learning projects on Apache Spark, you need a holistic approach to data engineering: finding & discovering, ingesting & integrating, server-less processing at scale, and data governance. Stop by this session for an overview on how to set up AI/ML projects for success while Informatica takes the heavy lifting out of your data engineering.
Ad
More from Databricks (20)
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDatabricks The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.
Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1Databricks The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse.
Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today.
Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow.
This is an educational event.
Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Databricks The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse.
Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today.
Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow.
This is an educational event.
Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.
Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2Databricks The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse.
Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today.
Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow.
This is an educational event.
Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Databricks The document discusses the challenges of modern data, analytics, and AI workloads. Most enterprises struggle with siloed data systems that make integration and productivity difficult. The future of data lies with a data lakehouse platform that can unify data engineering, analytics, data warehousing, and machine learning workloads on a single open platform. The Databricks Lakehouse platform aims to address these challenges with its open data lake approach and capabilities for data engineering, SQL analytics, governance, and machine learning.
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of HadoopDatabricks In this session, learn how to quickly supplement your on-premises Hadoop environment with a simple, open, and collaborative cloud architecture that enables you to generate greater value with scaled application of analytics and AI on all your data. You will also learn five critical steps for a successful migration to the Databricks Lakehouse Platform along with the resources available to help you begin to re-skill your data teams.
Democratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDatabricks Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale.
At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including:
Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal
Performing data quality validations using libraries built to work with spark
Dynamically generating pipelines that can be abstracted away from users
Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers
Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time
Learn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceDatabricks Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.
Why APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML MonitoringDatabricks Application performance monitoring (APM) has become the cornerstone of software engineering allowing engineering teams to quickly identify and remedy production issues. However, as the world moves to intelligent software applications that are built using machine learning, traditional APM quickly becomes insufficient to identify and remedy production issues encountered in these modern software applications.
As a lead software engineer at NewRelic, my team built high-performance monitoring systems including Insights, Mobile, and SixthSense. As I transitioned to building ML Monitoring software, I found the architectural principles and design choices underlying APM to not be a good fit for this brand new world. In fact, blindly following APM designs led us down paths that would have been better left unexplored.
In this talk, I draw upon my (and my team’s) experience building an ML Monitoring system from the ground up and deploying it on customer workloads running large-scale ML training with Spark as well as real-time inference systems. I will highlight how the key principles and architectural choices of APM don’t apply to ML monitoring. You’ll learn why, understand what ML Monitoring can successfully borrow from APM, and hear what is required to build a scalable, robust ML Monitoring architecture.
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixDatabricks Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. We enable data scientists to deploy and operate their models independently, with minimal need for handoffs or gatekeeping. By writing a simple function and calling out to an intuitive API, data scientists can harness a suite of platform-provided tooling meant to make ML operations easy. In this talk, we will dive into the abstractions the Data Platform team has built to enable this. We will go over the interface data scientists use to specify a model and what that hooks into, including online deployment, batch execution on Spark, and metrics tracking and visualization.
Stage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI IntegrationDatabricks In this talk, I will dive into the stage level scheduling feature added to Apache Spark 3.1. Stage level scheduling extends upon Project Hydrogen by improving big data ETL and AI integration and also enables multiple other use cases. It is beneficial any time the user wants to change container resources between stages in a single Apache Spark application, whether those resources are CPU, Memory or GPUs. One of the most popular use cases is enabling end-to-end scalable Deep Learning and AI to efficiently use GPU resources. In this type of use case, users read from a distributed file system, do data manipulation and filtering to get the data into a format that the Deep Learning algorithm needs for training or inference and then sends the data into a Deep Learning algorithm. Using stage level scheduling combined with accelerator aware scheduling enables users to seamlessly go from ETL to Deep Learning running on the GPU by adjusting the container requirements for different stages in Spark within the same application. This makes writing these applications easier and can help with hardware utilization and costs.
There are other ETL use cases where users want to change CPU and memory resources between stages, for instance there is data skew or perhaps the data size is much larger in certain stages of the application. In this talk, I will go over the feature details, cluster requirements, the API and use cases. I will demo how the stage level scheduling API can be used by Horovod to seamlessly go from data preparation to training using the Tensorflow Keras API using GPUs.
The talk will also touch on other new Apache Spark 3.1 functionality, such as pluggable caching, which can be used to enable faster dataframe access when operating from GPUs.
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorchDatabricks In this talk, I would like to introduce an open-source tool built by our team that simplifies the data conversion from Apache Spark to deep learning frameworks.
Imagine you have a large dataset, say 20 GBs, and you want to use it to train a TensorFlow model. Before feeding the data to the model, you need to clean and preprocess your data using Spark. Now you have your dataset in a Spark DataFrame. When it comes to the training part, you may have the problem: How can I convert my Spark DataFrame to some format recognized by my TensorFlow model?
The existing data conversion process can be tedious. For example, to convert an Apache Spark DataFrame to a TensorFlow Dataset file format, you need to either save the Apache Spark DataFrame on a distributed filesystem in parquet format and load the converted data with third-party tools such as Petastorm, or save it directly in TFRecord files with spark-tensorflow-connector and load it back using TFRecordDataset. Both approaches take more than 20 lines of code to manage the intermediate data files, rely on different parsing syntax, and require extra attention for handling vector columns in the Spark DataFrames. In short, all these engineering frictions greatly reduced the data scientists’ productivity.
The Databricks Machine Learning team contributed a new Spark Dataset Converter API to Petastorm to simplify these tedious data conversion process steps. With the new API, it takes a few lines of code to convert a Spark DataFrame to a TensorFlow Dataset or a PyTorch DataLoader with default parameters.
In the talk, I will use an example to show how to use the Spark Dataset Converter to train a Tensorflow model and how simple it is to go from single-node training to distributed training on Databricks.
Scaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on KubernetesDatabricks There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal.
In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesDatabricks Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark.
Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations.
Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.
Sawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature AggregationsDatabricks In this talk about zipline, we will introduce a new type of windowing construct called a sawtooth window. We will describe various properties about sawtooth windows that we utilize to achieve online-offline consistency, while still maintaining high-throughput, low-read latency and tunable write latency for serving machine learning features.We will also talk about a simple deployment strategy for correcting feature drift – due operations that are not “abelian groups”, that operate over change data.
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen SinkDatabricks We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark.
Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue
· Why?
o Custom queries on top a table; We load the data once and query N times
· Why not Structured Streaming
· Working Solution using Redis
Niche 2 : Distributed Counters
· Problems with Spark Accumulators
· Utilize Redis Hashes as distributed counters
· Precautions for retries and speculative execution
· Pipelining to improve performance
Re-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and SparkDatabricks In the era of microservices, decentralized ML architectures and complex data pipelines, data quality has become a bigger challenge than ever. When data is involved in complex business processes and decisions, bad data can, and will, affect the bottom line. As a result, ensuring data quality across the entire ML pipeline is both costly, and cumbersome while data monitoring is often fragmented and performed ad hoc. To address these challenges, we built whylogs, an open source standard for data logging. It is a lightweight data profiling library that enables end-to-end data profiling across the entire software stack. The library implements a language and platform agnostic approach to data quality and data monitoring. It can work with different modes of data operations, including streaming, batch and IoT data.
In this talk, we will provide an overview of the whylogs architecture, including its lightweight statistical data collection approach and various integrations. We will demonstrate how the whylogs integration with Apache Spark achieves large scale data profiling, and we will show how users can apply this integration into existing data and ML pipelines.
Raven: End-to-end Optimization of ML Prediction Queries
Raven: End-to-end Optimization of ML Prediction QueriesDatabricks Machine learning (ML) models are typically part of prediction queries that consist of a data processing part (e.g., for joining, filtering, cleaning, featurization) and an ML part invoking one or more trained models. In this presentation, we identify significant and unexplored opportunities for optimization. To the best of our knowledge, this is the first effort to look at prediction queries holistically, optimizing across both the ML and SQL components.
We will present Raven, an end-to-end optimizer for prediction queries. Raven relies on a unified intermediate representation that captures both data processing and ML operators in a single graph structure.
This allows us to introduce optimization rules that
(i) reduce unnecessary computations by passing information between the data processing and ML operators
(ii) leverage operator transformations (e.g., turning a decision tree to a SQL expression or an equivalent neural network) to map operators to the right execution engine, and
(iii) integrate compiler techniques to take advantage of the most efficient hardware backend (e.g., CPU, GPU) for each operator.
We have implemented Raven as an extension to Spark’s Catalyst optimizer to enable the optimization of SparkSQL prediction queries. Our implementation also allows the optimization of prediction queries in SQL Server. As we will show, Raven is capable of improving prediction query performance on Apache Spark and SQL Server by up to 13.1x and 330x, respectively. For complex models, where GPU acceleration is beneficial, Raven provides up to 8x speedup compared to state-of-the-art systems. As part of the presentation, we will also give a demo showcasing Raven in action.
Processing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache SparkDatabricks Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis.
Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them.
Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy.
This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.
Massive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta LakeDatabricks At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences.
What are we storing?
Multi Source – Multi Channel Problem
Data Representation and Nested Schema Evolution
Performance Trade Offs with Various formats
Go over anti-patterns used
(String FTW)
Data Manipulation using UDFs
Writer Worries and How to Wipe them Away
Staging Tables FTW
Datalake Replication Lag Tracking
Performance Time!
Ad
Recently uploaded (20)
Decision Trees in Artificial-Intelligence.pdf
Decision Trees in Artificial-Intelligence.pdfSaikat Basu Have you heard of something called 'Decision Tree'? It's a simple concept which you can use in life to make decisions. Believe you me, AI also uses it.
Let's find out how it works in this short presentation. #AI #Decisionmaking #Decisions #Artificialintelligence #Data #Analysis
https://saikatbasu.me
computer organization and assembly language.docx
computer organization and assembly language.docxalisoftwareengineer1 computer organization and assembly language : its about types of programming language along with variable and array description..https://www.nfciet.edu.pk/
04302025_CCC TUG_DataVista: The Design Story
04302025_CCC TUG_DataVista: The Design Storyccctableauusergroup CCCCO and WestEd share the story behind how DataVista came together from a design standpoint and in Tableau.
Calories_Prediction_using_Linear_Regression.pptx
Calories_Prediction_using_Linear_Regression.pptxTijiLMAHESHWARI Calorie prediction using machine learning
Data Science Courses in India iim skills
Data Science Courses in India iim skillsdharnathakur29 This comprehensive Data Science course is designed to equip learners with the essential skills and knowledge required to analyze, interpret, and visualize complex data. Covering both theoretical concepts and practical applications, the course introduces tools and techniques used in the data science field, such as Python programming, data wrangling, statistical analysis, machine learning, and data visualization.
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management By James Francis, CEO of Paradigm Asset Management
In the landscape of urban safety innovation, Mt. Vernon is emerging as a compelling case study for neighboring Westchester County cities. The municipality’s recently launched Public Safety Camera Program not only represents a significant advancement in community protection but also offers valuable insights for New Rochelle and White Plains as they consider their own safety infrastructure enhancements.
Process Mining and Data Science in the Financial Industry
Process Mining and Data Science in the Financial IndustryProcess mining Evangelist Lalit Wangikar, a partner at CKM Advisors, is an experienced strategic consultant and analytics expert. He started looking for data driven ways of conducting process discovery workshops. When he read about process mining the first time around, about 2 years ago, the first feeling was: “I wish I knew of this while doing the last several projects!".
Interviews are subject to all the whims human recollection is subject to: specifically, recency, simplification and self preservation. Interview-based process discovery, therefore, leaves out a lot of “outliers” that usually end up being one of the biggest opportunity area. Process mining, in contrast, provides an unbiased, fact-based, and a very comprehensive understanding of actual process execution.
Classification_in_Machinee_Learning.pptx
Classification_in_Machinee_Learning.pptxwencyjorda88 Brief powerpoint presentation about different classification of machine learning
定制学历(美国Purdue毕业证)普渡大学电子版毕业证
定制学历(美国Purdue毕业证)普渡大学电子版毕业证Taqyea 2025年新版美国毕业证普渡大学文凭【q微1954292140】办理普渡大学毕业证(Purdue毕业证书)国外学位认证/毕业证购买【q微1954292140】普渡大学offer/学位证、留信官方学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作【q微1954292140】Buy Purdue University Diploma购买美国毕业证，购买英国毕业证，购买澳洲毕业证，购买加拿大毕业证，以及德国毕业证，购买法国毕业证（q微1954292140）购买荷兰毕业证、购买瑞士毕业证、购买日本毕业证、购买韩国毕业证、购买新西兰毕业证、购买新加坡毕业证、购买西班牙毕业证、购买马来西亚毕业证等。包括了本科毕业证，硕士毕业证。
主营项目：
1、真实教育部国外学历学位认证《美国毕业文凭证书快速办理普渡大学国外本科offer在线制作》【q微1954292140】《论文没过普渡大学正式成绩单》，教育部存档，教育部留服网站100%可查.
2、办理Purdue毕业证，改成绩单《Purdue毕业证明办理普渡大学制作成绩单》【Q/WeChat：1954292140】Buy Purdue University Certificates《正式成绩单论文没过》，普渡大学Offer、在读证明、学生卡、信封、证明信等全套材料，从防伪到印刷，从水印到钢印烫金，高精仿度跟学校原版100%相同.
3、真实使馆认证（即留学人员回国证明），使馆存档可通过大使馆查询确认.
4、留信网认证，国家专业人才认证中心颁发入库证书，留信网存档可查.
《普渡大学成绩单制作案例美国毕业证书办理Purdue2025年新版毕业证书》【q微1954292140】学位证1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。
高仿真还原美国文凭证书和外壳，定制美国普渡大学成绩单和信封。毕业证办理需要多久拿到？Purdue毕业证【q微1954292140】办理美国普渡大学毕业证(Purdue毕业证书)【q微1954292140】文凭办理普渡大学offer/学位证成绩单定制、留信官方学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作。帮你解决普渡大学学历学位认证难题。
美国文凭普渡大学成绩单，Purdue毕业证【q微1954292140】办理美国普渡大学毕业证(Purdue毕业证书)【q微1954292140】专业定制国外成绩单普渡大学offer/学位证成绩单温感光标、留信官方学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作。帮你解决普渡大学学历学位认证难题。
美国文凭购买，美国文凭定制，美国文凭补办。专业在线定制美国大学文凭，定做美国本科文凭，【q微1954292140】复制美国Purdue University completion letter。在线快速补办美国本科毕业证、硕士文凭证书，购买美国学位证、普渡大学Offer，美国大学文凭在线购买。
【q微1954292140】帮您解决在美国普渡大学未毕业难题（Purdue University）文凭购买、毕业证购买、大学文凭购买、大学毕业证购买、买文凭、日韩文凭、英国大学文凭、美国大学文凭、澳洲大学文凭、加拿大大学文凭（q微1954292140）新加坡大学文凭、新西兰大学文凭、爱尔兰文凭、西班牙文凭、德国文凭、教育部认证，买毕业证，毕业证购买，买大学文凭，购买日韩毕业证、英国大学毕业证、美国大学毕业证、澳洲大学毕业证、加拿大大学毕业证（q微1954292140）新加坡大学毕业证、新西兰大学毕业证、爱尔兰毕业证、西班牙毕业证、德国毕业证，回国证明，留信网认证，留信认证办理，学历认证。从而完成就业。普渡大学毕业证办理，普渡大学文凭办理，普渡大学成绩单办理和真实留信认证、留服认证、普渡大学学历认证。学院文凭定制，普渡大学原版文凭补办，扫描件文凭定做，100%文凭复刻。
特殊原因导致无法毕业，也可以联系我们帮您办理相关材料：
１：在普渡大学挂科了，不想读了，成绩不理想怎么办？？？
2：打算回国了，找工作的时候，需要提供认证《Purdue成绩单购买办理普渡大学毕业证书范本》【Q/WeChat：1954292140】Buy Purdue University Diploma《正式成绩单论文没过》有文凭却得不到认证。又该怎么办？？？美国毕业证购买，美国文凭购买，
Deloitte Analytics - Applying Process Mining in an audit context
Deloitte Analytics - Applying Process Mining in an audit contextProcess mining Evangelist Mieke Jans is a Manager at Deloitte Analytics Belgium. She learned about process mining from her PhD supervisor while she was collaborating with a large SAP-using company for her dissertation.
Mieke extended her research topic to investigate the data availability of process mining data in SAP and the new analysis possibilities that emerge from it. It took her 8-9 months to find the right data and prepare it for her process mining analysis. She needed insights from both process owners and IT experts. For example, one person knew exactly how the procurement process took place at the front end of SAP, and another person helped her with the structure of the SAP-tables. She then combined the knowledge of these different persons.
Perencanaan Pengendalian-Proyek-Konstruksi-MS-PROJECT.pptx
Perencanaan Pengendalian-Proyek-Konstruksi-MS-PROJECT.pptxPareaRusan planning and calculation monitoring project
Deloitte - A Framework for Process Mining Projects
Deloitte - A Framework for Process Mining ProjectsProcess mining Evangelist Tijn van der Heijden is a business analyst with Deloitte. He learned about process mining during his studies in a BPM course at Eindhoven University of Technology and became fascinated with the fact that it was possible to get a process model and so much performance information out of automatically logged events of an information system.
Tijn successfully introduced process mining as a new standard to achieve continuous improvement for the Rabobank during his Master project. At his work at Deloitte, Tijn has now successfully been using this framework in client projects.
CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...
CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS Prediction of Aluminium wire rod physical properties through AI, ML
or any modern technique for better productivity and quality control.
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management
Risk Management Framework Using Intel FPGA, Apache Spark, and Persistent RDDs with Reza Sadri and Mutema Pittman
- 1. Reza Sadri, Levyx Mutema Pittman, Intel June 2018 Financial Risk Analytics Acceleration Framework Using Intel FPGAs, Apache Spark, and Persistent DataFrames #HWCSAIS15
- 2. The Deluge O Data Is Here And Accelerating 2 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential Modern Financial Institutions are Cutting-Edge Technology Companies Goldman Sachs at AWS Re:Invent Keynote 2017
- 3. The Deluge O Data Is Here And Accelerating 3 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential Risk Management is a Critical Capability
- 4. The Deluge O Data Is Here And Accelerating 4 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential Risk Management is Driving Insatiable Demand for More Compute 4
- 5. The Deluge O Data Is Here And Accelerating 5 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential Market Need for Improving the Metrics of Financial Risk Analysis Time = Money for financial firms § Investing in new approaches using alternative data § Budgets shifting from compliance to competitive advantage models Shift to public cloud enables Risk Management for more areas: • More data • More diverse data • More calculations • Sharing live data becomes important
- 6. Case Study: Financial Risk Analytics • Application: Risk analytics acceleration framework (financial backtesting) • Current solution: Deploy a cluster of CPUs or GPUs with complex data access • Challenge: Compute intensive, time consuming applications --10+ hours for financial backtesting • Solution Value Proposition: – > 5x performance improvement using FPGA acceleration and SSD-optimized data engines – Financial Derivatives calculations of risk and pricing simultaneously – Integrated solution with Apache Spark, SSD access and FPGA implementation abstracted away 6
- 7. The Deluge O Data Is Here And Accelerating 7 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential Why Persistent Dataframes are Important • Reside on Flash, cached in memory à better use of hardware • Same exact syntax • No need to re-compute after the job is done • Shareable across jobs
- 8. The Deluge O Data Is Here And Accelerating 8 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential Using Levyx Xenon™ with Intel’s Optane™ and FPGA/Finlib in a Risk Analytics Platform ™ Optane™ Xeon Cores Intel FPGA(s) NIC/ RDMA Xenon Spark Connector Direct Interface To Hardware Risk Management Spark Applications (Written in Python, R or Scala) Finlib User Defined Functions - Historical Market Data - Private Transaction Data - Alternative Data Institution’s Data Lake - Reference Data - Risk and Back Testing Results
- 9. 9 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential 4 584,396,032 53,681,757 17,218,533 20 2,968,752,864 54,329,582 18,697,911 80 11,309,534,720 45,834,352 16,867,758 SYMBOLS XENON/FPGA TIME OPTIONS/SEC # OF OPTIONS Levyx Xenon™ Combined With Intel FPGAs & SSDs is 3X Faster than Conventional Spark SPARK CALCULATION OPTIONS/SEC Symbols: The number of stock symbols used i.e. IBM, APPL, MSFT, etc. Options: An individual “Black-Scholes Function” call to value a specific Call or Put option with a specific set of parameters as might be traded on the Chicago Board of Options Exchange
- 10. FPGA Acceleration Stack with Libraries 10 BSP (Board Support Package)Driver Low-Level FPGA Management OpenCL SDKOpenCL Runtime OpenCL API Provides Kernel Abstraction and Accelerator Management User DesignUser Application Library Infrastructure Library Kernels Compute Primitives Infra. Primitives Library Orchestration Software Frameworks Increase Abstraction Increase User Base Libraries are callable from high level languages (e.g. C/C++, Python) and software frameworks (MKL) - making FPGA accelerated computation easy to access Intel Programmable Acceleration Card
- 11. 11 Completion of all three FinLib phases will provide functionality equivalent to top commercial financial analytics libraries FinLib – Functionality Roadmap
- 12. The Deluge O Data Is Here And Accelerating 12 © Copyright 2013-2018 Levyx Inc. Proprietary and Confidential Summary • Using of accelerated standardized risk functions to compute JIT large-scale data tables using FPGAs yields breakthrough performance • No custom coding required • DRAM consumption “right-sized” using Flash/NVM • Bypass kernel for I/O