Data Tools and the Data Scientist Shortage

Nov 5, 201515 likes3,676 views

Wes McKinney discusses the shortage of data scientists and analysts. There is a shortage of 140,000-190,000 people with analytics expertise and 1.5 million managers/analysts with skills to understand and make decisions based on big data analysis in the United States alone. This shortage can be addressed through improved education, tools, and a cultural shift. New approaches and tools are needed to make data science accessible to more people and bring analytics capabilities to various industries.

1
©
Cloudera,
Inc.
All
rights
reserved.

Data
Tools
and
the
Data

Scien;st
Shortage

Wes
McKinney
@wesmckinn

Data
Summit
@
Web
Summit
2015-‐11-‐04

2
©
Cloudera,
Inc.
All
rights
reserved.

Me

3
©
Cloudera,
Inc.
All
rights
reserved.

Career
theme:
Serial
creator
of
data
tools

4
©
Cloudera,
Inc.
All
rights
reserved.

hMps://hbr.org/2012/10/data-‐scien;st-‐the-‐sexiest-‐job-‐of-‐the-‐21st-‐century/

5
©
Cloudera,
Inc.
All
rights
reserved.

hMp://www.bloomberg.com/news/ar;cles/2015-‐06-‐04/help-‐wanted-‐black-‐belts-‐in-‐data

6
©
Cloudera,
Inc.
All
rights
reserved.

“The
United
States
alone
faces
a
shortage
of
140,000

to
190,000
people
with
analy;cal
exper;se
and
1.5

million
managers
and
analysts
with
the
skills
to

understand
and
make
decisions
based
on
the
analysis

of
big
data.”

McKinsey
&
Co

hMp://www.mckinsey.com/features/big_data

7
©
Cloudera,
Inc.
All
rights
reserved.

Source:
Drew
Conway,
“The
Data
Science
Venn
Diagram”

Tradi;onal
view
of
Data
Science

8
©
Cloudera,
Inc.
All
rights
reserved.

Analyzing
the
Analyzers,
Harris,
Murphy,
Vaisman

Many
Kinds
of
“Data
People”

9
©
Cloudera,
Inc.
All
rights
reserved.

Analyzing
the
Analyzers,
Harris,
Murphy,
Vaisman

Many
Kinds
of
“Data
People”

10
©
Cloudera,
Inc.
All
rights
reserved.

Addressing
the
analy;cal
shortage

Educa;on
Culture
Tools

11
©
Cloudera,
Inc.
All
rights
reserved.

Data
process

12
©
Cloudera,
Inc.
All
rights
reserved.

The
“Great
Decoupling”
for
Industry
Analy;cs

UI
ComputeStorage

13
©
Cloudera,
Inc.
All
rights
reserved.

The
“Great
Decoupling”
for
Industry
Analy;cs

UI
ComputeStorage
Accumula;on
of
user
;me

Legacy
technology:

ver;cally-‐integrated

solu;ons

14
©
Cloudera,
Inc.
All
rights
reserved.

Ubiquitous
Real-‐Time
Storage
and

Compute:
A
view
from
2040

15
©
Cloudera,
Inc.
All
rights
reserved.

Data
analysis
hierarchy
of
needs

Data Storage / Access
Clean Data
Analysis and Visualization
Productivity tools / UI

16
©
Cloudera,
Inc.
All
rights
reserved.

Some
data
tooling
UI
innova;ons

17
©
Cloudera,
Inc.
All
rights
reserved.

Rejec;ng
the
“Highlander
Fallacy”

18
©
Cloudera,
Inc.
All
rights
reserved.

SQL
Programming:
the
“mainframe

punch
cards”
of
our
;me

19
©
Cloudera,
Inc.
All
rights
reserved.

Many
SQL
engines

…
and
more

20
©
Cloudera,
Inc.
All
rights
reserved.

Execu;ng
data
science
languages
in
the
compute
layer

UI
Ibis, SQL, Spark API, …
Compute
Analytic SQL, Spark, MapReduce
Storage
HDFS, Kudu, HBase
Python,
R, Julia, …?

21
©
Cloudera,
Inc.
All
rights
reserved.

22
©
Cloudera,
Inc.
All
rights
reserved.

Thank
you

Wes
McKinney
@wesmckinn

Views
are
my
own

This document summarizes Wes McKinney's presentation on scaling Python analytics on Hadoop and Impala. The presentation discusses how Python has become popular for data science but does not currently scale to big data. It introduces Ibis as a project that aims to maximize Python productivity for data engineers and scientists by enabling them to solve big data problems without leaving Python. Ibis uses a composable Python DSL to generate SQL queries that can be executed using distributed SQL engines like Impala at scale.

PyData: The Next GenerationWes McKinney

Wes McKinney gave a talk at Data Day Texas 2015 about the past, present, and future of the Python data analysis community and ecosystem, known as PyData. He discussed how Python became a popular language for data analysis due to tools like NumPy, pandas, scikit-learn, and IPython that enabled interactive exploration and modeling of data. However, Python was not initially well-suited for large-scale "big data" problems involving Hadoop and Spark. Recent developments in PySpark and improved integration of Python with big data frameworks have helped address this, but challenges remain in data structures, emerging formats, and open-sourcing of Python big data solutions.

Hadoop Essentials -- The What, Why and How to Meet Agency ObjectivesCloudera, Inc.

This session will provide an executive overview of the Apache Hadoop ecosystem, its basic concepts, and its real-world applications. Attendees will learn how organizations worldwide are using the latest tools and strategies to harness their enterprise information to solve business problems and the types of data analysis commonly powered by Hadoop. Learn how various projects make up the Apache Hadoop ecosystem and the role each plays to improve data storage, management, interaction, and analysis. This is a valuable opportunity to gain insights into Hadoop functionality and how it can be applied to address compelling business challenges in your agency.

How to Build Continuous Ingestion for the Internet of ThingsCloudera, Inc.

The Internet of Things is moving into the mainstream and this new world of data-driven products is transforming a vast number of industry sectors and technologies. However, IoT creates a new challenge: how to build and operationalize continual data ingestion from such a wide and ever-changing array of endpoints so that the data arrives consumption-ready and can drive analysis and action within the business. In this webinar, Sean Anderson from Cloudera and Kirit Busu, Director of Product Management at StreamSets, will discuss Hadoop's ecosystem and IoT capabilities and provide advice about common patterns and best practices. Using specific examples, they will demonstrate how to build and run end-to-end IOT data flows using StreamSets and Cloudera infrastructure.

DataFrames: The Good, Bad, and UglyWes McKinney

The document discusses different data frame interfaces, including their strengths and weaknesses. It describes R data frames as having a thin layer on top of R lists with simple column/row selection. Key R packages like dplyr and data.table add functionality. Spark DataFrames provide a pandas-inspired API for tabular data manipulation across languages. While progressing towards decoupling, interfaces still bind users to their specific systems. The author advocates for quality tools forged through real-world usage.

Introduction To Big Data Analytics On Hadoop - SpringPeopleSpringPeople

Big data analytics uses tools like Hadoop and its components HDFS and MapReduce to store and analyze large datasets in a distributed environment. HDFS stores very large data sets reliably and streams them at high speeds, while MapReduce allows developers to write programs that process massive amounts of data in parallel across a distributed cluster. Other concepts discussed in the document include data preparation, visualization, hypothesis testing, and deductive vs inductive reasoning as they relate to big data analytics. The document aims to introduce readers to big data analytics using Hadoop and suggests the audience as data analysts, scientists, database managers, and consultants.

MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron (...DataWorks Summit

Apache Metron (Incubating) is a streaming cybersecurity application built on Apache Storm and Hadoop. One of its core missions is to enable advanced analytics through machine learning and data science to the users. Because of the relative immaturity of data science platform infrastructure integrated into Hadoop that is oriented to streaming analytics applications, we have been forced to create the requisite platform components out of necessity, utilizing many of the pieces of the Hadoop ecosystem. In this talk, we will speak about the Metron analytics architecture and how it utilizes a custom data science model deployment and autodiscovery service that is tightly integrated with Hadoop via Yarn and Zookeeper. We will discuss how we interact with the models deployed there via a custom domain specific language that can query models as data streams past. We will generally discuss the full-stack data science tooling that has been created to enable data science at scale on an advanced analytics streaming application.

Big data coursekiruthikab6

The Big Data Hadoop Certification Training Course aims to provide complete knowledge of Big Data and Hadoop technologies including HDFS, YARN, and MapReduce. It offers comprehensive knowledge of tools in the Hadoop ecosystem like Pig, Hive, Sqoop, Flume, Oozie, and HBase. Students will learn to ingest and analyze large datasets stored in HDFS using real-world industry projects covering domains such as banking, telecommunications, social media, insurance, and e-commerce. Graduates can expect average salaries of Rs. 7,12,453 per year for Hadoop engineers according to payscale.com.

Risk Management Framework Using Intel FPGA, Apache Spark, and Persistent RDDs...Databricks

Performing analytics for risk management purposes is applied in many fields, especially in financial services. We present a framework for accelerated risk analytics and show a large-scale financial sector application where this framework is used to run backtesting algorithms on risk-based securities such as options. These applications require highly computationally-intensive operations on extremely large data sets with objects numbering in the tens of billions. Intel FPGA and FinLib library for financial applications are used to offload the computation; however, another challenging problem (that we have resolved) is how to feed the data to the FPGA at the optimal speed without having to do customized coding. A combination of Apache Spark along with Levyx’s persistent dataframes are used to address this problem. These dataframes allow absorbing the computation from Spark and offloading it to Finlib in an automated way. This example can be expanded to many other areas of Risk Management such as Insurance and Cybersecurity.

Managed Cluster ServicesAdam Doyle

2016 Cybersecurity Analytics State of the UnionCloudera, Inc.

PyCon Singapore 2013 KeynoteWes McKinney

This document summarizes a talk given by Wes McKinney on the future of Python and data analysis. It discusses how pandas has helped make Python a good language for data preparation and analysis, as well as trends like the rise of web/cloud computing and big data that Python needs to keep up with. It suggests embracing JavaScript more to make Python more web-friendly, and notes opportunities around data on the web, golden ages of web visualization, and leveraging new JIT compiler technologies to keep Python relevant for data.

Managing the Dewey Decimal SystemDataWorks Summit

The Future of Column-Oriented Data Processing With Apache Arrow and Apache Pa...Dremio Corporation

Essentially every successful analytical DBMS in the market today makes use of column-oriented data structures. In the Hadoop ecosystem, Apache Parquet (and Apache ORC) provide similar advantages in terms of processing and storage efficiency. Apache Arrow is the in-memory counterpart to these formats and has been been embraced by over a dozen open source projects as the de facto standard for in-memory processing. In this session the PMC Chair for Apache Arrow and the PMC Chair for Apache Parquet discuss the future of column-oriented processing.

Webinar: Proofpoint, a pioneer in security-as-a-service protects people, info...DataStax

Proofpoint is a $3 billion public cloud security company that acquired Nexgate, an early DataStax customer, in 2014. Proofpoint uses Cassandra all over the organization, with a current production deployment of 3 TB of data across 23 nodes in 9 data centers across 4 clusters. Over time, Proofpoint has evolved its Cassandra usage from a single data center with 3 nodes in 2012 to multiple data centers with Solr deployment today. Proofpoint discussed several use cases for Cassandra including detecting phishing, analyzing spam patterns, trending topics analysis, archive searching, threat event correlation, and more. The presentation provided advice on Cassandra best practices and contacts for further information.

Deep Learning with ClouderaCloudera, Inc.

Deep learning expands boundaries of the possible. Detecting fraud. Predicting claims. Diagnosing cancer. Deep learning solves these problems and many others. However, organizations struggle to make deep learning work. Cloudera—with tools like the Cloudera Data Science Workbench—helps you bring deep learning to your data, for new insights and applications. A demonstration of Cloudera Data Science Workbench is included in the webinar.

Govern This! Data Discovery and the application of data governance with new s...Cloudera, Inc.

How to boost your datamanagement with Dremio ?Vincent Terrasi

Works with any source. Relational, non-relational, 3rd party apps. 5 years ago nobody was using Hadoop, MongoDB, and 5 years from now there will be new products. You need a solution that is future proof. Works with any BI tool. In every company multiple tools are in use. Each department has their favorite. We need to work with all of them. No ETL, data warehouse, cubes. This would need to give you a really good alternative to these options. Makes data self-service, collaborative. Probably most important of all, we need to change the dynamic between the business and IT. We need to make it so business users can get the data they want, in the shape they want it, without waiting on IT. Makes Big Data feels small. It needs to make billions of rows feel like a spreadsheet on your desktop. Open source. It’s 2017, so we think this has to be open source.

Introduction to hadoopdhruv_gairola

This document provides an introduction to Dhruv Gairola and Datachili Inc. It discusses big data challenges related to volume, variety, velocity and veracity of data. It then provides an overview of Hadoop components like HDFS, MapReduce and YARN. Examples of Hadoop usage include fraud detection, data warehousing, and click analytics. The document also discusses Spark as part of the Hadoop ecosystem and how Datachili uses Hadoop and Spark for data preparation, joining datasets, and automatic analytics.

Hadoop Tutorial For BeginnersDataflair Web Services Pvt Ltd

Dremio introductionAlexis Gendronneau

This document discusses Apache Dremio, an open source data virtualization platform that provides self-service SQL access to data sources like Elasticsearch, MongoDB, HDFS, and relational databases. It aims to make data analytics faster by avoiding the need for data staging, warehouses, cubes, and extracts. Dremio uses techniques like reflections, pushdowns, and a universal relational algebra to optimize queries and leverage caches. It is based on projects like Apache Drill, Calcite, Arrow, and Parquet and can be deployed on Hadoop or the cloud. The presentation includes a demo of using Dremio to create datasets, curate/prepare data, accelerate queries with reflections, and manage resources.

Introduction of Big data and Hadoop Arohi Khandelwal

Seeking Cybersecurity--Strategies to Protect the DataCloudera, Inc.

Agency professionals are responsible for protecting the data they collect, store, analyze, and share. While Hadoop has been especially popular for data analytics given its ability to handle volume, velocity, and variety of data, this flexibility and scale can present challenges for securing and governing the data. Plan to attend this session to understand the Hadoop Security Maturity Model—from the fundamentals to the latest developments--and how to ensure your data analytics cluster complies with the latest INFOSEC standards and audit requirements. Bring your experience and your questions to this informative and interactive cybersecurity session.

Available platforms for Big Data 2.0Petr Novotný

Big Data, a recent phenomenon. Everyone talks about it, but do you really know what Big Data is? Join our four-part series about Big Data and you will get answers to your questions! We will cover Introduction to Big Data and available platforms which we can use to deal with Big Data. And in the end, we are going to give you an insight into the possible future of dealing with Big Data. Spark, Flink, Presto and many others. This is just a sample of frameworks which are used in real companies and we will talk about some of them. In the previous episode of this Big Data series, we talked about the basic information concerning Big Data. This presentation, however, will be much more technical as we will be covering the most popular platforms you can use to deal with Big Data 2.0 Systems and learn about the key differences between these platforms. Let’s go! #CHEDTEB www.chedteb.eu

Big Data and Hadoop - key drivers, ecosystem and use casesJeff Kelly

This document discusses big data and Hadoop. It defines big data as extremely large data sets that are difficult to process using traditional databases. Three key drivers of big data are identified as volume, variety and velocity of data. Hadoop is introduced as an open source framework for storing and processing big data across multiple machines in parallel. Examples of big data pioneers using Hadoop like Yahoo, Facebook and LinkedIn are provided. Potential uses of big data in the financial services industry are also briefly outlined.

Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit

Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.

NOVA Data Science Meetup 2-21-2018 Presentation Cloudera Data Science WorkbenchNOVA DATASCIENCE

This document discusses Cloudera's Data Science Workbench (CDSW) product. It begins with an introduction and agenda. It then discusses challenges with data science projects and how CDSW aims to help by providing a shared platform for data access, analytics and model deployment. The document outlines CDSW's architecture built on Docker and Kubernetes. It demonstrates CDSW's capabilities and integrations with Cloudera's Data Hub platform before concluding with information about Cloudera's research team.

Next-Gen ML/AI PlatformJosh Yeh

The document discusses challenges in building machine learning platforms and pipelines. It covers topics like data exploration challenges due to versioning issues; managing large numbers of model experiments with different hyperparameters, datasets, and performance tracking; and difficulties deploying models at scale for monitoring. The presentation demonstrates examples of machine learning applications in industries like telecommunications, manufacturing, and finance. It also discusses trends in deep learning, distributed learning, transfer learning, and edge device machine learning.

The Vision & Challenge of Applied Machine LearningCloudera, Inc.

Analytics, Everywhere. Keys to Effective Analytics and Data DiscoveryDLT Solutions

More Related Content

What's hot (19)

Risk Management Framework Using Intel FPGA, Apache Spark, and Persistent RDDs...Databricks

Managed Cluster ServicesAdam Doyle

2016 Cybersecurity Analytics State of the UnionCloudera, Inc.

PyCon Singapore 2013 KeynoteWes McKinney

Managing the Dewey Decimal SystemDataWorks Summit

The Future of Column-Oriented Data Processing With Apache Arrow and Apache Pa...Dremio Corporation

Webinar: Proofpoint, a pioneer in security-as-a-service protects people, info...DataStax

Deep Learning with ClouderaCloudera, Inc.

Govern This! Data Discovery and the application of data governance with new s...Cloudera, Inc.

How to boost your datamanagement with Dremio ?Vincent Terrasi

Introduction to hadoopdhruv_gairola

Hadoop Tutorial For BeginnersDataflair Web Services Pvt Ltd

Dremio introductionAlexis Gendronneau

Introduction of Big data and Hadoop Arohi Khandelwal

Seeking Cybersecurity--Strategies to Protect the DataCloudera, Inc.

Available platforms for Big Data 2.0Petr Novotný

Big Data and Hadoop - key drivers, ecosystem and use casesJeff Kelly

Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit

NOVA Data Science Meetup 2-21-2018 Presentation Cloudera Data Science WorkbenchNOVA DATASCIENCE

Risk Management Framework Using Intel FPGA, Apache Spark, and Persistent RDDs...Databricks

Managed Cluster ServicesAdam Doyle

2016 Cybersecurity Analytics State of the UnionCloudera, Inc.

PyCon Singapore 2013 KeynoteWes McKinney

Managing the Dewey Decimal SystemDataWorks Summit

The Future of Column-Oriented Data Processing With Apache Arrow and Apache Pa...Dremio Corporation

Webinar: Proofpoint, a pioneer in security-as-a-service protects people, info...DataStax

Deep Learning with ClouderaCloudera, Inc.

Govern This! Data Discovery and the application of data governance with new s...Cloudera, Inc.

How to boost your datamanagement with Dremio ?Vincent Terrasi

Introduction to hadoopdhruv_gairola

Hadoop Tutorial For BeginnersDataflair Web Services Pvt Ltd

Dremio introductionAlexis Gendronneau

Introduction of Big data and Hadoop Arohi Khandelwal

Seeking Cybersecurity--Strategies to Protect the DataCloudera, Inc.

Available platforms for Big Data 2.0Petr Novotný

Big Data and Hadoop - key drivers, ecosystem and use casesJeff Kelly

Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit

NOVA Data Science Meetup 2-21-2018 Presentation Cloudera Data Science WorkbenchNOVA DATASCIENCE

Similar to Data Tools and the Data Scientist Shortage (20)

Next-Gen ML/AI PlatformJosh Yeh

The Vision & Challenge of Applied Machine LearningCloudera, Inc.

Analytics, Everywhere. Keys to Effective Analytics and Data DiscoveryDLT Solutions

Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.

Build a modern platform for anti-money laundering 9.19.18Cloudera, Inc.

Keynote: The Journey to Pervasive AnalyticsCloudera, Inc.

We are in the middle of a data rush. When you are right in the center of a storm, it can seem overwhelming. Where should I start? What do I need to think about? What is the best long-term bet? But don’t forget that more data should mean great news. More data should mean more insight, more guidance, and more strategic direction. However, more data doesn’t automatically rally your entire business around common goals and insights. You need a platform and architecture that can support a thriving, analytic-driven business culture that embraces a pervasive analytics strategy.

Technology Primer: Hey IT—Your Big Data Infrastructure Can’t Sit in a Silo An...CA Technologies

This document discusses the need for IT teams to unify big data and traditional IT infrastructure management. As big data projects grow rapidly within organizations in different silos, a unified approach is needed to manage complexity, scale, and visibility across systems. The CA Unified Infrastructure Management (CA UIM) solution provides a single pane of glass for monitoring big data frameworks like Hadoop, MongoDB, and Cassandra alongside traditional IT infrastructure. It offers comprehensive coverage, customizable dashboards, performance analytics and intelligent alerts to improve services levels and reduce costs associated with multiple monitoring tools.

151116 Sedania Cloudera BDA ProfileZarul Zaabah

This document discusses Cloudera's big data solutions and provides examples of how organizations have used Cloudera to optimize data and achieve business goals. It highlights Cloudera's large partner ecosystem and customer base across various industries. Specific use cases are presented on customer experience management, network optimization, and operational analytics. The document promotes Cloudera as enabling data-driven decisions, improved efficiencies, and new business opportunities through modern data architectures.

Edc event vienna presentation 1 oct 2019Cloudera, Inc.

The document outlines the agenda for Cloudera's Enterprise Data Cloud event in Vienna. It includes welcome remarks, keynotes on Cloudera's vision and customer success stories. There will be presentations on the new Cloudera Data Platform and customer case studies, followed by closing remarks. The schedule includes sessions on Cloudera's approach to data warehousing, machine learning, streaming and multi-cloud capabilities.

Building Data Science Teams: A Moneyball Approachjoshwills

Big data oracle_introduccionFran Navarro

The document discusses how big data and analytics can transform businesses. It notes that the volume of data is growing exponentially due to increases in smartphones, sensors, and other data producing devices. It also discusses how businesses can leverage big data by capturing massive data volumes, analyzing the data, and having a unified and secure platform. The document advocates that businesses implement the four pillars of data management: mobility, in-memory technologies, cloud computing, and big data in order to reduce the gap between data production and usage.

巨量資料入門 The evolution of data architectureWei-Chiu Chuang

There’s been a great interest in applying Data Science and Machine Learning algorithms for the insight from data lately. Fundamentally, these techniques are only made practical by today’s scale-out Modern Data Architecture, which enables these algorithms to perform computation at massive scale economically. The focus of this talk is on the scale-out architecture, pioneered by Google. Initially a niche system architecture that is only applicable to Google, the systems built on top of the same principles are used in production across many industries, empowering enterprises to get better insights into their business, to become more agile and to do more things that were not possible previously. I will start the talk by providing the historical context behind the evolution of data architecture, and then dive into the technical details of the scale-out system, Hadoop and its ecosystem. Afterwards, I will present a few notable production use cases of the system. Finally, I will touch upon some of the exciting challenges and opportunities lying ahead for the future data architecture.

Stl meetup cloudera platform - january 2020Adam Doyle

The 5 Biggest Data Myths in Telco: ExposedCloudera, Inc.

The document discusses common myths in the telecommunications industry regarding big data and analytics. It addresses five myths: 1) that data is too diverse to analyze, 2) that open source means open security, 3) that big data platforms do not provide adequate return on investment, 4) that big data tools are too difficult for teams to learn, and 5) that legacy systems cannot handle additional data solutions. For each myth, it provides facts and examples to demonstrate why the myths are unfounded and how organizations can leverage big data to drive insights.

Optimize your cloud strategy for machine learning and analyticsCloudera, Inc.

Join industry superstars Mike Olson (Cloudera CSO and co-founder) and Jim Curtis (451 Research senior analyst) as they outline the best practices for cloud-based machine learning and analytics in this “can’t miss” webinar. Hot topics include: Why enterprises are moving their analytics to the public cloud How to select the best cloud deployment model Design tricks that make cloud economics work Success stories, cautionary tales, and lessons learned James will share 451 Research findings and offer insights learned from surveying both the vendor landscape and enterprise practitioners. . Mike will regale you with his vision for the future of multi-disciplinary machine learning and analytics in hybrid- and multi-cloud environments 3 things to learn: Why enterprises are moving their analytics to the public cloud How to select the best cloud deployment model Design tricks that make cloud economics work

Cloudera 助力台灣大數據產業的發展Etu Solution

This document discusses big data and Cloudera's Enterprise Data Hub solution. It begins by noting that big data is growing exponentially and now includes structured, complex, and diverse data types from various sources. Traditional data architectures using relational databases cannot effectively handle this scale and variety of big data. The document then introduces Cloudera's Hadoop-based Enterprise Data Hub as an open, scalable, and cost-effective platform that can ingest and process all data types and bring compute capabilities to the data. It provides an overview of Cloudera's history and product offerings that make up its full big data platform.

Creating your Center of Excellence (CoE) for data driven use casesFrank Vullers

The document discusses creating a data-driven culture and organization. It provides advice on building a data-driven culture, developing the right team and skills, adopting an agile approach, efficiently operationalizing insights, and implementing proper data governance. Specific recommendations include establishing executive sponsorship, advocating for data use, developing data science, engineering, and analytics teams, prioritizing work using agile methodologies, and communicating a business roadmap to operationalize insights.

Addressing Challenges with IoT Edge ManagementDataWorks Summit

With the explosive growth of IoT, the edge is predicted to grow to 25 billion connected devices by 2020. But, enterprises are still struggling to manage hundreds of devices that they have deployed. Not from a device management standpoint but more from a data management standpoint. Enterprises are unable to capture and process data directly from the edge devices for immediate analysis and gaining real-time actionable intelligence. So, if that is not possible, IoT initiatives are failing to become successful. How can an enterprise gather real-time data from edge devices? How can it change the behavior of such data collection processes? How can it ensure that data will be analyzed immediately? How can it understand the lineage of the data from edge to enterprise? How can it manage edge agents? What is an edge management hub? Attend this session to get a detailed understanding of key edge management challenges and how to address them with the correct solutions.

Spark in the Hadoop Ecosystem-(Mike Olson, Cloudera)Spark Summit

Big Data Day LA 2016/ Use Case Driven track - How to Use Design Thinking to J...Data Con LA

Next-Gen ML/AI PlatformJosh Yeh

The Vision & Challenge of Applied Machine LearningCloudera, Inc.

Analytics, Everywhere. Keys to Effective Analytics and Data DiscoveryDLT Solutions

Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.

Build a modern platform for anti-money laundering 9.19.18Cloudera, Inc.

Keynote: The Journey to Pervasive AnalyticsCloudera, Inc.

Technology Primer: Hey IT—Your Big Data Infrastructure Can’t Sit in a Silo An...CA Technologies

151116 Sedania Cloudera BDA ProfileZarul Zaabah

Edc event vienna presentation 1 oct 2019Cloudera, Inc.

Building Data Science Teams: A Moneyball Approachjoshwills

Big data oracle_introduccionFran Navarro

巨量資料入門 The evolution of data architectureWei-Chiu Chuang

Stl meetup cloudera platform - january 2020Adam Doyle

The 5 Biggest Data Myths in Telco: ExposedCloudera, Inc.

Optimize your cloud strategy for machine learning and analyticsCloudera, Inc.

Cloudera 助力台灣大數據產業的發展Etu Solution

Creating your Center of Excellence (CoE) for data driven use casesFrank Vullers

Addressing Challenges with IoT Edge ManagementDataWorks Summit

Spark in the Hadoop Ecosystem-(Mike Olson, Cloudera)Spark Summit

Big Data Day LA 2016/ Use Case Driven track - How to Use Design Thinking to J...Data Con LA

More from Wes McKinney (20)

The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney

The document discusses the future of composable data systems and provides an overview from Wes McKinney. Some key points: - Composable data systems are designed to be modular and reusable across different components through open standards and protocols. This allows new engines to be developed more easily. - The data landscape is shifting to an era of composability, where monolithic systems will be replaced by modular, reusable pieces. - Areas of focus for composable systems include execution engines, query interfaces, storage protocols, and optimization. - Projects like Apache Arrow, Ibis, Substrait, and modular engines like DuckDB, DataFusion, and Velox are moving the industry toward composability.

Solving Enterprise Data Challenges with Apache ArrowWes McKinney

This document discusses Apache Arrow, an open-source library that enables fast and efficient data interchange and processing. It summarizes the growth of Arrow and its ecosystem, including new features like the Arrow C++ query engine and Arrow Rust DataFusion. It also highlights how enterprises are using Arrow to solve challenges around data interoperability, access speed, query performance, and embeddable analytics. Case studies describe how companies like Microsoft, Google Cloud, Snowflake, and Meta leverage Arrow in their products and platforms. The presenter promotes Voltron Data's enterprise subscription and upcoming conference to support business use of Apache Arrow.

Apache Arrow: Open Source Standard Becomes an Enterprise NecessityWes McKinney

Apache Arrow: High Performance Columnar Data FrameworkWes McKinney

New Directions for Apache ArrowWes McKinney

- Apache Arrow is an open-source project that provides a shared data format and library for high performance data analytics across multiple languages. It aims to unify database and data science technology stacks. - In 2021, Ursa Labs joined forces with GPU-accelerated computing pioneers to form Voltron Data, continuing development of Apache Arrow and related projects like Arrow Flight and the Arrow R package. - Upcoming releases of the Arrow R package will bring additional query execution capabilities like joins and window functions to improve performance and efficiency of analytics workflows in R.

Apache Arrow Flight: A New Gold Standard for Data TransportWes McKinney

This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.

ACM TechTalks : Apache Arrow and the Future of Data FramesWes McKinney

Wes McKinney gave a talk on Apache Arrow and the future of data frames. He discussed how Arrow aims to standardize columnar data formats and reduce inefficiencies in data processing. It defines an efficient binary format for transferring data between systems and programming languages. As more tools support Arrow natively, it will become more efficient to process data directly in Arrow format rather than converting between data structures. Arrow is gaining adoption in popular data tools like Spark, BigQuery, and InfluxDB to improve performance.

Apache Arrow: Present and Future @ ScaledML 2020Wes McKinney

This document discusses Apache Arrow, an open source project that provides cross-language data structures and algorithms for efficient data analytics. It summarizes the history and goals of Arrow, provides examples of how it has been adopted, and outlines ongoing development initiatives. Key points include that Arrow aims to accelerate data processing by standardizing columnar data formats and protocols, it has seen widespread adoption with over 50M installs in 2019, and active areas of work include the C++ development platform and Arrow Flight RPC framework.

PyCon Colombia 2020 Python for Data Analysis: Past, Present, and Future Wes McKinney

Wes McKinney gave a presentation on the past, present, and future of Python for data analysis. He discussed the origins and development of pandas over the past 12 years from the first open source release in 2009 to the current state. Key points included pandas receiving its first formal funding in 2019, its large community of contributors, and factors driving Python's growth for data science like its package ecosystem and education. McKinney also addressed early concerns about Python and looked to the future, highlighting projects like Apache Arrow that aim to improve performance and interoperability.

Apache Arrow: Leveling Up the Analytics StackWes McKinney

This document discusses the development of Apache Arrow, an open source in-memory data format designed for efficient analytical data processing on modern hardware. It provides a brief history of big data and analytics technologies leading to the need for Arrow. Key points about Arrow include that it aims to eliminate data serialization, enable code sharing across languages, and has over 400 contributors representing 11 programming languages. Notable subcomponents include DataFusion, Gandiva, and Plasma; and development is supported by organizations like Ursa Labs.

Apache Arrow Workshop at VLDB 2019 / BOSS SessionWes McKinney

Apache Arrow: Leveling Up the Data Science StackWes McKinney

Ursa Labs builds cross-language libraries like Apache Arrow for data science. Arrow provides a columnar data format and utilities for efficient serialization, IO, and querying across programming languages. Ursa Labs contributes to Arrow and funds open source developers to grow the Arrow ecosystem. Their goal is to reduce the CPU time spent on data serialization and enable faster data analysis in languages like R.

Ursa Labs and Apache Arrow in 2019Wes McKinney

PyCon.DE / PyData Karlsruhe keynote: "Looking backward, looking forward"Wes McKinney

This document discusses the history and development of Python data analysis tools, including pandas. It covers Wes McKinney's work on pandas from 2008 to the present, including the motivations for making data analysis easier and more productive. It also summarizes the development of related projects like Apache Arrow for standardizing columnar data representations to improve code reuse across languages.

Apache Arrow at DataEngConf Barcelona 2018Wes McKinney

Wes McKinney is a leading open source developer who created Python's pandas library and now leads the Apache Arrow project. Apache Arrow is an open standard for in-memory analytics that aims to improve data sharing and reuse across systems by defining a common columnar data format and memory layout. It allows data to be accessed and algorithms to be reused across different programming languages with near-zero data copying. Arrow is being integrated into various data systems and is working to expand its computational libraries and language support.

Apache Arrow: Cross-language Development Platform for In-memory DataWes McKinney

Apache Arrow is an open standard for in-memory columnar data and an analytical data processing platform. It aims to simplify system architectures, improve interoperability between systems, and enable data and algorithms to be reused across different programming languages. Arrow provides a portable in-memory data format and computational libraries to build analytical data processing systems. It is language-independent and supports data sharing and algorithm reuse between libraries and processes via shared memory with near-zero overhead.

Apache Arrow -- Cross-language development platform for in-memory dataWes McKinney

Wes McKinney is the creator of Python's pandas project and a primary developer of Apache Arrow, Apache Parquet, and other open-source projects. Apache Arrow is an open-source cross-language development platform for in-memory analytics that aims to improve data science tools. It provides a shared standard for memory interoperability and computation across languages through its columnar memory format and libraries. Apache Arrow has growing adoption in data science systems and is working to expand language support and computational capabilities.

Shared Infrastructure for Data ScienceWes McKinney

Wes McKinney discussed the evolution of data science tools and infrastructure over the past 10 years and a vision for the next 10 years. He argued that current data science languages like Python, R, and Julia operate in "silos" with separate implementations for data storage, processing, and analytics. However, new projects like Apache Arrow aim to break down these silos by establishing shared standards for in-memory data formats and interchange that can unite the implementations across languages. Arrow provides a portable data frame format, zero-copy interchange capabilities, and potential for high performance data access and flexible computation engines. This would allow data science work to be more portable across programming languages while improving performance.

Data Science Without Borders (JupyterCon 2017)Wes McKinney

Memory Interoperability in Analytics and Machine LearningWes McKinney

Wes McKinney gave a talk on Apache Arrow, an open source project for memory interoperability between analytics and machine learning systems. Arrow provides efficient columnar memory structures and zero-copy sharing of data between applications. It defines common data types and schemas that can be used across programming languages. Arrow is implemented in C++ and provides language bindings for other languages like Python. It aims to improve performance for tasks like data loading, preprocessing, modeling and serving. Projects like pandas, Spark and Ray are exploring using Arrow internally for more efficient data handling.

The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney

Solving Enterprise Data Challenges with Apache ArrowWes McKinney

Apache Arrow: Open Source Standard Becomes an Enterprise NecessityWes McKinney

Apache Arrow: High Performance Columnar Data FrameworkWes McKinney

New Directions for Apache ArrowWes McKinney

Apache Arrow Flight: A New Gold Standard for Data TransportWes McKinney

ACM TechTalks : Apache Arrow and the Future of Data FramesWes McKinney

Apache Arrow: Present and Future @ ScaledML 2020Wes McKinney

PyCon Colombia 2020 Python for Data Analysis: Past, Present, and Future Wes McKinney

Apache Arrow: Leveling Up the Analytics StackWes McKinney

Apache Arrow Workshop at VLDB 2019 / BOSS SessionWes McKinney

Apache Arrow: Leveling Up the Data Science StackWes McKinney

Ursa Labs and Apache Arrow in 2019Wes McKinney

PyCon.DE / PyData Karlsruhe keynote: "Looking backward, looking forward"Wes McKinney

Apache Arrow at DataEngConf Barcelona 2018Wes McKinney

Apache Arrow: Cross-language Development Platform for In-memory DataWes McKinney

Apache Arrow -- Cross-language development platform for in-memory dataWes McKinney

Shared Infrastructure for Data ScienceWes McKinney

Data Science Without Borders (JupyterCon 2017)Wes McKinney

Memory Interoperability in Analytics and Machine LearningWes McKinney

Recently uploaded (20)

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts! 📕 Agenda Welcome & Introductions Orchestrator API Overview Exploring the Swagger Interface Test Manager API Highlights Streamlining Automation & Testing with APIs (Demo) Q&A and Open Discussion Perfect for developers, testers, and automation enthusiasts! 👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/ This session streamed live on April 29, 2025, 18:00 CET. Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how: • Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules. • Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance. • Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity. • Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications. • Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market. With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications. Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/ HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client. Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience. In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including - Accessing the console - Locating and interpreting log files - Accessing the data folder within the browser’s cache (using OPFS) - Understand the difference between single- and multi-user scenarios - Utilizing Client Clocking

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Procurement Insights Cost To Value Guide.pptxJon Hansen

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Cyber Awareness overview for 2025 month of securityriccardosl1

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Splunk Security Update | Public Sector Summit Germany 2025Splunk

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

AI and Data Privacy in 2025: Global TrendsInData Labs

In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy. This infographic contains: -AI and data privacy: Key findings -Statistics on AI data privacy in the today’s world -Tips on how to overcome data privacy challenges -Benefits of AI data security investments. Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies