SlideShare a Scribd company logo
Introduction to
NetGuardians’
Big Data
Software Stack
Jerome Kehrli, Head of R&D
Geneva, September 2017
Agenda
• Introducing NetGuardians
• Software Stack
• Typical Architecture
• NetGuardians’ Use Cases
• ElasticSearch / Spark / Mesos
Constraints and Behaviour
About NetGuardians
• Top Fintech Europe Company
• Behavioural analysis based on risk
models combining human actions
relative to channels, technical
layers and transactions.
• Stay on top of new regulatory
needs and anti-fraud patterns
using profiling and analytics
• Our intelligence updates
automatically deliver new controls
XXXXXX XXX
E-BANKINGE-BANKING
IT layers
Transactions
Channels
The Problem
70% is internal
Fraud costs the world
$3trillion per year
Certified Fraud Examiners,
Report to the Nations, 2014
$6 trillion
Projected cyber crime
cost by 2021
Cyber Security Ventures, 2016
It takes 18 months on average
to detect fraud.
Most remains undetected.
Certified Fraud Examiners, Report to the
Nations, 2014
$6
trillion
$3
trillion
The fine
one single bank was slapped
with due to inadequate
internal controls and slow
documentation process
Bloomberg, April 2015
$2.5
billion
All the caps you need
One single platform
Unique solution made for banks
All the caps you need
One single platform
Unique solution made for banks
References
Retail banking
Private banking
Scalable Big Data Technology
Analytics Platform
Software Stack
Mesos is a distributed systems kernel.
Runs on every machine and provides applications (…) with
API’s for resource management and scheduling across
entire datacenter and cloud environments.
Apache Spark is a fast and general engine for large-scale
data processing.
Provides programmers with an API functioning as a working
set for distributed programs that offers a versatile form of
distributed shared memory.
ElasticSearch is a distributed, real-time, RESTful search and
analytics document-oriented storage engine.
Lets one perform and combine many types of searches -
structured, unstructured, geo, metric - in real time.
Apache
(V1.3 = July 2017
V1.0 = July 2016)
Apache
(V2.2 = July 2017
V1.0 = May 2014)
ElasticSearch
(V6.0b = July 2017
V1.0 = February 2014)
ES-Hadoop : connect the massive data storage and deep
processing power of Hadoop with the real-time search and
analytics of Elasticsearch.
Interestingly, Spark can perfectly use ES-Hadoop to load from
or store data to ElasticSearch outside of an Hadoop stack.
The spark connector from the ES-Hadoop library has no
dependency on a Hadoop stack whatsoever.
ES-Hadoop
ES
ELK-MS
Architecture
ELK-MS - Technical Architecture
ELK-MS - System Architecture
ELK-MS - Typical Application Architecture
NetGuardians
Use Cases
Analytics approach
Pattern Based Intelligence
• Fundamentally rule based
• Implemented as pyspark scripts
• Custom approach (no framework)
Profiling
• Statistical Model
• Natively implemented using both
ES and spark statistics functions
• Custom approach (no framework)
Machine Learning
• Advanced algorithms
• Prototyped using Python SciKit
learn
• Industrialized using Spark MLlib
Typical Data Flow
Data-locality optimization is not optional for us !
ES / Spark / Mesos
Constraints and
behaviour
ES-Hadoop and Data Locality
Data-locality enforcement works well.
• ES-Hadoop makes Spark understand the
topology of the shards on ES
• Mesos / Spark respects locality requirements,
creates as many partitions as shards.
It works only under nominal conditions.
Several factors compromise data-locality:
→ Spark waits only for
spark.locality.wait=10s trying to get the
processing executed on the spark node co-
located to an ES shard
← If ES on co-located node is busy, ES can decide
to answer from another node
Mesos / Spark Scheduling Mode
In Coarse Grained scheduling mode, Mesos only
knows spark executor processes.
• Mesos books as much cluster resources as
possible to allocate Spark executors for a job.
Historically, Mesos on Spark can use Fine
Grained scheduling mode, where Mesos
schedules each and every individual spark task.
• Kills performances !
• Deprecated:
https://issues.apache.org/jira/browse/SPARK
-11857
Spark Static Resource Allocation vs. Dynamic Allocation (1/2)
Static Resource Allocation
• Mesos / Spark decides allocated resources at
job init time
• Allocated resources are kept until the job
completes
• 2 noteworthy consequences :
1. By default, every single job running
alone gets the whole cluster.
A following job would need to wait.
2. Several jobs arriving together would get
the cluster fairly shared.
If only one job is long-lived, that job
would still need to complete its
execution on his small portion.
Spark Static Resource Allocation vs. Dynamic Allocation (2/2)
Dynamic Allocation
• Designed as a solution the previous problems
• But … Spark‘s Dynamic Allocation messes up
data locality optimization completely.
• ES-Hadoop makes spark request as many
executors as shards and indicates
as preferred location the nodes owning the
ES shards.
• Dynamic allocation bypasses this
completely and screws data-locality
optimization
Dynamic Allocation
• Designed as a solution the previous problems
• Works out of the Box
Other concerns
• Python latency
• Java and Scala jobs run natively in the Spark JVM.
• Pyspark launches “some tasks” in a separate process than the Spark JVM.
• DataFrame or RDD methods exposed to python scripts are actually implemented in native
Scala underneath.
• One noticeable exception: UDF (User Defined functions) implemented in python!
• One can very well still use pyspark but write UDF in Scala.
• Repartitioning
• A redistribution of a dataset on the cluster is only hardly achievable … and not necessarily
desirable.
• Advanced ES queries
• The ES-Hadoop connector can only submit “simple” requests to ES, with filtering (now)
• Advanced features such as aggregation queries cannot be used
ES / Spark / Mesos
Why is it cool ?
Why cool ? (1/5)
Spark’s API is brilliant for our use cases (NetGuardians)
Pattern Based Intelligence
• Implementing our rules in pyspark
is straightforward
• We are now considering DRESS on
spark streaming
Profiling
• Out of the box with Spark’s
statistics functions
• Here as well we consider spark
streaming for event scoring
Machine Learning
• We prototype with Python SciKit
Learn
• Implementation on spark is easy
with Spark MLlib
Why cool ? (2/5)
What do we want ?
Initial situation
Why cool ? (2/5)
What do we want ?
Working with a small
subset of the data
Why cool ? (2/5)
What do we want ?
Working with a full
month of data
Why cool ? (2/5)
What do we want ?
Working with the
whole dataset
Why cool ? (3/5)
Processing Distribution scaling linearly with Data Distribution
Works Out of the box with
• Dynamic Allocation in Spark + Mesos
• ES-Hadoop / ES-Spark connector data locality optimization
Why cool ? (4/5)
Processing Distribution scaling linearly with Data Distribution
ES / Spark / Mesos provide the basic building blocks to distribute
and scale the processing exactly how we want
• ES-Hadoop : Data locality optimization
• Mesos / Spark : spark.cores.max=X configuration
• ElasticSearch : search_shards API
Golden Rule : use spark.core.max = Nbr Shards
Why cool ? (5/5)
“One ring to rule them all ...”
• ES, Spark and Mesos are
designed to run on large clusters
• But they work very well as well
on one single fat machine with
tons of CPUs and RAM
• We deploy the same platform in
tier 1 banks and small banks.
THANK YOU!
NetGuardians SA Headquarters
Rue Galilée 6
1400 Yverdon-les-Bains
Switzerland
Tel: +41 24 425 97 60
Email: info@netguardians.ch
www.netguardians.ch
Linkedin.com/company/netguardians
Facebook.com/NetGuardians
@netguardians

More Related Content

What's hot (20)

PDF
02 a holistic approach to big data
Raul Chong
 
PDF
Understanding Big Data Analytics - solutions for growing businesses - Rafał M...
GetInData
 
PDF
DAMA & Denodo Webinar: Modernizing Data Architecture Using Data Virtualization
Denodo
 
PPTX
ParStream - Big Data for Business Users
ParStream Inc.
 
PDF
A Successful Data Strategy for Insurers in Volatile Times (ASEAN)
Denodo
 
PDF
Transforming GE Healthcare with Data Platform Strategy
Databricks
 
PDF
Data engineering design patterns
Valdas Maksimavičius
 
PDF
Future of Data Strategy (ASEAN)
Denodo
 
PDF
What is the future of data strategy?
Denodo
 
PDF
Smart data for a predictive bank
DataWorks Summit/Hadoop Summit
 
PDF
Msst 2019 v4
Nisha Talagala
 
PDF
Stephen Cantrell, kdb+ Developer at Kx Systems “Kdb+: How Wall Street Tech c...
Dataconomy Media
 
PDF
Strategyzing big data in telco industry
Parviz Iskhakov
 
PDF
Accelerate Self-Service Analytics with Data Virtualization and Visualization
Denodo
 
PPT
Information Technology
Sahil Mahajan
 
PPTX
San Antonio’s electric utility making big data analytics the business of the ...
DataWorks Summit
 
PDF
Moving Targets: Harnessing Real-time Value from Data in Motion
Inside Analysis
 
PDF
End to End Supply Chain Control Tower
Databricks
 
PPTX
Scaling Face Recognition with Big Data
Bogdan Bocse
 
PPTX
Monitizing Big Data at Telecom Service Providers
DataWorks Summit
 
02 a holistic approach to big data
Raul Chong
 
Understanding Big Data Analytics - solutions for growing businesses - Rafał M...
GetInData
 
DAMA & Denodo Webinar: Modernizing Data Architecture Using Data Virtualization
Denodo
 
ParStream - Big Data for Business Users
ParStream Inc.
 
A Successful Data Strategy for Insurers in Volatile Times (ASEAN)
Denodo
 
Transforming GE Healthcare with Data Platform Strategy
Databricks
 
Data engineering design patterns
Valdas Maksimavičius
 
Future of Data Strategy (ASEAN)
Denodo
 
What is the future of data strategy?
Denodo
 
Smart data for a predictive bank
DataWorks Summit/Hadoop Summit
 
Msst 2019 v4
Nisha Talagala
 
Stephen Cantrell, kdb+ Developer at Kx Systems “Kdb+: How Wall Street Tech c...
Dataconomy Media
 
Strategyzing big data in telco industry
Parviz Iskhakov
 
Accelerate Self-Service Analytics with Data Virtualization and Visualization
Denodo
 
Information Technology
Sahil Mahajan
 
San Antonio’s electric utility making big data analytics the business of the ...
DataWorks Summit
 
Moving Targets: Harnessing Real-time Value from Data in Motion
Inside Analysis
 
End to End Supply Chain Control Tower
Databricks
 
Scaling Face Recognition with Big Data
Bogdan Bocse
 
Monitizing Big Data at Telecom Service Providers
DataWorks Summit
 

Similar to Introduction to NetGuardians' Big Data Software Stack (20)

PDF
DoneDeal - AWS Data Analytics Platform
martinbpeters
 
PPTX
From Pipelines to Refineries: scaling big data applications with Tim Hunter
Databricks
 
PPTX
Pyspark presentationsfspfsjfspfjsfpsjfspfjsfpsjfsfsf
sasuke20y4sh
 
PDF
Big Telco Real-Time Network Analytics
Yousun Jeong
 
PDF
Big Telco - Yousun Jeong
Spark Summit
 
PPTX
PEARC 17: Spark On the ARC
Himanshu Bedi
 
PPTX
Advanced Visualization of Spark jobs
DataWorks Summit/Hadoop Summit
 
PDF
Deep Learning on Apache® Spark™ : Workflows and Best Practices
Jen Aman
 
PDF
Deep Learning on Apache® Spark™: Workflows and Best Practices
Databricks
 
PDF
Deep Learning on Apache® Spark™: Workflows and Best Practices
Jen Aman
 
PDF
DataOps with Project Amaterasu
DataWorks Summit/Hadoop Summit
 
PPTX
Apache Spark in Scientific Applciations
Dr. Mirko Kämpf
 
PPTX
Apache Spark in Scientific Applications
Dr. Mirko Kämpf
 
PDF
SnappyData Toronto Meetup Nov 2017
SnappyData
 
PDF
Spark
Nitish Upreti
 
PDF
Fast Data Analytics with Spark and Python
Benjamin Bengfort
 
PPTX
Big Data Processing with Apache Spark 2014
mahchiev
 
PDF
Build Deep Learning Applications for Big Data Platforms (CVPR 2018 tutorial)
Jason Dai
 
PPTX
Apache Spark Fundamentals
Zahra Eskandari
 
PDF
Sa introduction to big data pipelining with cassandra & spark west mins...
Simon Ambridge
 
DoneDeal - AWS Data Analytics Platform
martinbpeters
 
From Pipelines to Refineries: scaling big data applications with Tim Hunter
Databricks
 
Pyspark presentationsfspfsjfspfjsfpsjfspfjsfpsjfsfsf
sasuke20y4sh
 
Big Telco Real-Time Network Analytics
Yousun Jeong
 
Big Telco - Yousun Jeong
Spark Summit
 
PEARC 17: Spark On the ARC
Himanshu Bedi
 
Advanced Visualization of Spark jobs
DataWorks Summit/Hadoop Summit
 
Deep Learning on Apache® Spark™ : Workflows and Best Practices
Jen Aman
 
Deep Learning on Apache® Spark™: Workflows and Best Practices
Databricks
 
Deep Learning on Apache® Spark™: Workflows and Best Practices
Jen Aman
 
DataOps with Project Amaterasu
DataWorks Summit/Hadoop Summit
 
Apache Spark in Scientific Applciations
Dr. Mirko Kämpf
 
Apache Spark in Scientific Applications
Dr. Mirko Kämpf
 
SnappyData Toronto Meetup Nov 2017
SnappyData
 
Fast Data Analytics with Spark and Python
Benjamin Bengfort
 
Big Data Processing with Apache Spark 2014
mahchiev
 
Build Deep Learning Applications for Big Data Platforms (CVPR 2018 tutorial)
Jason Dai
 
Apache Spark Fundamentals
Zahra Eskandari
 
Sa introduction to big data pipelining with cassandra & spark west mins...
Simon Ambridge
 
Ad

More from Jérôme Kehrli (18)

PPTX
Introduction to Operating Systems
Jérôme Kehrli
 
PPTX
Introduction to Modern Software Architecture
Jérôme Kehrli
 
PDF
A proposed framework for Agile Roadmap Design and Maintenance
Jérôme Kehrli
 
PPTX
The search for Product-Market Fit
Jérôme Kehrli
 
PPTX
Big data in Private Banking
Jérôme Kehrli
 
PPTX
From Product Vision to Story Map - Lean / Agile Product shaping
Jérôme Kehrli
 
PPTX
Artificial Intelligence and Digital Banking - What about fraud prevention ?
Jérôme Kehrli
 
PPTX
Artificial Intelligence for Banking Fraud Prevention
Jérôme Kehrli
 
PPTX
Linux and Java - Understanding and Troubleshooting
Jérôme Kehrli
 
PPTX
Deciphering the Bengladesh bank heist
Jérôme Kehrli
 
PPTX
Periodic Table of Agile Principles and Practices
Jérôme Kehrli
 
PPTX
Agility and planning : tools and processes
Jérôme Kehrli
 
PPTX
Bytecode manipulation with Javassist for fun and profit
Jérôme Kehrli
 
PPTX
DevOps explained
Jérôme Kehrli
 
PPTX
Digitalization: A Challenge and An Opportunity for Banks
Jérôme Kehrli
 
PPTX
Lean startup
Jérôme Kehrli
 
PPTX
Blockchain 2.0
Jérôme Kehrli
 
PPTX
The Blockchain - The Technology behind Bitcoin
Jérôme Kehrli
 
Introduction to Operating Systems
Jérôme Kehrli
 
Introduction to Modern Software Architecture
Jérôme Kehrli
 
A proposed framework for Agile Roadmap Design and Maintenance
Jérôme Kehrli
 
The search for Product-Market Fit
Jérôme Kehrli
 
Big data in Private Banking
Jérôme Kehrli
 
From Product Vision to Story Map - Lean / Agile Product shaping
Jérôme Kehrli
 
Artificial Intelligence and Digital Banking - What about fraud prevention ?
Jérôme Kehrli
 
Artificial Intelligence for Banking Fraud Prevention
Jérôme Kehrli
 
Linux and Java - Understanding and Troubleshooting
Jérôme Kehrli
 
Deciphering the Bengladesh bank heist
Jérôme Kehrli
 
Periodic Table of Agile Principles and Practices
Jérôme Kehrli
 
Agility and planning : tools and processes
Jérôme Kehrli
 
Bytecode manipulation with Javassist for fun and profit
Jérôme Kehrli
 
DevOps explained
Jérôme Kehrli
 
Digitalization: A Challenge and An Opportunity for Banks
Jérôme Kehrli
 
Lean startup
Jérôme Kehrli
 
Blockchain 2.0
Jérôme Kehrli
 
The Blockchain - The Technology behind Bitcoin
Jérôme Kehrli
 
Ad

Recently uploaded (20)

PPT
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
PPTX
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
PDF
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
PPTX
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
PDF
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PPTX
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
PPTX
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
PPTX
Agentforce World Tour Toronto '25 - Supercharge MuleSoft Development with Mod...
Alexandra N. Martinez
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
PDF
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
DOCX
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PPTX
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
Agentforce World Tour Toronto '25 - Supercharge MuleSoft Development with Mod...
Alexandra N. Martinez
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 

Introduction to NetGuardians' Big Data Software Stack

  • 1. Introduction to NetGuardians’ Big Data Software Stack Jerome Kehrli, Head of R&D Geneva, September 2017
  • 2. Agenda • Introducing NetGuardians • Software Stack • Typical Architecture • NetGuardians’ Use Cases • ElasticSearch / Spark / Mesos Constraints and Behaviour
  • 3. About NetGuardians • Top Fintech Europe Company • Behavioural analysis based on risk models combining human actions relative to channels, technical layers and transactions. • Stay on top of new regulatory needs and anti-fraud patterns using profiling and analytics • Our intelligence updates automatically deliver new controls XXXXXX XXX E-BANKINGE-BANKING IT layers Transactions Channels
  • 4. The Problem 70% is internal Fraud costs the world $3trillion per year Certified Fraud Examiners, Report to the Nations, 2014 $6 trillion Projected cyber crime cost by 2021 Cyber Security Ventures, 2016 It takes 18 months on average to detect fraud. Most remains undetected. Certified Fraud Examiners, Report to the Nations, 2014 $6 trillion $3 trillion The fine one single bank was slapped with due to inadequate internal controls and slow documentation process Bloomberg, April 2015 $2.5 billion
  • 5. All the caps you need One single platform Unique solution made for banks
  • 6. All the caps you need One single platform Unique solution made for banks
  • 8. Scalable Big Data Technology
  • 10. Mesos is a distributed systems kernel. Runs on every machine and provides applications (…) with API’s for resource management and scheduling across entire datacenter and cloud environments. Apache Spark is a fast and general engine for large-scale data processing. Provides programmers with an API functioning as a working set for distributed programs that offers a versatile form of distributed shared memory. ElasticSearch is a distributed, real-time, RESTful search and analytics document-oriented storage engine. Lets one perform and combine many types of searches - structured, unstructured, geo, metric - in real time. Apache (V1.3 = July 2017 V1.0 = July 2016) Apache (V2.2 = July 2017 V1.0 = May 2014) ElasticSearch (V6.0b = July 2017 V1.0 = February 2014)
  • 11. ES-Hadoop : connect the massive data storage and deep processing power of Hadoop with the real-time search and analytics of Elasticsearch. Interestingly, Spark can perfectly use ES-Hadoop to load from or store data to ElasticSearch outside of an Hadoop stack. The spark connector from the ES-Hadoop library has no dependency on a Hadoop stack whatsoever. ES-Hadoop ES
  • 13. ELK-MS - Technical Architecture
  • 14. ELK-MS - System Architecture
  • 15. ELK-MS - Typical Application Architecture
  • 17. Analytics approach Pattern Based Intelligence • Fundamentally rule based • Implemented as pyspark scripts • Custom approach (no framework) Profiling • Statistical Model • Natively implemented using both ES and spark statistics functions • Custom approach (no framework) Machine Learning • Advanced algorithms • Prototyped using Python SciKit learn • Industrialized using Spark MLlib
  • 18. Typical Data Flow Data-locality optimization is not optional for us !
  • 19. ES / Spark / Mesos Constraints and behaviour
  • 20. ES-Hadoop and Data Locality Data-locality enforcement works well. • ES-Hadoop makes Spark understand the topology of the shards on ES • Mesos / Spark respects locality requirements, creates as many partitions as shards. It works only under nominal conditions. Several factors compromise data-locality: → Spark waits only for spark.locality.wait=10s trying to get the processing executed on the spark node co- located to an ES shard ← If ES on co-located node is busy, ES can decide to answer from another node
  • 21. Mesos / Spark Scheduling Mode In Coarse Grained scheduling mode, Mesos only knows spark executor processes. • Mesos books as much cluster resources as possible to allocate Spark executors for a job. Historically, Mesos on Spark can use Fine Grained scheduling mode, where Mesos schedules each and every individual spark task. • Kills performances ! • Deprecated: https://issues.apache.org/jira/browse/SPARK -11857
  • 22. Spark Static Resource Allocation vs. Dynamic Allocation (1/2) Static Resource Allocation • Mesos / Spark decides allocated resources at job init time • Allocated resources are kept until the job completes • 2 noteworthy consequences : 1. By default, every single job running alone gets the whole cluster. A following job would need to wait. 2. Several jobs arriving together would get the cluster fairly shared. If only one job is long-lived, that job would still need to complete its execution on his small portion.
  • 23. Spark Static Resource Allocation vs. Dynamic Allocation (2/2) Dynamic Allocation • Designed as a solution the previous problems • But … Spark‘s Dynamic Allocation messes up data locality optimization completely. • ES-Hadoop makes spark request as many executors as shards and indicates as preferred location the nodes owning the ES shards. • Dynamic allocation bypasses this completely and screws data-locality optimization Dynamic Allocation • Designed as a solution the previous problems • Works out of the Box
  • 24. Other concerns • Python latency • Java and Scala jobs run natively in the Spark JVM. • Pyspark launches “some tasks” in a separate process than the Spark JVM. • DataFrame or RDD methods exposed to python scripts are actually implemented in native Scala underneath. • One noticeable exception: UDF (User Defined functions) implemented in python! • One can very well still use pyspark but write UDF in Scala. • Repartitioning • A redistribution of a dataset on the cluster is only hardly achievable … and not necessarily desirable. • Advanced ES queries • The ES-Hadoop connector can only submit “simple” requests to ES, with filtering (now) • Advanced features such as aggregation queries cannot be used
  • 25. ES / Spark / Mesos Why is it cool ?
  • 26. Why cool ? (1/5) Spark’s API is brilliant for our use cases (NetGuardians) Pattern Based Intelligence • Implementing our rules in pyspark is straightforward • We are now considering DRESS on spark streaming Profiling • Out of the box with Spark’s statistics functions • Here as well we consider spark streaming for event scoring Machine Learning • We prototype with Python SciKit Learn • Implementation on spark is easy with Spark MLlib
  • 27. Why cool ? (2/5) What do we want ? Initial situation
  • 28. Why cool ? (2/5) What do we want ? Working with a small subset of the data
  • 29. Why cool ? (2/5) What do we want ? Working with a full month of data
  • 30. Why cool ? (2/5) What do we want ? Working with the whole dataset
  • 31. Why cool ? (3/5) Processing Distribution scaling linearly with Data Distribution Works Out of the box with • Dynamic Allocation in Spark + Mesos • ES-Hadoop / ES-Spark connector data locality optimization
  • 32. Why cool ? (4/5) Processing Distribution scaling linearly with Data Distribution ES / Spark / Mesos provide the basic building blocks to distribute and scale the processing exactly how we want • ES-Hadoop : Data locality optimization • Mesos / Spark : spark.cores.max=X configuration • ElasticSearch : search_shards API Golden Rule : use spark.core.max = Nbr Shards
  • 33. Why cool ? (5/5) “One ring to rule them all ...” • ES, Spark and Mesos are designed to run on large clusters • But they work very well as well on one single fat machine with tons of CPUs and RAM • We deploy the same platform in tier 1 banks and small banks.
  • 34. THANK YOU! NetGuardians SA Headquarters Rue Galilée 6 1400 Yverdon-les-Bains Switzerland Tel: +41 24 425 97 60 Email: [email protected] www.netguardians.ch Linkedin.com/company/netguardians Facebook.com/NetGuardians @netguardians

Editor's Notes

  • #2: Disclaimer Je ne vais pas faire une intro à big data, je pars du principe que l’audience est familiarisée à Big Data (Moving processing to the data nodes, distribution of the form of partitioning and replicating, etc.) Je vais me concentrer sur la spécificité de la stack technologique chez NetGuardians
  • #4: Our intelligent software platform will give you a greater capability to detect the emerging insider fraud & risk threats, delivering ROI straight away.
  • #11: OK
  • #28: !! Préparer l’explication !!!
  • #29: !! Préparer l’explication !!!
  • #30: !! Préparer l’explication !!!
  • #31: !! Préparer l’explication !!!
  • #35: 27 -> 20 !!!! Présentation NetGuardians “un peu” plus courte Plus de détail, allez voir notre site web !! Vocabulaire : Big data = distribution = partitionnement (sharding) et réplication ! Les 3 slides super-textuels !!! “Vous aurez les slides ….” Juste préparer un résumé …..