SlideShare a Scribd company logo
The role of Dremio in a Data Mesh
architecture
Presented by: Paolo Platter – CTO & Co-founder @ Agile Lab
Who we are :
• We value transparency, collaboration and results
• Totally decentralized and self-managed
• International culture and mindset
• Customer laser focused
What we do:
• Data Engineering is our mission since 2013
• Crafting end-to-end data platforms
• Data Strategy
• Managed Data Service
www.agilelab.it
Data Mesh Principles
Domain Driven
Data Ownership
Architecture
Data as a product
Self-Serve
Infrastructure as a
Platform
Federated
Computational
Governance
Data Product
Data + Metadata
(syntax+semantic, expected behaviour, access control )
Infrastructure
Data Pipeline Data Access API
Observability API
Internal processes
(GDPR, DQ, etc )
Stream Processing
Information
API
Control Ports
code
data
Input Ports
• Operational systems
• Other Data Products
• External services
Output Ports
• Events
• SQLView
• Raw/Files
• Graph/RDF
infrastructure
Technology Independence
• Addressability
• Interoperability
• Self-Serve provisioning
• Independently deployable
• Data Mesh is a practice
• Each Data Product team can select the
technology that best fits the use case.
• The technology must be compliant with Data
Product features and requirements
• Multi-cloud needs
Output Ports
Output Port API
Data Consumer
• Descriptive schema
• Audit
• Access Control
• Decoupling(uri and protocol)
• SLO
Read data through native protocol
Events
SQL
Files
Output Port API
Data Consumer
• GraphQL or HTTP
• Data is flowing throught API
• Zero coupling, low performances
• Not suitable for all use cases of data consumption
Events
SQL
Files
• Data is flowing throught native protocol
• Low coupling, good performances
• Fully Polyglot
Pre-flight
www.agilelab.it
Problem
Connecting a BI tool to an Output Port
BI Tool
SQL
Files
Output Port API
GraphQL or other HTTP based protocols are not
widely supported by BI Tools.
Also thinking to have a custom pre-flight and
dynamically discover the protocol of the source is
something not easy
In order to query directly a file/object storage you
need a SQL Engine, tipically not available inside BI
Tools
JDBC/ODBC connection is a good and standard
option for BI Tools, but this is hiding problems
Client-side coupling is not good
Consumer
JDBC
driver
Athena
JDBC
driver
Redshift
JDBC
driver
Aurora
• One driver doesn’t fit all
• Coupling is becoming a problem for change management
• DP is not indipendently deployable
Resons why you need multiple technologies
in a data mesh:
- Not all the use cases fit with a single tech
- Data Mesh is an evolutionary
architecture, technologies will evolve and
change over time and DPs will adopt
them indipendently
- Your data mesh is expanding on a multi-
cloud landscape
How to integrate legacy systems
Data Mesh Data Lake
Migration will require time...
Consumer
j
d
b
c
jdbc
What if we need to join data coming from different
JDBC channels ?
Huge impact on performances
Join must be resolved
at this level
www.agilelab.it
Solution
Dremio Uniqueness
SQL Execution Engine
OLAP Accelleration
Virtualization
Extensible
Semantic Layer Scalable Cache
Cloud & Tech Agnostic
Flexible & Self Serve
deployment
Fitting into the big picture
Native integration
with data lakes
Bridge also other
enteprise assets
Data Consumer
• Single interface to access all the silos and no coupling
between data consumers and multiple specific technologies
• Single catalog of data
You can use it
as SQL Query
Engine inside
Data Products
Interfacing
other DBMS
Cloud agnostic
• Efficient join between DPs leveraging different underlying
technologies
• Query federation between data mesh and other data assets
across the organization
• Native integration with DataLakes to facilitate the transition to
the DataMesh
Self-Serve provisioning
API
Catalog
Execution Engine
ACL
CI/CD
Data Product
specification
Provision:
• Execution resources
• SQL View
• ACL
Provision:
• Storage
• DB
• Policies
• IAM
• etc
Deploy
Data-Product Caching
Main Entity Pre-aggregated
and
denormalized
views
Query Acceleration & Caching
Leverage external reflections to speed-up
queries automatically without adding
complexity to the data consumer
Query
Dremio can create such pre-aggregation
automatically without the need to implement
custom jobs for such purpose
Data Consumer
Data Consumer interacts with a single logical entity, but queries
will speed-up due to the cache and reflections.
Thank You! – Q&A Time
Contact me at:
paolo.platter@agilelab.it
https://www.agilelab.it/data-mesh-in-action/
Ad

More Related Content

Similar to The role of Dremio in a data mesh architecture (20)

Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)
Denodo
 
Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI
Jade Global
 
A Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data VirtualizationA Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data Virtualization
Denodo
 
Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...
Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...
Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...
APPSeCONNECT
 
Vue d'ensemble Dremio
Vue d'ensemble DremioVue d'ensemble Dremio
Vue d'ensemble Dremio
Modern Data Stack France
 
Logical Data Fabric and Data Mesh – Driving Business Outcomes
Logical Data Fabric and Data Mesh – Driving Business OutcomesLogical Data Fabric and Data Mesh – Driving Business Outcomes
Logical Data Fabric and Data Mesh – Driving Business Outcomes
Denodo
 
Speak to Your Data
Speak to Your DataSpeak to Your Data
Speak to Your Data
Amer Radwan , PMP , CSM
 
Lecture 5- Data Collection and Storage.pptx
Lecture 5- Data Collection and Storage.pptxLecture 5- Data Collection and Storage.pptx
Lecture 5- Data Collection and Storage.pptx
Brianc34
 
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Igor De Souza
 
Flash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lonFlash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lon
Jeffrey T. Pollock
 
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Modernizing Global Shared Data Analytics Platform and our Alluxio JourneyModernizing Global Shared Data Analytics Platform and our Alluxio Journey
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Alluxio, Inc.
 
Introduction to Data Engineering
Introduction to Data EngineeringIntroduction to Data Engineering
Introduction to Data Engineering
Durga Gadiraju
 
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
SnapLogic
 
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Denodo
 
oracle.pptx
oracle.pptxoracle.pptx
oracle.pptx
Minakshee Patil
 
Building Blocks for Hybrid IT
Building Blocks for Hybrid ITBuilding Blocks for Hybrid IT
Building Blocks for Hybrid IT
RightScale
 
Data virtualization
Data virtualizationData virtualization
Data virtualization
Hamed Hatami
 
BPM and SOA are going mobile - An architectural perspective
BPM and SOA are going mobile - An architectural perspectiveBPM and SOA are going mobile - An architectural perspective
BPM and SOA are going mobile - An architectural perspective
OPITZ CONSULTING Deutschland
 
BPM und SOA machen mobil - Ein Architekturüberblick
BPM und SOA machen mobil - Ein ArchitekturüberblickBPM und SOA machen mobil - Ein Architekturüberblick
BPM und SOA machen mobil - Ein Architekturüberblick
OPITZ CONSULTING Deutschland
 
Break Free from Oracle
Break Free from OracleBreak Free from Oracle
Break Free from Oracle
EDB
 
Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)
Denodo
 
Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI
Jade Global
 
A Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data VirtualizationA Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data Virtualization
Denodo
 
Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...
Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...
Webinar: Out of the Box Features of an iPaaS - Cloud Integration Platform as ...
APPSeCONNECT
 
Logical Data Fabric and Data Mesh – Driving Business Outcomes
Logical Data Fabric and Data Mesh – Driving Business OutcomesLogical Data Fabric and Data Mesh – Driving Business Outcomes
Logical Data Fabric and Data Mesh – Driving Business Outcomes
Denodo
 
Lecture 5- Data Collection and Storage.pptx
Lecture 5- Data Collection and Storage.pptxLecture 5- Data Collection and Storage.pptx
Lecture 5- Data Collection and Storage.pptx
Brianc34
 
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Igor De Souza
 
Flash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lonFlash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lon
Jeffrey T. Pollock
 
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Modernizing Global Shared Data Analytics Platform and our Alluxio JourneyModernizing Global Shared Data Analytics Platform and our Alluxio Journey
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Alluxio, Inc.
 
Introduction to Data Engineering
Introduction to Data EngineeringIntroduction to Data Engineering
Introduction to Data Engineering
Durga Gadiraju
 
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
SnapLogic
 
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Denodo
 
Building Blocks for Hybrid IT
Building Blocks for Hybrid ITBuilding Blocks for Hybrid IT
Building Blocks for Hybrid IT
RightScale
 
Data virtualization
Data virtualizationData virtualization
Data virtualization
Hamed Hatami
 
BPM and SOA are going mobile - An architectural perspective
BPM and SOA are going mobile - An architectural perspectiveBPM and SOA are going mobile - An architectural perspective
BPM and SOA are going mobile - An architectural perspective
OPITZ CONSULTING Deutschland
 
BPM und SOA machen mobil - Ein Architekturüberblick
BPM und SOA machen mobil - Ein ArchitekturüberblickBPM und SOA machen mobil - Ein Architekturüberblick
BPM und SOA machen mobil - Ein Architekturüberblick
OPITZ CONSULTING Deutschland
 
Break Free from Oracle
Break Free from OracleBreak Free from Oracle
Break Free from Oracle
EDB
 

More from Paolo Platter (12)

Witboost Platform for decentralization of data management
Witboost Platform for decentralization of data managementWitboost Platform for decentralization of data management
Witboost Platform for decentralization of data management
Paolo Platter
 
Platform Strategy for decentralization.pptx
Platform Strategy for decentralization.pptxPlatform Strategy for decentralization.pptx
Platform Strategy for decentralization.pptx
Paolo Platter
 
DAMA Norway - Computational Governance Model
DAMA Norway - Computational Governance ModelDAMA Norway - Computational Governance Model
DAMA Norway - Computational Governance Model
Paolo Platter
 
Data Mesh Implementation - a practical journey
Data Mesh Implementation - a practical journeyData Mesh Implementation - a practical journey
Data Mesh Implementation - a practical journey
Paolo Platter
 
kafka simplicity and complexity
kafka simplicity and complexitykafka simplicity and complexity
kafka simplicity and complexity
Paolo Platter
 
Wasp2 - IoT and Streaming Platform
Wasp2 - IoT and Streaming PlatformWasp2 - IoT and Streaming Platform
Wasp2 - IoT and Streaming Platform
Paolo Platter
 
Meetup tensorframes
Meetup tensorframesMeetup tensorframes
Meetup tensorframes
Paolo Platter
 
Bringing Deep Learning into production
Bringing Deep Learning into production Bringing Deep Learning into production
Bringing Deep Learning into production
Paolo Platter
 
Agile Lab_BigData_Meetup_AKKA
Agile Lab_BigData_Meetup_AKKAAgile Lab_BigData_Meetup_AKKA
Agile Lab_BigData_Meetup_AKKA
Paolo Platter
 
Agile Lab_BigData_Meetup
Agile Lab_BigData_MeetupAgile Lab_BigData_Meetup
Agile Lab_BigData_Meetup
Paolo Platter
 
Massive Streaming Analytics with Spark Streaming
Massive Streaming Analytics with Spark StreamingMassive Streaming Analytics with Spark Streaming
Massive Streaming Analytics with Spark Streaming
Paolo Platter
 
Scala Intro
Scala IntroScala Intro
Scala Intro
Paolo Platter
 
Witboost Platform for decentralization of data management
Witboost Platform for decentralization of data managementWitboost Platform for decentralization of data management
Witboost Platform for decentralization of data management
Paolo Platter
 
Platform Strategy for decentralization.pptx
Platform Strategy for decentralization.pptxPlatform Strategy for decentralization.pptx
Platform Strategy for decentralization.pptx
Paolo Platter
 
DAMA Norway - Computational Governance Model
DAMA Norway - Computational Governance ModelDAMA Norway - Computational Governance Model
DAMA Norway - Computational Governance Model
Paolo Platter
 
Data Mesh Implementation - a practical journey
Data Mesh Implementation - a practical journeyData Mesh Implementation - a practical journey
Data Mesh Implementation - a practical journey
Paolo Platter
 
kafka simplicity and complexity
kafka simplicity and complexitykafka simplicity and complexity
kafka simplicity and complexity
Paolo Platter
 
Wasp2 - IoT and Streaming Platform
Wasp2 - IoT and Streaming PlatformWasp2 - IoT and Streaming Platform
Wasp2 - IoT and Streaming Platform
Paolo Platter
 
Bringing Deep Learning into production
Bringing Deep Learning into production Bringing Deep Learning into production
Bringing Deep Learning into production
Paolo Platter
 
Agile Lab_BigData_Meetup_AKKA
Agile Lab_BigData_Meetup_AKKAAgile Lab_BigData_Meetup_AKKA
Agile Lab_BigData_Meetup_AKKA
Paolo Platter
 
Agile Lab_BigData_Meetup
Agile Lab_BigData_MeetupAgile Lab_BigData_Meetup
Agile Lab_BigData_Meetup
Paolo Platter
 
Massive Streaming Analytics with Spark Streaming
Massive Streaming Analytics with Spark StreamingMassive Streaming Analytics with Spark Streaming
Massive Streaming Analytics with Spark Streaming
Paolo Platter
 
Ad

Recently uploaded (20)

Douwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License codeDouwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License code
aneelaramzan63
 
Societal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainabilitySocietal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainability
Jordi Cabot
 
Download Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With LatestDownload Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With Latest
tahirabibi60507
 
FL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full VersionFL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full Version
tahirabibi60507
 
Expand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchangeExpand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchange
Fexle Services Pvt. Ltd.
 
Download YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full ActivatedDownload YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full Activated
saniamalik72555
 
Adobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest VersionAdobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest Version
kashifyounis067
 
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...
Eric D. Schabell
 
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Lionel Briand
 
Revolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptxRevolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptx
nidhisingh691197
 
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Dele Amefo
 
WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)
sh607827
 
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
Andre Hora
 
Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025
kashifyounis067
 
Scaling GraphRAG: Efficient Knowledge Retrieval for Enterprise AI
Scaling GraphRAG:  Efficient Knowledge Retrieval for Enterprise AIScaling GraphRAG:  Efficient Knowledge Retrieval for Enterprise AI
Scaling GraphRAG: Efficient Knowledge Retrieval for Enterprise AI
danshalev
 
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
Andre Hora
 
The Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdfThe Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdf
drewplanas10
 
How can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptxHow can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptx
laravinson24
 
Solidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license codeSolidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license code
aneelaramzan63
 
Top 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docxTop 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docx
Portli
 
Douwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License codeDouwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License code
aneelaramzan63
 
Societal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainabilitySocietal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainability
Jordi Cabot
 
Download Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With LatestDownload Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With Latest
tahirabibi60507
 
FL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full VersionFL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full Version
tahirabibi60507
 
Expand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchangeExpand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchange
Fexle Services Pvt. Ltd.
 
Download YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full ActivatedDownload YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full Activated
saniamalik72555
 
Adobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest VersionAdobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest Version
kashifyounis067
 
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...
Eric D. Schabell
 
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Lionel Briand
 
Revolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptxRevolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptx
nidhisingh691197
 
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Dele Amefo
 
WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)
sh607827
 
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
Andre Hora
 
Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025
kashifyounis067
 
Scaling GraphRAG: Efficient Knowledge Retrieval for Enterprise AI
Scaling GraphRAG:  Efficient Knowledge Retrieval for Enterprise AIScaling GraphRAG:  Efficient Knowledge Retrieval for Enterprise AI
Scaling GraphRAG: Efficient Knowledge Retrieval for Enterprise AI
danshalev
 
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
Andre Hora
 
The Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdfThe Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdf
drewplanas10
 
How can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptxHow can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptx
laravinson24
 
Solidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license codeSolidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license code
aneelaramzan63
 
Top 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docxTop 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docx
Portli
 
Ad

The role of Dremio in a data mesh architecture

  • 1. The role of Dremio in a Data Mesh architecture Presented by: Paolo Platter – CTO & Co-founder @ Agile Lab
  • 2. Who we are : • We value transparency, collaboration and results • Totally decentralized and self-managed • International culture and mindset • Customer laser focused What we do: • Data Engineering is our mission since 2013 • Crafting end-to-end data platforms • Data Strategy • Managed Data Service www.agilelab.it
  • 3. Data Mesh Principles Domain Driven Data Ownership Architecture Data as a product Self-Serve Infrastructure as a Platform Federated Computational Governance
  • 4. Data Product Data + Metadata (syntax+semantic, expected behaviour, access control ) Infrastructure Data Pipeline Data Access API Observability API Internal processes (GDPR, DQ, etc ) Stream Processing Information API Control Ports code data Input Ports • Operational systems • Other Data Products • External services Output Ports • Events • SQLView • Raw/Files • Graph/RDF infrastructure
  • 5. Technology Independence • Addressability • Interoperability • Self-Serve provisioning • Independently deployable • Data Mesh is a practice • Each Data Product team can select the technology that best fits the use case. • The technology must be compliant with Data Product features and requirements • Multi-cloud needs
  • 6. Output Ports Output Port API Data Consumer • Descriptive schema • Audit • Access Control • Decoupling(uri and protocol) • SLO Read data through native protocol Events SQL Files Output Port API Data Consumer • GraphQL or HTTP • Data is flowing throught API • Zero coupling, low performances • Not suitable for all use cases of data consumption Events SQL Files • Data is flowing throught native protocol • Low coupling, good performances • Fully Polyglot Pre-flight
  • 8. Connecting a BI tool to an Output Port BI Tool SQL Files Output Port API GraphQL or other HTTP based protocols are not widely supported by BI Tools. Also thinking to have a custom pre-flight and dynamically discover the protocol of the source is something not easy In order to query directly a file/object storage you need a SQL Engine, tipically not available inside BI Tools JDBC/ODBC connection is a good and standard option for BI Tools, but this is hiding problems
  • 9. Client-side coupling is not good Consumer JDBC driver Athena JDBC driver Redshift JDBC driver Aurora • One driver doesn’t fit all • Coupling is becoming a problem for change management • DP is not indipendently deployable Resons why you need multiple technologies in a data mesh: - Not all the use cases fit with a single tech - Data Mesh is an evolutionary architecture, technologies will evolve and change over time and DPs will adopt them indipendently - Your data mesh is expanding on a multi- cloud landscape
  • 10. How to integrate legacy systems Data Mesh Data Lake Migration will require time... Consumer j d b c jdbc What if we need to join data coming from different JDBC channels ? Huge impact on performances Join must be resolved at this level
  • 12. Dremio Uniqueness SQL Execution Engine OLAP Accelleration Virtualization Extensible Semantic Layer Scalable Cache Cloud & Tech Agnostic Flexible & Self Serve deployment
  • 13. Fitting into the big picture Native integration with data lakes Bridge also other enteprise assets Data Consumer • Single interface to access all the silos and no coupling between data consumers and multiple specific technologies • Single catalog of data You can use it as SQL Query Engine inside Data Products Interfacing other DBMS Cloud agnostic • Efficient join between DPs leveraging different underlying technologies • Query federation between data mesh and other data assets across the organization • Native integration with DataLakes to facilitate the transition to the DataMesh
  • 14. Self-Serve provisioning API Catalog Execution Engine ACL CI/CD Data Product specification Provision: • Execution resources • SQL View • ACL Provision: • Storage • DB • Policies • IAM • etc Deploy
  • 15. Data-Product Caching Main Entity Pre-aggregated and denormalized views Query Acceleration & Caching Leverage external reflections to speed-up queries automatically without adding complexity to the data consumer Query Dremio can create such pre-aggregation automatically without the need to implement custom jobs for such purpose Data Consumer Data Consumer interacts with a single logical entity, but queries will speed-up due to the cache and reflections.
  • 16. Thank You! – Q&A Time Contact me at: [email protected] https://www.agilelab.it/data-mesh-in-action/