"Data Mesh in Kubernetes", Andrii Syniuk

Sep 23, 20230 likes218 views

PwC Germany is working with a lot of data from different domains and sources, access to which should be properly governed. To tackle those problems and to make access to the data more transparent and straight-forward, we’re building our internal Data Ecosystem. From this talk, we will cover the following topics: Data storage and analytics evolution What is Data Mesh? How do we build it in Kubernetes? Challenges that we were dealing with, and see in front of us.

"Data Mesh in Kubernetes", Andrii Syniuk

1. Introduction
2. Data storage and analytics evolution
3 What is Data Mesh?
4. Motivation
5. Components, Implementation, Evolution
6. Challenges
7. Q&A
Agenda

whoami
https://www.linkedin.com/in/andrii-syniuk/
Domain Driven Design enthusiast
Software Engineer
Site Reliability Engineer
Data Ecosystem Tech Lead
Data curious gold digger

Modern Software architectures and data in it
Monolithic Architecture
microservices.io
Data Warehouse

Modern Software architectures and data in it

Main principles
• Data as a Product
• Federated Governance
• Domain Ownership
• Self serve
Data Mesh
9

Why bother? It was working just fine.
- Reusing data products
- Faster time to market
- Decentralized data teams

Landscape
DB
Data products
Pipelines
Governance Self Service

Challenges
- Policy As A Code
- Data quality
- A lot of small databases
- Avoiding vendor lock-in
- Platformization
- Get it work together

Join the team of experts from Netwoven and Microsoft to learn more about things you must know about taking a Hybrid SharePoint strategy as the industry moves towards cloud based computing. SharePoint 2013 and its online counterpart via the latest rev of Office 365 has been in the market for quite some time now. IT decision makers must choose whether to deploy it in-house, use fully on cloud or use a hybrid implementation. According to a survey – 55 percent of users intend to continue running SharePoint entirely in-house and only 10 percent plan to run it purely in the cloud. The remaining 35 percent are planning hybrid SharePoint implementations.

Finding Your Ideal Data Architecture: Data Fabric, Data Mesh or Both?Denodo

Watch full webinar here: https://bit.ly/3Y2TBXB Two of the most talked about topics in data management today are Data Fabric and Data Mesh. However, there is a lot of confusion around them. Are they alternative options, or are they complementary? Many organizations are struggling with these questions when trying to modernize their data architecture. Mike Ferguson, Managing Director of Intelligent Business Strategies, will help clear up the confusion by looking at what Data Fabric and Data Mesh are and how they can best be used to help shorten time to value in companies seeking to become data-driven enterprises. Mike will help address many of your questions, including: - What is a Data Fabric and Data Mesh, and the business value of each? - What are the key concepts and capabilities of each, and what do they make possible? - The implications of decentralizing data engineering, and how do you co-ordinate data product development? - How can a Data Fabric help in building a Data Mesh? Following Mike's presentation, we will be joined by Kevin Bohan of Denodo, who will discuss the foundational capabilities you should be putting in place if you are planning on adopting a Data Mesh strategy.

A Key to Real-time Insights in a Post-COVID World (ASEAN)Denodo

Watch full webinar here: https://bit.ly/2EpHGyd Presented at Data Champions, Online Asia 2020 Businesses and individuals around the world are experiencing the impact of a global pandemic. With many workers and potential shoppers still sequestered, COVID-19 is proving to have a momentous impact on the global economy. Regardless of the current situation and post-pandemic era, real-time data becomes even more critical to healthcare practitioners, business owners, government officials, and the public at large where holistic and timely information are important to make quick decisions. It enables doctors to make quick decisions about where to focus the care, business owners to alter production schedules to meet the demand, government agencies to contain the epidemic, and the public to be informed about prevention. In this on-demand session, you will learn about the capabilities of data virtualization as a modern data integration technique and how can organisations: - Rapidly unify information from disparate data sources to make accurate decisions and analyse data in real-time - Build a single engine for security that provides audit and control by geographies - Accelerate delivery of insights from your advanced analytics project

Data Mesh using Microsoft FabricNathan Bijnens

The document discusses Microsoft's approach to implementing a data mesh architecture using their Azure Data Fabric. It describes how the Fabric can provide a unified foundation for data governance, security, and compliance while also enabling business units to independently manage their own domain-specific data products and analytics using automated data services. The Fabric aims to overcome issues with centralized data architectures by empowering lines of business and reducing dependencies on central teams. It also discusses how domains, workspaces, and "shortcuts" can help virtualize and share data across business units and data platforms while maintaining appropriate access controls and governance.

Boston Data Engineering: Designing and Implementing Data Mesh at Your Company...Boston Data Engineering

Data Mesh is a fairly new approach to help companies do more with data, faster. It requires both organizational and technical changes to enable autonomy and self-service, treat data as a product and encourage secure collaboration. In this session, we will discuss practical approaches you can implement today to help your company start benefiting from Data Mesh. We'll show you how to create autonomy by splitting responsibility between data producers and consumers, share datasets and make data discovery easy. We'll show a demo with producers building an ingestion pipeline that publishes datasets to consumer accounts (data mesh domains). SQL templates will be provided for members to follow along and build on their own. We'll present these use cases built with data mesh design patterns: 1. A multi-tenant data lake that allows data producers to share datasets with consumers outside of the organization (3rd parties). 2. A security data lake that allows different teams to publish curated logs to their local Elasticsearch clusters for analysis, and to a central data lake for retention, auditing and historical analysis. We'll also discuss managing data contracts/schemas between producers and consumers, to enable ownership and better data quality when sharing datasets. Meetup: https://www.meetup.com/boston-data-engineering/events/291383661/ Video: https://youtu.be/lIcmomYZ3mo

Introduction To SQL Server 2014Vishal Pawar

The document provides an overview of the speaker's experience and qualifications. It lists the speaker's 7+ years of experience working with Microsoft's BI stack, current role as a MS BI Solution Architect, consulting experience, certifications including MCSA and MCTS in SQL Server, education including a BE in Computer Science and an MBA. It also provides links to the speaker's blogs and details on various SQL and BI articles authored.

Feature Store as a Data Foundation for Machine LearningProvectus

This document discusses feature stores and their role in modern machine learning infrastructure. It begins with an introduction and agenda. It then covers challenges with modern data platforms and emerging architectural shifts towards things like data meshes and feature stores. The remainder discusses what a feature store is, reference architectures, and recommendations for adopting feature stores including leveraging existing AWS services for storage, catalog, query, and more.

Building the Artificially Intelligent EnterpriseDatabricks

Mike Ferguson is Managing Director of Intelligent Business Strategies Limited and specializes in business intelligence/analytics and data management. He discusses building the artificially intelligent enterprise and transitioning to a self-learning enterprise. Some key challenges discussed include the siloed and fractured nature of current data and analytics efforts, with many tools and scripts in use without integration. He advocates sorting out the data foundation, implementing DataOps and MLOps, creating a data and analytics marketplace, and integrating analytics into business processes to drive value from AI.

Ibm db2update2019 icp4 dataGustav Lundström

DAMA & Denodo Webinar: Modernizing Data Architecture Using Data Virtualization Denodo

Watch here: https://bit.ly/2NGQD7R In an era increasingly dominated by advancements in cloud computing, AI and advanced analytics it may come as a shock that many organizations still rely on data architectures built before the turn of the century. But that scenario is rapidly changing with the increasing adoption of real-time data virtualization - a paradigm shift in the approach that organizations take towards accessing, integrating, and provisioning data required to meet business goals. As data analytics and data-driven intelligence takes centre stage in today’s digital economy, logical data integration across the widest variety of data sources, with proper security and governance structure in place has become mission-critical. Attend this session to learn: - Learn how you can meet cloud and data science challenges with data virtualization. - Why data virtualization is increasingly finding enterprise-wide adoption - Discover how customers are reducing costs and improving ROI with data virtualization

Belgium & Luxembourg dedicated online Data Virtualization discovery workshopDenodo

Watch full webinar here: https://bit.ly/33yYuQm Data virtualization has become an essential part of enterprise data architectures, bridging the gap between IT and business users and delivering significant cost and time savings. This technology revolutionizes the way data is accessed, delivered, consumed and governed regardless of its format and location. This 1.5 hour discovery session will show help you identify the benefits of this modern and agile data integration and management technology for your organisation.

Curiosity Software and RCG Global Services Present - Solving Test Data: the g...Curiosity Software Ireland

This webinar was co-hosted by Curiosity and RCG Global Services on January 20th, 2022. Watch the webinar on demand: https://www.curiositysoftware.ie/solving-test-data-webinar Outdated test data management practices are today a sinkhole for testing and development time. They stifle release velocity, risk costly legislative non-compliance, and yet still do not provide the data needed to protect releases from damaging bugs. To achieve true quality at speed, the test data paradigm must shift. Enterprises must move beyond slowly copying large sets of production data to a limited number of out-of-date test environments. In this webinar, Global Head of Quality Engineering at RCG, Niko Mangahas, draws on extensive project experience to define the test data challenges facing enterprises today. He then helps you identify the right test data solution for your organisation, setting out principles for effective requirements gathering and program design. Niko then hands over to veteran test data inventor, Huw Price, who demoes some of the latest techniques for making complete and compliant data available on-the-fly during parallel testing, development, and CI/CD.

Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...Denodo

Watch full webinar here: https://bit.ly/3fBpO2M Data Fabric has been a hot topic in town and Gartner has termed it as one of the top strategic technology trends for 2022. Noticeably, many mid-to-large organizations are also starting to adopt this logical data fabric architecture while others are still curious about how it works. With a better understanding of data fabric, you will be able to architect a logical data fabric to enable agile data solutions that honor enterprise governance and security, support operations with automated recommendations, and ultimately, reduce the cost of maintaining hybrid environments. In this on-demand session, you will learn: - What is a data fabric? - How is a physical data fabric different from a logical data fabric? - Which one should you use and when? - What’s the underlying technology that makes up the data fabric? - Which companies are successfully using it and for what use case? - How can I get started and what are the best practices to avoid pitfalls?

When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY

Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020. Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms. Data lakes will be built in cloud object storage. We’ll discuss the options there as well. Get this data point for your data lake journey.

Data Systems Integration & Business Value Pt. 2: CloudData Blueprint

The document discusses cloud-based integration and its prerequisites. It states that for organizations to benefit from cloud integration, data must be (1) of higher quality, (2) lower volume, and (3) more shareable than data residing outside the cloud. Investments in data engineering are needed to cleanse, reduce the size of, and increase the shareability of datasets so that organizations can realize increased capacity, flexibility, and cost savings from cloud-based computing. The webinar will show how to identify opportunities for cloud integration and properly oversee efforts to capitalize on those opportunities.

Data Systems Integration & Business Value Pt. 2: CloudDATAVERSITY

Certain systems are more data focused than others. Usually their primary focus is on accomplishing integration of disparate data. In these cases, failure is most often attributable to the adoption of a single pillar (silver bullet). The three webinars in the Data Systems Integration and Business Value series are designed to illustrate that good systems development more often depends on at least three DM disciplines (pie wedges) in order to provide a solid foundation.

All Together Now: A Recipe for Successful Data GovernanceInside Analysis

The Briefing Room with David Loshin and Phasic Systems Slides from the Live Webcast on July 10, 2012 Getting disparate groups of professionals to agree on business terminology can take forever, especially when big dollars or major issues are at stake. Many data governance programs languish indefinitely because of simple hang-ups. But a new approach has recently achieved monumental results for the United States Navy. The detailed process has since been codified and combined with a NoSQL technology that enables even the most complex data models and definitions to be distilled into simple, functional data flows. Check out this episode of The Briefing Room to hear Analyst David Loshin of Knowledge Integrity explain why effective Data Governance requires cooperation. Loshin will be briefed by Geoffrey Malafsky of Phasic Systems who will tout his company's proprietary protocol for extracting, defining and managing critical information assets and processes. He'll explain how their approach allows everyone to be "correct" in their definitions, without causing data quality or performance issues in associated information systems. And he'll explain how their Corporate NoSQL engine enables real-time harmonization of definitions and dimensions. Visit us at: http://www.insideanalysis.com

awari-ds-aula1.pdfMarcos993896

The document provides an overview of data science including definitions, careers, applications and tools. It defines data science, describes the typical steps in a data science project including understanding the problem, acquiring and preparing data, analyzing data, modeling data, visualizing results and deploying solutions. It also discusses careers in data engineering, data analysis, machine learning engineering and as a data scientist. Finally, it covers popular tools and frameworks used in data science like Anaconda, Jupyter Notebooks and examples of data science applications.

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)Denodo

Watch full webinar here: https://bit.ly/3aIofv9 The best of breed big data fabrics should deliver actionable insights to the business users with minimal effort, provide end-to-end security to the entire enterprise data platform and provide real-time data integration, while delivering self-service data platform to business users. While big data initiatives have become necessary for any business to generate actionable insights, big data fabric has become a necessity for any successful big data initiative. The best of breed big data fabrics should deliver actionable insights to the business users with minimal effort, provide end-to-end security to the entire enterprise data platform and provide real-time data integration, while delivering self-service data platform to business users. Attend this session to learn how big data fabric enabled by data virtualization: - Provides lightning fast self-service data access to business users - Centralizes data security, governance and data privacy - Fulfills the promise of data lakes to provide actionable insights

Data DevOps: An OverviewScott W. Ambler

Data, the way that we process it and store it, is one of many important aspects of IT. Data is the lifeblood of our organizations, supporting real-time business processes and decision-making. For our DevOps strategy to be truly effective we must be able to safely and quickly evolve production databases, just as we safely and quickly evolve production code. Yet for many organizations their data sources prove to be less than trustworthy and their data-oriented development efforts little more than productivity sinkholes. We can, and must, do better. This presentation begins with a collection of agile principles for data professionals and of data principles for agile developers - the first step in working together is to understand and appreciate the priorities and strengths of the people that we work with. Our focus is on a collection of practices that enable development teams to easily and safely evolve and deploy databases. These techniques include agile data modeling, database refactoring, database regression testing, continuous database integration, and continuous database deployment. We also work through operational strategies required of production databases to support your DevOps strategy. If data sources aren’t an explicit part of your DevOps strategy then you’re not really doing DevOps, are you?

Strata San Jose 2017 - Ben Sharma PresentationZaloni

The document discusses creating a modern data architecture using a data lake. It describes Zaloni as a provider of data lake management solutions, including a data lake management and governance platform and self-service data platform. It outlines key features of a data lake such as storing different types of data, creating standardized datasets, and providing shorter time to insights. The document also discusses Zaloni's data lake maturity model and reference architecture.

Driving Business Value Through Agile Data AssetsEmbarcadero Technologies

Bbbt presentation 210415_final_2Roland Bullivant

Data+Management+Masterclasssdfsdfsdfsd.pdfNguyễn Hải

Lecture 5- Data Collection and Storage.pptxBrianc34

Building a Data Driven Culture and AI Revolution With Gregory Little | Curren...HostedbyConfluent

Building a Data Driven Culture and AI Revolution With Gregory Little | Current 2022 Transforming business or mission through AI/ML doesn't start with technology but with culture…and an audit. At least as much is true for the US Department of Defense (DoD), which presents significant modernization challenges because of its mission scope, expansive global footprint, and massive size - with over 2.8 million people, it is the largest employer in the world. Greg Little discusses how establishing the DoD’s annual audit became a surprising accelerator for the department’s data and analytics journey. It revealed the foundational needs for data management to run a $3 trillion in assets enterprise, and its successful implementation required breaking through deeply entrenched cultural and organizational resistance across DoD. In this session, Greg will discuss what it will take to guide the evolution of technology and culture in parallel: leadership, technology that enables rapid scale and a complete & reliable data flow, and a data driven culture.

Cloud Computing - DemystifiedLadderMinds

Від KPI до OKR: як синхронізувати продажі, маркетинг і продукт, щоб бізнес ре...Fwdays

Часто компанії застрягають у постійній боротьбі між департаментами: маркетинг веде мало (або «не тих») лідів, сейли не можуть їм продати, а коли таки продають, то продакти кажуть, що клієнти «не ті» й у нас немає для них продукту… Звісно, що CEO хоче краще RoI і закриті показники вже вчора. В результаті – кожен успішно виконує свої KPI, але компанія не росте. Знайомо? Ця доповідь про те, як об’єднати ключові функції навколо спільних цілей і створити реальну систему завдяки OKR. 💸 Ми розберемо: Чому KPI не допомагають рости, а OKR – можуть. Як правильно формувати цілі і ключові результати, щоб усі працювали в одному напрямку. Як інтегрувати OKR у продуктову стратегію. Як уникнути типових фейлів впровадження OKR. Ну і традиційно живі приклади, трохи сарказму і практичні поради.

"Demand Generation: How a Founder’s Brand Turns Content into Leads", Alex Her...Fwdays

More Related Content

Similar to "Data Mesh in Kubernetes", Andrii Syniuk (20)