EDA Governance Model: a multicloud approach based on GitOps | Alejandro Alija and Marco Laucelli, Galeo Tech
Download as PPTX, PDF0 likes416 views
This document discusses Galeo's EDA governance model based on GitOps for multi-cloud environments. The model defines a standardized platform and service catalogue using a unified documentation model consolidated in Git repositories. This allows governance procedures like project approval, infrastructure provisioning, and component deployment to be automated through a single CI/CD pipeline without additional documentation overhead. The catalogue is kept up-to-date based on what is actually deployed, facilitating reuse and standardization across projects.
1 of 14
Download to read offline
Ad
Recommended
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...HostedbyConfluent A fast and efficient integration of end device data into data processing systems is becoming increasingly important in the Internet of Things. Factors such as secure and reliable data transmission, real-time data processing and the analysis of huge amounts of data afterwards play a major role. In order to implement an architecture that enables smooth communication between end devices and systems, tools are needed that are designed for a type of communication that takes these factors into account.
This presentation shows the strengths and application areas of MQTT, the de-facto standard communication protocol for the Internet of Things, and Apache Kafka, which is often used for data streaming in the Internet of Things, and explains how and why the technologies complement each other ideally.
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...HostedbyConfluent As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...HostedbyConfluent Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...HostedbyConfluent We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...confluent This document summarizes BNY Mellon's implementation of a data distribution hub to improve data management across the enterprise. Key points:
1) BNY Mellon developed a data distribution hub using a microservices architecture and Kafka to decouple data sources and consumers, centralize data transformation and security, and improve data lineage management.
2) The hub scales horizontally using three dimensions - functional decomposition of microservices, horizontal scaling within an internal cloud, and data sharding.
3) Kafka is used extensively to integrate microservices, maintain processing offsets, and allow idempotent execution.
4) Both batch and real-time data processing are supported, with the latter using Storm on Kafka and loading data into
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...HostedbyConfluent Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services.
At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure.
This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.
0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche
0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, PorscheHostedbyConfluent The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...HostedbyConfluent This document discusses testing event-driven architectures. It begins by defining common event-driven architecture patterns like event notifications and event sourcing. It then discusses brokering the complexity of event-driven architectures by describing how events are communicated between producers and consumers via channels. The document outlines what information should be included in events like payloads and headers. It also discusses the difference between orchestration and choreography in event-driven systems. It provides an example of how events can be used to mediate changes within a system using order validation. Finally, it demonstrates how to test event-driven architectures using specifications and discusses accelerating API quality through testing tools that support multiple protocols and definitions.
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, JupiterHostedbyConfluent Stream processing allows analyzing streaming data in real-time using tools like KSQL. KSQL builds on Kafka Streams to provide a SQL-like interface for transforming, filtering, and creating derived streams from immutable event data ingested into Kafka. Jupiter, a digital banking platform, uses KSQL to reconcile transactions from different interfaces in real-time and check rewards eligibility by analyzing consolidated transaction streams. While powerful, KSQL deployments require monitoring, richer SQL support, and integration with big data platforms for long-term storage.
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Comparing three data ingestion approaches where Apache Kafka integrates with ...HostedbyConfluent Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...HostedbyConfluent Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more. In looking at the tradeoffs behind a system like Kafka, we learn to reason more clearly about distributed systems and to make high-stakes technology adoption decisions more effectively. These are skills we all want to improve!
In this talk, we'll examine trade-offs on which our favorite distributed messaging system takes opinionated positions:
- Whether to store data contiguously or using an index
- How many storage tiers are best?
- Where should metadata live?
- And more.
It's always useful to dissect a modern distributed system with the goal of understanding it better, and it's even better to learn to deeper architectural principles in the process. Come to this talk for a generous helping of both.
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...confluent (Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018
Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications.
This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures.
Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...HostedbyConfluent Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions
Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...HostedbyConfluent Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost.
in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...HostedbyConfluent Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.
Digital transformation: Highly resilient streaming architecture and strategie...
Digital transformation: Highly resilient streaming architecture and strategie...HostedbyConfluent Failure is inevitable in any distributed system but anticipating failures and building systems to recover from failures instantaneous makes the system highly resilient. At Capital One we process billions of events everyday and we leverage cloud, microservices, streaming and machine learning technologies to solve customer problems and provide the best customer experience.
As part of this session I will be talking about highly resilient streaming architecture that is supporting processing of billions of events every day then some of the strategies & best practices to build highly available and fault-tolerant systems utilizing Kafka and Cloud environments.
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...HostedbyConfluent One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Death of the dumb pipes: Using Apache Kafka® for Integration projectsHostedbyConfluent Guru Sattanathan, Confluent, Senior Solutions Engineer
Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats!
Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts?
If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today.
In this session, we will go in this order:
-Recap on Enterprise Integration technologies
-What are the key flaws & What needs improvement?
-What is Apache Kafka?
-Rethinking Integration using Apache Kafka
https://www.meetup.com/KafkaMelbourne/events/280590162/
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...HostedbyConfluent Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...HostedbyConfluent At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQHostedbyConfluent Joins in Kafka Streams and ksqlDB are a killer-feature for data processing and basic join semantics are well understood. However, in a streaming world records are associated with timestamps that impact the semantics of joins: welcome to the fabulous world of _temporal_ join semantics. For joins, timestamps are as important as the actual data and it is important to understand how they impact the join result.
In this talk we want to deep dive on the different types of joins, with a focus of their temporal aspect. Furthermore, we relate the individual join operators to the overall ""time engine"" of the Kafka Streams query runtime and explain its relationship to operator semantics. To allow developers to apply their knowledge on temporal join semantics, we provide best practices, tip and tricks to ""bend"" time, and configuration advice to get the desired join results. Last, we give an overview of recent, and an outlook to future, development that improves joins even further.
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...HostedbyConfluent The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...HostedbyConfluent When the University of California, San Diego launched its largest investment in tech in 2018, they planned to future proof their business processes and systems. Unexpectedly, it also prepared them to handle a global pandemic that changed every norm for the campus. With shelter-in-place orders taking immediate effect, they needed to quickly set up a robust online learning platform - one with powerful analytics to track student success. And, for the times students and staff are on campus, a contact tracing application was essential for their safety. We’d like to offer a conversation with Scott Lee to tell you more about UC San Diego’s rapid transformation from a traditional, on-campus institution to one of the leading examples of remote learning, and the critical role data connectivity played in making this possible.
From legacy systems to microservices and back | Andera Gioia, Quantyca
From legacy systems to microservices and back | Andera Gioia, QuantycaHostedbyConfluent Legacy systems are the kings of our IT architecture. They rule the evolution of the technology ecosystem that hosts them thanks to the control they have acquired over time on core data and key business processes. Legacy systems however pose severe limits in addressing critical business needs, as well as seizing opportunities for future growth.
Digital Integration Hub (DIH) is a quite popular architecture to modernize legacy systems allowing the IT to evolve faster and better integrate modern technologies.
In this talk we’ll:
- Review DIH general architecture
- Illustrate different patterns to offload data from legacy systems to Kafka with pros and cons in term of consistency and latency
- Present different ways to consume data offloaded from legacy system to Kafka and their fit with different use cases
- Show how to close the loop and use Kafka and CQRS pattern to handle all the data moving backward from applications to legacy systems
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, AivenHostedbyConfluent When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...HostedbyConfluent Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...HostedbyConfluent If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.
Belgium & Luxembourg dedicated online Data Virtualization discovery workshop
Belgium & Luxembourg dedicated online Data Virtualization discovery workshopDenodo Watch full webinar here: https://bit.ly/33yYuQm
Data virtualization has become an essential part of enterprise data architectures, bridging the gap between IT and business users and delivering significant cost and time savings. This technology revolutionizes the way data is accessed, delivered, consumed and governed regardless of its format and location.
This 1.5 hour discovery session will show help you identify the benefits of this modern and agile data integration and management technology for your organisation.
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformation
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformationSlobodan Sipcic The document introduces an evolutionary event-driven architecture called the Enterprise Digital Transformation Platform (EDTP) for accelerating digital transformation. The EDTP is a 4-tier platform based on cloud, containers, microservices, events and streaming. It addresses challenges of data integration and decoupling through architectural concepts like event-driven design, microservices and templates. The EDTP provides full-stack deployment automation and microservice templating to accelerate development. Use cases from Toyota Financial Services are presented to demonstrate the EDTP's capabilities.
Ad
More Related Content
What's hot (20)
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, JupiterHostedbyConfluent Stream processing allows analyzing streaming data in real-time using tools like KSQL. KSQL builds on Kafka Streams to provide a SQL-like interface for transforming, filtering, and creating derived streams from immutable event data ingested into Kafka. Jupiter, a digital banking platform, uses KSQL to reconcile transactions from different interfaces in real-time and check rewards eligibility by analyzing consolidated transaction streams. While powerful, KSQL deployments require monitoring, richer SQL support, and integration with big data platforms for long-term storage.
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Comparing three data ingestion approaches where Apache Kafka integrates with ...HostedbyConfluent Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...HostedbyConfluent Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more. In looking at the tradeoffs behind a system like Kafka, we learn to reason more clearly about distributed systems and to make high-stakes technology adoption decisions more effectively. These are skills we all want to improve!
In this talk, we'll examine trade-offs on which our favorite distributed messaging system takes opinionated positions:
- Whether to store data contiguously or using an index
- How many storage tiers are best?
- Where should metadata live?
- And more.
It's always useful to dissect a modern distributed system with the goal of understanding it better, and it's even better to learn to deeper architectural principles in the process. Come to this talk for a generous helping of both.
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...confluent (Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018
Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications.
This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures.
Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...HostedbyConfluent Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions
Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...HostedbyConfluent Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost.
in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...HostedbyConfluent Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.
Digital transformation: Highly resilient streaming architecture and strategie...
Digital transformation: Highly resilient streaming architecture and strategie...HostedbyConfluent Failure is inevitable in any distributed system but anticipating failures and building systems to recover from failures instantaneous makes the system highly resilient. At Capital One we process billions of events everyday and we leverage cloud, microservices, streaming and machine learning technologies to solve customer problems and provide the best customer experience.
As part of this session I will be talking about highly resilient streaming architecture that is supporting processing of billions of events every day then some of the strategies & best practices to build highly available and fault-tolerant systems utilizing Kafka and Cloud environments.
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...HostedbyConfluent One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Death of the dumb pipes: Using Apache Kafka® for Integration projectsHostedbyConfluent Guru Sattanathan, Confluent, Senior Solutions Engineer
Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats!
Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts?
If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today.
In this session, we will go in this order:
-Recap on Enterprise Integration technologies
-What are the key flaws & What needs improvement?
-What is Apache Kafka?
-Rethinking Integration using Apache Kafka
https://www.meetup.com/KafkaMelbourne/events/280590162/
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...HostedbyConfluent Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...HostedbyConfluent At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQHostedbyConfluent Joins in Kafka Streams and ksqlDB are a killer-feature for data processing and basic join semantics are well understood. However, in a streaming world records are associated with timestamps that impact the semantics of joins: welcome to the fabulous world of _temporal_ join semantics. For joins, timestamps are as important as the actual data and it is important to understand how they impact the join result.
In this talk we want to deep dive on the different types of joins, with a focus of their temporal aspect. Furthermore, we relate the individual join operators to the overall ""time engine"" of the Kafka Streams query runtime and explain its relationship to operator semantics. To allow developers to apply their knowledge on temporal join semantics, we provide best practices, tip and tricks to ""bend"" time, and configuration advice to get the desired join results. Last, we give an overview of recent, and an outlook to future, development that improves joins even further.
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...HostedbyConfluent The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...HostedbyConfluent When the University of California, San Diego launched its largest investment in tech in 2018, they planned to future proof their business processes and systems. Unexpectedly, it also prepared them to handle a global pandemic that changed every norm for the campus. With shelter-in-place orders taking immediate effect, they needed to quickly set up a robust online learning platform - one with powerful analytics to track student success. And, for the times students and staff are on campus, a contact tracing application was essential for their safety. We’d like to offer a conversation with Scott Lee to tell you more about UC San Diego’s rapid transformation from a traditional, on-campus institution to one of the leading examples of remote learning, and the critical role data connectivity played in making this possible.
From legacy systems to microservices and back | Andera Gioia, Quantyca
From legacy systems to microservices and back | Andera Gioia, QuantycaHostedbyConfluent Legacy systems are the kings of our IT architecture. They rule the evolution of the technology ecosystem that hosts them thanks to the control they have acquired over time on core data and key business processes. Legacy systems however pose severe limits in addressing critical business needs, as well as seizing opportunities for future growth.
Digital Integration Hub (DIH) is a quite popular architecture to modernize legacy systems allowing the IT to evolve faster and better integrate modern technologies.
In this talk we’ll:
- Review DIH general architecture
- Illustrate different patterns to offload data from legacy systems to Kafka with pros and cons in term of consistency and latency
- Present different ways to consume data offloaded from legacy system to Kafka and their fit with different use cases
- Show how to close the loop and use Kafka and CQRS pattern to handle all the data moving backward from applications to legacy systems
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, AivenHostedbyConfluent When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...HostedbyConfluent Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...HostedbyConfluent If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.
Similar to EDA Governance Model: a multicloud approach based on GitOps | Alejandro Alija and Marco Laucelli, Galeo Tech (20)
Belgium & Luxembourg dedicated online Data Virtualization discovery workshop
Belgium & Luxembourg dedicated online Data Virtualization discovery workshopDenodo Watch full webinar here: https://bit.ly/33yYuQm
Data virtualization has become an essential part of enterprise data architectures, bridging the gap between IT and business users and delivering significant cost and time savings. This technology revolutionizes the way data is accessed, delivered, consumed and governed regardless of its format and location.
This 1.5 hour discovery session will show help you identify the benefits of this modern and agile data integration and management technology for your organisation.
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformation
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformationSlobodan Sipcic The document introduces an evolutionary event-driven architecture called the Enterprise Digital Transformation Platform (EDTP) for accelerating digital transformation. The EDTP is a 4-tier platform based on cloud, containers, microservices, events and streaming. It addresses challenges of data integration and decoupling through architectural concepts like event-driven design, microservices and templates. The EDTP provides full-stack deployment automation and microservice templating to accelerate development. Use cases from Toyota Financial Services are presented to demonstrate the EDTP's capabilities.
Business and IT agility through DevOps and microservice architecture powered ...
Business and IT agility through DevOps and microservice architecture powered ...Lucas Jellema IT needs to run in production in order to generate business value. DevOps is among other things a way of thinking focusing on production software. A business application requires a tailor made platform to generate business value. The combination of application and its platform is a DevOps product. The DevOps team has full responsibility for that product through its entire lifecycle.
The microservices architecture promises flexibility, scalability, and optimal use of compute resources. Via independent components with well-defined scope and responsibility, interface, and ownership that are evolved and managed in an automated DevOps process, this architecture leverages current technologies and hard-learned insights from past decades.
This session defines the objectives of Business with IT, of microservices and DevOps and introduces Containers and the container platform Kubernetes as crucial ingredients for making DevOps happen.
IBM Blockchain Platform - Architectural Good Practices v1.0
IBM Blockchain Platform - Architectural Good Practices v1.0Matt Lucas This document discusses architectural good practices for blockchains and Hyperledger Fabric performance. It provides an overview of key concepts like transaction processing in Fabric and performance metrics. It also covers optimizing different parts of the Fabric network like client applications, peers, ordering service, and chaincode. The document recommends using tools like Hyperledger Caliper and custom test harnesses for performance testing and monitoring Fabric deployments. It highlights lessons learned from real projects around reusing connections and load balancing requests.
DevOps Digital Transformation: A real life use case enabled by Alien4Cloud
DevOps Digital Transformation: A real life use case enabled by Alien4CloudCloudify Community View the presentation that went along with the September 5th Webinar on DevOps Digital Transformation with Alien4Cloud based on Cloudify.
Data platform modernization with Databricks.pptx
Data platform modernization with Databricks.pptxCalvinSim10 The document discusses modernizing a healthcare organization's data platform from version 1.0 to 2.0 using Azure Databricks. Version 1.0 used Azure HDInsight (HDI) which was challenging to scale and maintain. It presented performance issues and lacked integrations. Version 2.0 with Databricks will provide improved scalability, cost optimization, governance, and ease of use through features like Delta Lake, Unity Catalog, and collaborative notebooks. This will help address challenges faced by consumers, data engineers, and the client.
Re-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudCarter Wickstrom Are you planning to move existing applications to the cloud and want to avoid setbacks? These slides are from a webinar jointly presented by Atmosera and iTrellis, LLC. The webinar can help you find out how to assess your needs, plan out a migration and successfully operate your applications in a modern cloud environment. The webinar will provide the following answers:
* What re-platforming means and why you need to think about it
* How to take full advantage of a cloud such as Azure: agility, flexibility, and cost savings
* Lessons learned and best practices for planning a successful move to a modern cloud.
The full webinar playback URL is at https://www.atmosera.com/webinar-replatforming-application-cloud/
Neil-Leslie-CV
Neil-Leslie-CVNeil Leslie This CV summarizes Neil Leslie's experience as a database administrator and technical leader over 14 years at GE and previous roles. He has extensive experience leading teams and managing complex database migrations, from on-premise to cloud environments. His skills include Oracle, SQL Server, Linux, and AWS technologies. He is currently the CTO Cloud and Database Leader at GE Oil and Gas, where he helped insource technical resources and reduce costs through various initiatives.
Chugging Our Own "Craft Brew” – HPE’s Journey Towards Containers-as-a-Service...
Chugging Our Own "Craft Brew” – HPE’s Journey Towards Containers-as-a-Service...Docker, Inc. Hewlett Packard Enterprise has a vast IT organization that consists of 15k Dev + Ops professionals, operating in 11 countries, with hundreds of development teams working together on over 1000 projects. Millions of lines of code get changed every day, generating 20,000+ change request in an average year. And now, we’re on a journey to building a Docker environment for this massive organization, to serve the needs of our own multi-billion dollar enterprise. This session showcases our own learnings across multiple phases of our containerization project, with real life experiences from both the Dev and Ops perspectives. We’ll also talk about shared experiences from some of HPE’s customers. “Coding” our infrastructure with Docker, one application image, one deployment command, multiple deployment models – this is an “extreme” tale of how one of the world’s largest enterprises is fully embracing Docker.
Production-Ready Kubernetes: It's Not About Technology
Production-Ready Kubernetes: It's Not About TechnologyAntoine Craske Adopting cloud-native technologies can seem like a technological challenge, but it's far from being the only theme.
The organization, culture and people aspects is the main challenge to overcome resistance to change, drive value creation and accelerate the transformation.
This talk shares our journey in adopting a Cloud-native application stack with a Kubernetes PaaS, focusing on the transformation journey.
DITA Interoperability
DITA InteroperabilityKristen Eberlein Session at tcworld 2016. Organized by Kristen James Eberlein (Eberlein Consulting LLC); other participants were Joe Gollner (Gnostyx), George Bina (SyncroSoft), Jean-François Ameye (IXIASOFT), and Eliot Kimber (Contrext).
Oracle Forms Modernization Roadmap
Oracle Forms Modernization RoadmapKai-Uwe Möller How can Oracle Forms (or other legacy) applications be modernized to fit in a contemporary IT architecture? Trends, concepts and technologies are discussed.
Software Architecture and Architectors: useless VS valuable
Software Architecture and Architectors: useless VS valuableComsysto Reply GmbH The document discusses several cases where architecture decisions hurt software teams. It provides lessons learned from each case:
- Overly sharing code across domains can cause dependencies and fragility. Architectural boundaries should separate domains.
- Attempting to satisfy all use cases with a single highly configurable solution often satisfies no one. Specific solutions may be preferable.
- Microservices are not a universal solution and will not improve a flawed design alone. Modules should only be split across microservices at domain boundaries.
- Systems need to be designed to evolve dynamically over time as requirements change. Centralized control hinders this.
DevOps Operations Challenges
DevOps Operations ChallengesVijaya K The document discusses operations challenges in adopting DevOps practices. It notes that DevOps requires breaking down barriers between development and operations teams through collaboration, communication, and shared tools and processes. Some challenges include disagreements over tool ownership, pressure to adopt new technologies quickly, integrating legacy systems with cloud technologies, testing approaches for microservices, and aligning global teams with different time zones and priorities. Solutions proposed include automating delivery pipelines, deploying and testing changes frequently, monitoring all environments, and embracing changes in culture, processes, and technologies.
Implementing dev ops to face a two speed it architecture
Implementing dev ops to face a two speed it architectureDavide Veronese The document discusses implementing DevOps to address challenges of a "two speed IT" architecture with both innovative and industrialized parts. It proposes adopting a DevOps methodology to break down silos, address execution challenges, and bring startup flexibility to the enterprise. This includes cultural, architectural and DevOps transformations to balance agility and stability across edge applications, core applications and shared services. It provides an example roadmap for a phased DevOps adoption with initial proofs of concept and incremental implementations.
Coud computing
Coud computingBenila Mendus Cloud Computing is a term used to refer to a model of network computing where a program or application runs on a connected server or servers rather than a local computing device such as a PC, tablet or Smartphone.
www.ipsrglobal.com
Ravi namboori ppt
Ravi namboori pptravi namboori Cloud computing, or in simpler shorthand just "the cloud", also focuses on maximizing the effectiveness of the shared resources. Cloud resources are usually not only shared by multiple users but are also dynamically reallocated per demand. This can work for allocating resources to users. For example, a cloud computer facility that serves European users during European business hours with a specific application (e.g., email) may reallocate the same resources to serve North American users during North America's business hours with a different application (e.g., a web server). This approach should maximize the use of computing power thus reducing environmental damage as well since less power, air conditioning, rackspace, etc. are required for a variety of functions. With cloud computing, multiple users can access a single server to retrieve and update their data without purchasing licenses for different applications.
Ravi namboori-cloud computing
Ravi namboori-cloud computingravi namboori Cloud computing relies on sharing of resources to achieve coherence and economies of scale, similar to a utility (like the electricity grid) over a network.[1] At the foundation of cloud computing is the broader concept of converged infrastructure and shared services.
Cloud computing, or in simpler shorthand just "the cloud", also focuses on maximizing the effectiveness of the shared resources. Cloud resources are usually not only shared by multiple users but are also dynamically reallocated per demand. This can work for allocating resources to users. For example, a cloud computer facility that serves European users during European business hours with a specific application (e.g., email) may reallocate the same resources to serve North American users during North America's business hours with a different application (e.g., a web server). This approach should maximize the use of computing power thus reducing environmental damage as well since less power, air conditioning, rackspace, etc. are required for a variety of functions. With cloud computing, multiple users can access a single server to retrieve and update their data without purchasing licenses for different applications.
Amit_Kumar_CV
Amit_Kumar_CVAmit Kumar This document contains a summary of Amit Kumar's professional experience and qualifications. He has over 9 years of IT experience, including 8 years of data warehouse and business intelligence experience. Currently he works as a data architect at Capgemini, leading a team of 12 on an oil and gas equipment install base project. He has extensive experience designing and developing data integration solutions using tools like Informatica, Hadoop, and SQL.
SCM Migration Webinar - English
SCM Migration Webinar - EnglishCollabNet In this informative webinar, learn how migrating from a proprietary SCM solution such as Rational® ClearCase®, Serena PVCS®, CA® Harvest, etc., to Subversion or Git will make an impact on your organization and/or enterprise.
Join us as we take the lessons we've learned from successfully migrating thousands of users to today's market leading SCM solutions, and provide you with best practices in building an actionable business case and conducting a smooth transition.
Key Takeaways:
Build a business case to adopt Git and/or Subversion in your organization
How CollabNet's TeamForge platform can provide the enterprise capabilities to enable Git and/or Subversion in your enterprise
Our recommended migration strategy proven with thousands of users
Considerations for extending your SCM solution
Ad
More from HostedbyConfluent (20)
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent "In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way.
Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks.
Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case.
Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."
Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent "While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics.
Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent "Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights.
We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string.
Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring.
The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent "Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service.
While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications.
The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection.
Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent "Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output.
Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing.
In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent "In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium.
This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption.
By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent What is tiered storage and what is it good for? After this session you will know how to leverage the tiered storage feature to enable longer retention than the storage attached to brokers allows. You will get acquainted with the different configuration options and know what to expect when you enable the feature, like for example when will the first upload to the remote object storage take place.
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent "Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work.
However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business.
You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want.
Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware.
What you’ll take away from our talk:
- What were the pain points in the previous environment
- How we transitioned to Confluent without service downtime
- Creating a self-service stream processing portal built on top of Connect and ksqlDB
- Use case of stream process portal"
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent "Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics.
Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent "In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application.
At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent "There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option!
In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent "In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met.
I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines.
Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent "Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many.
LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data.
Large language models for your streaming application - explained with a little maths and a lot of pub stories"
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent "Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them.
This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent "In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance.
- Introduction to Kafka producer internal and workflow
- Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance
- Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient.
- Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees.
- Q&A session with attendees to address specific questions and concerns."
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondHostedbyConfluent "Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element.
In this talk, we’ll:
1. see why data contracts are so important but also difficult to implement;
2. identify the characteristics of a well-designed data contract:
discuss the anatomy of a data contract, its main elements and, how to formally describe them;
3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent "In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications.
We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications.
Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent "Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you!
[1] https://debezium.io/"
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent "Separation of compute and storage has become the de-facto standard in the data industry for batch processing.
The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world.
In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.
Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it.
This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."
Ad
Recently uploaded (20)
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock Building 10x Organizations with Modern Productivity Metrics
10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’
Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them.
But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?
Heap, Types of Heap, Insertion and Deletion
Heap, Types of Heap, Insertion and DeletionJaydeep Kale This pdf will explain what is heap, its type, insertion and deletion in heap and Heap sort
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell With expertise in data architecture, performance tracking, and revenue forecasting, Andrew Marnell plays a vital role in aligning business strategies with data insights. Andrew Marnell’s ability to lead cross-functional teams ensures businesses achieve sustainable growth and operational excellence.
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp This is the keynote of the Into the Box conference, highlighting the release of the BoxLang JVM language, its key enhancements, and its vision for the future.
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok In today's fast-paced retail environment, efficiency is key. Every minute counts, and every penny matters. One tool that can significantly boost your store's efficiency is a well-executed planogram. These visual merchandising blueprints not only enhance store layouts but also save time and money in the process.
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3 The latest updates on Manifest's pre-seed stage progress.
TrsLabs - Fintech Product & Business Consulting
TrsLabs - Fintech Product & Business ConsultingTrs Labs Hybrid Growth Mandate Model with TrsLabs
Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant.
An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices.
Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company
Talk to us & Unlock the competitive advantage
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how:
• Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules.
• Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance.
• Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity.
• Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications.
• Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market.
With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications.
Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market
For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family
How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?Daniel Lehner 𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨?
Everyone’s talking about AI but is anyone really using it to create real value?
Most companies want to leverage AI. Few know 𝗵𝗼𝘄.
✅ What exactly should you ask to find real AI opportunities?
✅ Which AI techniques actually fit your business?
✅ Is your data even ready for AI?
If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.
How analogue intelligence complements AI
How analogue intelligence complements AIPaul Rowe
Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results.
News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc. Impelsys provided a robust testing solution, leveraging a risk-based and requirement-mapped approach to validate ICU Connect and CritiXpert. A well-defined test suite was developed to assess data communication, clinical data collection, transformation, and visualization across integrated devices.
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies In late April 2025, a significant portion of Europe, particularly Spain, Portugal, and parts of southern France, experienced widespread, rolling power outages that continue to affect millions of residents, businesses, and infrastructure systems.
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john Analyze the growth of meme coins from mere online jokes to potential assets in the digital economy. Explore the community, culture, and utility as they elevate themselves to a new era in cryptocurrency.
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityriccardosl1 Cyber awareness training educates employees on risk associated with internet and malicious emails
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBrian McKeiver May 2nd, 2025 talk at StirTrek 2025 Conference.
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat
EDA Governance Model: a multicloud approach based on GitOps | Alejandro Alija and Marco Laucelli, Galeo Tech
- 1. Kafka Summit 2021 EDA Governance Model: A multi-cloud approach based on GitOps Alejandro Alija & MarcoLaucelli
- 2. Galeo in one sentence We build digital platforms to be the agile fabric where digital products are conceived, developed, and scaled. We are hands-on. We deliver.
- 3. Define and govern the new digital architecture for the future. The challenge Standardized platform 01 Cost Control & allocation 03 Service Catalogue 05 Asset reuse 04 Project Flexibility & Shared nothing 02 Governed lifecycle 06
- 4. As promoters of a new architecture, the team put a lot of focus on simplifying developer’s experience. • Is EDA approach suitable for my project? • How should the infrastructure for my project be defined and provisioned? • How should the EDA components of my system be described and deployed? • How can I access and reuse existing assets? The challenge: DX
- 5. Everything together: governance 1. Bureaucraticoverhead: mostGovernance documentationmodelsrequireadditional documentationeffortsfromdevelopersthatusually don’tpay back fortheir specificduties. 2. Cataloguesdiffer from reality: independentdocumentationfor governanceand forgovernance andcatalogues end updifferingfromwhatis actuallybeingdeployed. Reuseand standardizationgetimpacted.Apart fromgovernanceitself. Based on the customer experiences two main problems needed to be avoided
- 6. The governance approach. The underlyingidea isthatevery governancestep will beconsideredasapartofthewhole deploymentprocesses,andso theyshouldbe integrated into a single orchestrated pipeline thatwill be ableto covergovernance,supervision, approval anddeploymenttasks asa partofa unifiedprocess. The orchestrationofthe processis basedon aunified,machinereadable documentation modelthatcontainsthe requiredinformationtovalidateservice design andcomponent specifications.Thisunderlying unifieddocumentationmodel allows executing theoverall catalogueanddeploymenttasksinasynchronised anduniform way.
- 7. We defined a EDA Governance Framework based on an operational support for the whole lifecycle management. EDA Governance framework Theprocessisminimallyintermediated,sohumaninterventionisonlyrequiredforsupervision andapprovaltasks.Nomanualinterventionon thecataloguesortheactualEDA environmentis required.
- 8. The Catalogue. EDA Governance framework Theprocessisminimallyintermediated,sohumaninterventionisonlyrequiredforsupervision andapprovaltasks.Nomanualinterventionon thecataloguesortheactualEDA environment isrequired. Key component of the EDA Gov: • Keeps record of the deployed asset and services. • Acts as theoverall orchestrator of the deployment process. For each asset it keeps trackof: • Infrastructure • Topologies and Topics • Metadata • Security and access configuration Consolidated in the corporate Git Repos.
- 9. The documental model. Based on standardized description templates that are readable for both the actual deployement pipelines and the catalogue reporter. Non-functionalcomponent descriptionsand infrastructure specification. Descriptionfothe wholetopology includingproducers,consumersand othereventualcomponents.
- 10. EDA Governance procedures EDA Infrastructure & Project Procedures • Project sign-up and approval procedure: • Project approval procedure • Request a Confluent Environment • Request a Confluent Cluster EDA Service & Component Procedures • Request access to an existing Confluent resource: topic, ksqlDB table, … • Request modification to an existing resource: topic, ksqlDB table, … • Accept Access Request by a third-party procedure • Request and Create a new topology
- 11. Catalogue & Procedures Based on corporate Git repositories and governance CI/CD procedures, that allowexecute and synchonyze development and governance procedures witouth incrementing bureaucratic efforts.
- 12. Demo Developer My NewProject - Topology_description.yaml CI/CDPipeline - ACLscreationinConfluentCloud Webhook –Pull request Trigger Confluent CLI– ACLs creation Producers Consumers
- 13. Conclusions 1. GitOps&CI/CDfor topologies: is a very convenient modelfor OperationalGovernance 2. Cataloguesstaysup todate: based on independentlawyers 3. Platform evolution andcompletion (API,MS) is easilyintegratedwiththesame Governance model 4. Reuse is facilitated: as theinformationis up2dateand theprocedures are very lightweight procedures are very lightweight
- 14. EDA Governance Model: Thanks [email protected] [email protected] www.galeo.tech WE ARE HIRING