“Alexa, be quiet!”: End-to-end near-real time model building and evaluation in Amazon Alexa

Aug 9, 2022Download as PPTX, PDF0 likes258 views

Flink Forward San Francisco 2022. To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy. by Aansh Shah

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates.
“Alexa, be quiet!”: End-to-end
near-real time model building and
evaluation in Amazon Alexa
Aansh Shah
Software Development Engineer
Secure AI Foundations (Alexa AI), Amazon

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
Overview
Continual Learning
Platform Overview
Kinesis Data Analytics and Flink
Use Case
Challenges
Roadmap
2

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
What is not Continual Learning?
3

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
What is Continual Learning?
4

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
Why?
• Adaptability
• Cost
5

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
How does the platform work?
6

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
Why Flink and Kinesis Data Analytics?
• Generate windows on streaming data
• Easily transform raw data stream into other representations
• Focus on use case logic and not infrastructure
7

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
Feedback Detection
8

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
What challenges do we face today?
• Abstraction and simplification
• Dataflow transparency
9

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
What’s next?
• Doubling down on our success with Kinesis Data Analytics and Flink
to replace components within our data storage
10

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates. 11

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates. 12

“ALEXA, BE QUIET!”: END-TO-END NEAR-REAL TIME MODEL BUILDING AND EVALUATION IN AMAZON ALEXA
© 2022, Amazon Web Services, Inc. or its affiliates.
Thank you!
© 2022, Amazon Web Services, Inc. or its affiliates.

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Autoscaling Flink with Reactive ModeFlink Forward

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Flink powered stream processing platform at PinterestFlink Forward

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Evening out the uneven: dealing with skew in FlinkFlink Forward

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

Changelog Stream Processing with Apache FlinkFlink Forward

Flink Forward San Francisco 2022. The world is constantly changing. Data is continuously produced and thus should be consumed in a similar fashion by enterprise systems. Only this enables real-time decisions at scale. Message logs such as Apache Kafka can be found in almost every architecture, while databases and other batch systems still provide the foundation. Change Data Capture (CDC) propagates changes downstream. In this talk, we will highlight what it means to be a general data processor and how Flink can act as an integration hub. We present the current state of Flink and how it can power various use cases on both finite and infinite streams. We demonstrate Flink's SQL engine as a changelog processor that is shipped with an ecosystem tailored to process CDC data and maintain materialized views. We will use Kafka as an upsert log, Debezium for connecting to databases, and enrich streams of various sources. Finally, we will combine Flink's Table API with DataStream API for event-driven applications beyond SQL. by Timo Walther

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Flink Forward

Flink Forward San Francisco 2022. Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way. by Jeff Chao

Producer Performance Tuning for Apache KafkaJiangjie Qin

Tame the small files problem and optimize data layout for streaming ingestion...Flink Forward

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Extending Flink SQL for stream processing use casesFlink Forward

1. For streaming data, Flink SQL uses STREAMs for append-only queries and CHANGELOGs for upsert queries instead of tables. 2. Stateless queries on streaming data, such as projections and filters, result in new STREAMs or CHANGELOGs. 3. Stateful queries, such as aggregations, produce STREAMs or CHANGELOGs depending on whether they are windowed or not. Join queries between streaming sources also result in STREAM outputs.

Deploying Flink on Kubernetes - David AndersonVerverica

Practical learnings from running thousands of Flink jobsFlink Forward

Flink Forward San Francisco 2022. Task Managers constantly running out of memory? Flink job keeps restarting from cryptic Akka exceptions? Flink job running but doesn’t seem to be processing any records? We share practical learnings from running thousands of Flink Jobs for different use-cases and take a look at common challenges they have experienced such as out-of-memory errors, timeouts and job stability. We will cover memory tuning, S3 and Akka configurations to address common pitfalls and the approaches that we take on automating health monitoring and management of Flink jobs at scale. by Hong Teoh & Usamah Jassat

One sink to rule them all: Introducing the new Async SinkFlink Forward

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

The top 3 challenges running multi-tenant Flink at scaleFlink Forward

Apache Flink is the foundation for Decodable's real-time SaaS data platform. Flink runs critical data processing jobs with strong security requirements. In addition, Decodable has to scale to thousands of tenants, power various use cases, provide an intuitive user experience and maintain cost-efficiency. We've learned a lot of lessons while building and maintaining the platform. In this talk, I'll share the top 3 toughest challenges building and operating this platform with Flink, and how we solved them.

Introduction to Kafka Cruise ControlJiangjie Qin

Introducing the Apache Flink Kubernetes OperatorFlink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

kafkaAmikam Snir

Kafka is an open-source distributed commit log service that provides high-throughput messaging functionality. It is designed to handle large volumes of data and different use cases like online and offline processing more efficiently than alternatives like RabbitMQ. Kafka works by partitioning topics into segments spread across clusters of machines, and replicates across these partitions for fault tolerance. It can be used as a central data hub or pipeline for collecting, transforming, and streaming data between systems and applications.

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Flink Forward

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Apache KafkaDiego Pacheco

Apache Kafka is a high-throughput distributed messaging system that allows for both streaming and offline log processing. It uses Apache Zookeeper for coordination and supports activity stream processing and real-time pub/sub messaging. Kafka bridges the gaps between pure offline log processing and traditional messaging systems by providing features like batching, transactions, persistence, and support for multiple consumers.

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent

RocksDB is the default state store for Kafka Streams. In this talk, we will discuss how to improve single node performance of the state store by tuning RocksDB and how to efficiently identify issues in the setup. We start with a short description of the RocksDB architecture. We discuss how Kafka Streams restores the state stores from Kafka by leveraging RocksDB features for bulk loading of data. We give examples of hand-tuning the RocksDB state stores based on Kafka Streams metrics and RocksDB’s metrics. At the end, we dive into a few RocksDB command line utilities that allow you to debug your setup and dump data from a state store. We illustrate the usage of the utilities with a few real-life use cases. The key takeaway from the session is the ability to understand the internal details of the default state store in Kafka Streams so that engineers can fine-tune their performance for different varieties of workloads and operate the state stores in a more robust manner.

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Flink Forward

Netflix’s playback data records every user interaction with video on the service, from trailers on the home page to full-length movies. This is a critical dataset with high volume that is used broadly across Netflix, powering product experiences, AB test metrics, and offline insights. In processing playback data, we depend heavily on event-time partitioning to handle a long tail of late arriving events. In this talk, I’ll provide an overview of our recent implementation of generic event-time partitioning on high volume streams using Apache Flink and Apache Iceberg (Incubating). Built as configurable Flink components that leverage Iceberg as a new output table format, we are now able to write playback data and other large scale datasets directly from a stream into a table partitioned on event time, replacing the common pattern of relying on a post-processing batch job that “puts the data in the right place”. We’ll talk through what it took to apply this to our playback data in practice, as well as challenges we hit along the way and tradeoffs with a streaming approach to event-time partitioning.

Processing Semantically-Ordered Streams in Financial ServicesFlink Forward

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Apache Flink in the Cloud-Native EraFlink Forward

Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

Kafka Streams State Stores Being Persistentconfluent

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop. It's also enabling many real-time system frameworks and use cases. Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API. Also talk about the best practices involved in running a producer/consumer. In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects. We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing Kafka ACLs and monitoring Consumer offsets.

The Current State of Table API in 2022Flink Forward

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

A Deep Dive into Kafka Controllerconfluent

Presentation at Strata Data Conference 2018, New York The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure. Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker. Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.

Single View of Dataconfluent

The document discusses strategies for establishing a single view of data across an organization. It recommends bringing all organizational data together in a single platform to provide a comprehensive and consistent view for all stakeholders. This allows gaining new insights from data that may have previously been overlooked or unused. It also discusses challenges like data silos and quality issues and how democratizing data access can improve business outcomes through greater collaboration and innovation. The document concludes by examining emerging technologies at the leading edge of data and AI like AI assistants and how establishing a data mesh architecture can help decentralize data management.

현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...Amazon Web Services Korea

The document discusses the development journey of an AI-powered unmanned convenience store by Hyundai IT&E and AWS Prototyping. It describes how they used agile methodologies like Scrum and built the store as a modern cloud application using AWS services. This allowed them to iteratively implement capabilities like detecting and tracking customers, building ML pipelines, and understanding purchase behavior to open an unmanned trial store.

More Related Content

What's hot (20)

Producer Performance Tuning for Apache KafkaJiangjie Qin

Tame the small files problem and optimize data layout for streaming ingestion...Flink Forward

Extending Flink SQL for stream processing use casesFlink Forward

Deploying Flink on Kubernetes - David AndersonVerverica

Practical learnings from running thousands of Flink jobsFlink Forward

One sink to rule them all: Introducing the new Async SinkFlink Forward

The top 3 challenges running multi-tenant Flink at scaleFlink Forward

Introduction to Kafka Cruise ControlJiangjie Qin

Introducing the Apache Flink Kubernetes OperatorFlink Forward

kafkaAmikam Snir

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Flink Forward

Apache KafkaDiego Pacheco

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Flink Forward

Processing Semantically-Ordered Streams in Financial ServicesFlink Forward

Apache Flink in the Cloud-Native EraFlink Forward

Kafka Streams State Stores Being Persistentconfluent

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

The Current State of Table API in 2022Flink Forward

A Deep Dive into Kafka Controllerconfluent

Producer Performance Tuning for Apache KafkaJiangjie Qin

Tame the small files problem and optimize data layout for streaming ingestion...Flink Forward

Extending Flink SQL for stream processing use casesFlink Forward

Deploying Flink on Kubernetes - David AndersonVerverica

Practical learnings from running thousands of Flink jobsFlink Forward

One sink to rule them all: Introducing the new Async SinkFlink Forward

The top 3 challenges running multi-tenant Flink at scaleFlink Forward

Introduction to Kafka Cruise ControlJiangjie Qin

Introducing the Apache Flink Kubernetes OperatorFlink Forward

kafkaAmikam Snir

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Flink Forward

Apache KafkaDiego Pacheco

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Flink Forward

Processing Semantically-Ordered Streams in Financial ServicesFlink Forward

Apache Flink in the Cloud-Native EraFlink Forward

Kafka Streams State Stores Being Persistentconfluent

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

The Current State of Table API in 2022Flink Forward

A Deep Dive into Kafka Controllerconfluent

Similar to “Alexa, be quiet!”: End-to-end near-real time model building and evaluation in Amazon Alexa (20)

Single View of Dataconfluent

현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...Amazon Web Services Korea

Private Equity Value Creation Carve Outs, Divestitures and mergersTom Laszewski

20210608 - Desarrollo de aplicaciones en la nubeMarcia Villalba

The document provides an overview of developing applications on AWS using AWS Amplify. It discusses how AWS Amplify can be used across the full development lifecycle, including developing, deploying, and managing applications. It demonstrates how to use AWS Amplify's command line interface and administration UI to add features like authentication, data storage, continuous integration/deployment, and machine learning to a sample application. AWS Amplify aims to simplify developing full-stack applications on AWS by handling tasks like provisioning infrastructure and integrating AWS services.

Speed up data preparation for ML pipelines on AWSData Science Milan

Abstract: Data preparation and modelling are the activities that take most of the time in a typical data scientist workday. In this session we’ll see how AWS services for Analytics and data management can be effectively used and integrated in AI/ML pipelines. We’ll focus on AWS Glue, AWS Glue DataBrew and AWS Data Wrangler with a bit of theory and hands-on demos. Bio: Francesco Marelli is a senior solutions architect at Amazon Web Services. He has lived and worked in UK, italy, Switzerland and other countries in EMEA. He is specialized in the design and implementation of Analytics, Data Management and Big Data systems. Francesco also has a strong experience in systems integration and design and implementation of applications. Topics: machine learning pipelines, AWS, cloud.

Architecting Web Applications for the Cloud - Design Principles and Practical...Adnene Guabtni

Leveraging Generative AI to Drive Nonprofit InnovationTechSoup

Confluent_AWS_ImmersionDay_Q42023.pdfAhmed791434

This document outlines an agenda for a webinar on building secure, event-driven microservices with Confluent Cloud on AWS. The agenda includes presentations on building modern streaming analytics with Confluent on AWS, event streaming made easy with Confluent, and a lab on building end-to-end streaming data pipelines with Confluent Cloud. The hosts for the webinar are Ahmed Zamzam from Confluent and Nuno Barreto from AWS.

Build real-time streaming data pipelines to AWS with Confluentconfluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

AWS DevDay Berlin - Automating building blocks choices you will face with con...Cobus Bernard

Best Practices for Using AWS CreditsTechSoup

Module 3 - QuickSight OverviewLam Le

Amazon QuickSight is a business intelligence service that allows users to connect to data sources, create interactive dashboards, and securely share them across organizations. It offers auto-scaling, high availability, integration with AWS services, and pay-per-use pricing starting at $5/month for readers. QuickSight provides machine learning capabilities like anomaly detection and forecasting. It also allows embedding dashboards in applications. Customers like Capital One, Comcast, and the NFL use QuickSight for self-service analytics, embedded analytics, and delivering insights to large numbers of users through its reader role and usage-based pricing.

Build and Run Streaming Applications with Apache Flink and Amazon Kinesis Dat...Flink Forward

Stream processing facilitates the collection, processing, and analysis of real-time data and enables the continuous generation of insights and quick reactions to emerging situations. Yet, despite these advantages compared to traditional batch-oriented analytics applications, streaming applications are much more challenging to operate. Some of these challenges include the ability to provide and maintain low end-to-end latency, to seamlessly recover from failure, and to deal with a varying amount of throughput. We all know and love Flink to take on those challenges with grace. In this session, we explore an end to end example that shows how you can use Apache Flink and Amazon Kinesis Data Analytics for Java Applications to build a reliable, scalable, and highly available streaming applications. We discuss how you can leverage managed services to quickly build Flink based streaming applications and show managed services can help to substantially reduce the operational overhead that is required to run the application. We also review best practices for running streaming applications with Apache Flink on AWS. So you will not only see how to actually build streaming applications with Apache Flink on AWS, you will also learn how leveraging managed services can help to reduce the overhead that is usually required to build and operate streaming applications to a bare minimum.

Data Summer Conf 2018, “Build, train, and deploy machine learning models at s...Provectus

This document summarizes a presentation about Amazon SageMaker, AWS's fully managed machine learning service. It discusses SageMaker's key components like built-in algorithms, bringing your own algorithms or scripts, notebook instances, SDKs and the AWS console. It also covers how SageMaker handles distributed training, hyperparameter tuning, hosting models for inference, and auto-scaling of hosted models. The goal of SageMaker is to make machine learning accessible to developers and data scientists and allow them to focus on the science rather than infrastructure setup and management.

AWS Reinvent 2020 - Recap Amazon Builder's Library sessionGuillaume Marchand

AWS DevDay Cologne - Automating building blocks choices you will face with co...Cobus Bernard

AWS DevDay Vienna - Automating building blocks choices you will face with con...Cobus Bernard

콘텐츠는 여전히 왕이다 - 클라우드를 통한 미디어 자산 관리와 공급망 혁신, SM Entertainment의 Digital Library...Amazon Web Services Korea

지금까지의 미디어 워크플로우에서 벗어나 확장성 높은 미디어 아카이빙부터 인공지능/기계 학습을 활용한 자동 메타데이터 생성까지, 클라우드로 가능한 미디어 자산 관리와 공급망의 현재와 미래를 소개합니다. 더불어, SM 엔터테인먼트의 Digital Library 사례를 통해 클라우드 미디어 공급망이 가져다주는 새로운 비즈니스 인사이트와 기회를 살펴봅니다.

DevConZM - Modern Applications Development in the CloudCobus Bernard

The document discusses developing modern applications in the cloud. It covers topics like building applications with serverless technologies, modeling infrastructure with code, structuring apps as microservices, containerization, continuous integration/delivery (CI/CD), and service meshes. The overall message is that cloud-native design approaches like these can help businesses rapidly innovate and gain a competitive advantage.

Frome Code to Cloud: Exploring AWS CDK for Infrastructure ManagementSujay Pillai

This document contains a presentation on AWS CDK (Cloud Development Kit). It discusses infrastructure as code, provides an overview and demo of AWS CDK, and lists some real-world applications and use cases. The agenda includes understanding AWS CDK, getting started, a demo, and real world uses. It also discusses infrastructure as code tools and the three steps to implement infrastructure as code across an enterprise.