Migrating pipelines into Docker
3 likes1,355 views
This document discusses Spotify's migration of data pipelines to Docker. It provides background on Spotify growing from 50 to 1000 engineers and the challenges of scaling their big data infrastructure. Spotify adopted Docker to help solve packaging and dependency issues, moving pipelines from cron jobs to a REST API and Docker images. Docker is allowing Spotify to transparently migrate their on-premise Hadoop cluster to Google Cloud, handling over 100 petabytes of data and growing.
1 of 22
Downloaded 30 times
Ad
Recommended
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Flink: the 4G of Big Data Analytics FrameworksDataWorks Summit/Hadoop Summit This document provides an overview of Apache Flink and discusses why it is suitable for real-world streaming analytics. The document contains an agenda that covers how Flink is a multi-purpose big data analytics framework, why streaming analytics are emerging, why Flink is suitable for real-world streaming analytics, novel use cases enabled by Flink, who is using Flink, and where to go from here. Key points include Flink innovations like custom memory management, its DataSet API, rich windowing semantics, and native iterative processing. Flink's streaming features that make it suitable for real-world use include its pipelined processing engine, stream abstraction, performance, windowing support, fault tolerance, and integration with Hadoop.
Running Spark in Production
Running Spark in ProductionDataWorks Summit/Hadoop Summit This document discusses best practices for running Spark in production. It begins with introductions from the presenters and an overview of Spark deployment modes on YARN. The main topics covered are Spark security using Kerberos authentication and authorization, communication channels and encryption in YARN cluster mode, common issues, and performance tuning. For performance, it recommends choosing executor and task sizes to balance efficiency and overhead, and increasing task parallelism to mitigate data skew problems. The goal is to understand workload patterns and monitor behavior to effectively tune Spark for different situations.
Securing Spark Applications
Securing Spark ApplicationsDataWorks Summit/Hadoop Summit This document discusses securing Spark applications. It covers encryption to protect data in transit and at rest, authentication using Kerberos to identify users, and authorization for access control through tools like Sentry and a proposed RecordService. While Spark can be secured today by leveraging Hadoop security, continued work is needed for easier encryption, improved Kerberos support for long-running jobs, and row/column-level authorization beyond file permissions.
Large-Scale Stream Processing in the Hadoop Ecosystem 
Large-Scale Stream Processing in the Hadoop Ecosystem DataWorks Summit/Hadoop Summit The document discusses large-scale stream processing in the Hadoop ecosystem. It provides examples of real-time stream processing use cases for computing player statistics and analyzing telco network data. It then summarizes several open source stream processing frameworks, including Apache Storm, Samza, Kafka Streams, Spark, Flink, and Apex. Key aspects like programming models, fault tolerance methods, and performance are compared for each framework. The document concludes with recommendations for further innovation in areas like dynamic scaling and batch integration.
Streaming in the Wild with Apache Flink
Streaming in the Wild with Apache FlinkDataWorks Summit/Hadoop Summit This document summarizes a presentation about streaming data processing with Apache Flink. It discusses how Flink enables real-time analysis and continuous applications. Case studies are presented showing how companies like Bouygues Telecom, Zalando, King.com, and Netflix use Flink for applications like monitoring, analytics, and building a stream processing service. Flink performance is discussed through benchmarks, and features like consistent snapshots and dynamic scaling are mentioned.
Tame that Beast
Tame that BeastDataWorks Summit/Hadoop Summit This document discusses improving the reliability and availability of Hadoop clusters. It notes that while Hadoop is taking on more database-like features, the uptime of many Hadoop clusters and lack of SLAs is still an afterthought. It proposes separating computing and storage to improve availability like cloud Hadoop offerings do. It also suggests building KPIs and monitoring around Hadoop clusters similar to how many companies monitor data warehouses. Centralizing Hadoop infrastructure management into a "Big Data as a Service" model is presented as another way to improve reliability.
Apache Hive 2.0: SQL, Speed, Scale
Apache Hive 2.0: SQL, Speed, ScaleDataWorks Summit/Hadoop Summit Apache Hive 2.0 provides major new features for SQL on Hadoop such as:
- HPLSQL which adds procedural SQL capabilities like loops and branches.
- LLAP which enables sub-second queries through persistent daemons and in-memory caching.
- Using HBase as the metastore which speeds up query planning times for queries involving thousands of partitions.
- Improvements to Hive on Spark and the cost-based optimizer.
- Many bug fixes and under-the-hood improvements were also made while maintaining backwards compatibility where possible.
Machine Learning in the IoT with Apache NiFi
Machine Learning in the IoT with Apache NiFiDataWorks Summit/Hadoop Summit Across the globe energy systems are changing, creating unprecedented challenges for the organisations tasked with ensuring the lights stay on. In the UK, National Grid is facing shrinking margins, looming capacity shortages and unpredictable peaks and troughs in energy supply caused by increasing levels of renewable penetration. Open Energi uses its IoT technology to unlock demand-side capacity - from industrial equipment, co-generation and batery storage systems - creating a smarter grid; one that is cleaner, cheaper, more secure and more efficient.
I'll talk about how we use Apache Nifi to orchestrate and coordinate Machine Learning microservices that operate on streams of data coming from IoT devices, providing a layer of fault-tolerance and traceability. With built-in retry logic, backpressure and clustering, Nifi helps us keep hard problems away from our code. It comes with processors that integrate with our cloud provider of choice (Microsoft Azure), fitting seamlessly into our processing pipeline.Finally, its straightforward graphical interface makes it easy enough to use that any team member can step in and troubleshoot a flow with little training.
Scale-Out Resource Management at Microsoft using Apache YARN
Scale-Out Resource Management at Microsoft using Apache YARNDataWorks Summit/Hadoop Summit This document discusses Microsoft's use of Apache YARN for scale-out resource management. It describes how YARN is used to manage vast amounts of data and compute resources across many different applications and workloads. The document outlines some limitations of YARN and Microsoft's contributions to address those limitations, including Rayon for improved scheduling, Mercury and Yaq for distributed scheduling, and work on federation to scale YARN across multiple clusters. It provides details on the implementation and evaluation of these contributions through papers, JIRAs, and integration into Apache Hadoop releases.
Hive on spark is blazing fast or is it final
Hive on spark is blazing fast or is it finalHortonworks This presentation was given at the Strata + Hadoop World, 2015 in San Jose.
Apache Hive is the most popular and most widely used SQL solution for Hadoop. To keep pace with Hadoop’s increasingly vital role in the Enterprise, Hive has transformed from a batch-only, high-latency system into a modern SQL engine capable of both batch and interactive queries over large datasets. Hive’s momentum is accelerating: With Spark integration and a shift to in-memory processing on the horizon, Hive continues to expand the boundaries of Big Data.
In this talk the speakers examined Hive performance, past, present and future. In particular they looked at Hive’s origins as a petabyte scale SQL engine.
Through some numbers and graphs, they showed how Hive became 100x faster by moving beyond MapReduce, by vectorizing execution and by introducing a cost-based optimizer.
They detailed and discussed the challenges of scalable SQL on Hadoop.
The looked into Hive’s sub-second future, powered by LLAP and Hive on Spark.
And showed just how fast Hive on Spark really is.
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in HiveDataWorks Summit/Hadoop Summit We discuss the current state of LLAP (Live Long and Process) – the concurrent sub-second execution of analytical queries engine for Hive 2.0. LLAP is a hybrid execution model that enables performance improvement in and across queries, such as caching of columnar data with cache coherence and intelligent eviction for disaggregated storage models (like S3, Isilon, Azure), JIT-friendly operator pipelines, asynchronous I/O, data pre-fetching and multi-threaded processing. LLAP features robust machine and service failure tolerance achieved by building on top of the time-tested fault tolerant subsystems, as well as a concurrency-directed design that achieves high utilization with low latency via resource sharing, reducing overheads for multiple queries, and enabling the system to preempt tasks of lower priority without failing any query in-flight. The talk also aims to cover the novel deployment model required for hybrid execution. The elasticity demands of the system are served by a long-lived YARN service interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework for query execution. We discuss the current state of the project, performance numbers, deployment and usage strategy, as well as future work, including how LLAP fits into a unified secure DataFrame access layer.
Network for the Large-scale Hadoop cluster at Yahoo! JAPAN
Network for the Large-scale Hadoop cluster at Yahoo! JAPANDataWorks Summit/Hadoop Summit Yahoo Japan transitioned their Hadoop cluster network architecture over time to address problems and scale needs. They moved from a stack architecture to an L2 fabric to an IP CLOS architecture. The IP CLOS architecture improved scalability, high availability, and reduced operating costs by allowing over 10,000 nodes with 100-200Gbps uplinks per rack and an oversubscription ratio of 1.25:1. This solved problems around switch failures, BUM traffic loads, decommissioning limitations, and scale-out limits they previously faced.
Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in ProductionDataWorks Summit/Hadoop Summit This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
The Future of Apache Ambari
The Future of Apache AmbariDataWorks Summit Apache Ambari is an extensible framework that simplifies provisioning, managing and monitoring Hadoop clusters. Apache Ambari was built on a standardized stack-based operations model. Stacks wrap services of all shapes and sizes with a consistent definition and lifecycle-control layer; thereby providing a consistent approach for managing and monitoring the services. This also provided a natural extension point for operators and the community to bring in their own add-on services and “plug-in” the new services into the stack.
However, one of the fundamental limitations of the current Apache Ambari architecture has been that there is a strong one-on-one coupling between entities. For instance, a cluster is tied to a single stack and a Hadoop operator can only deploy services defined in that stack, a cluster can have only a single instance of a service and a host can have only a single instance of a component. Taking into consideration various use case scenarios that cannot be enabled due to these limitations there is a growing need to revamp the Ambari architecture.
In this talk, we propose a revamped Apache Ambari architecture that will open up the floodgates for a wide range of scenarios that wouldn’t have been possible thus far. We will focus the discussion on a new mpack-based operations model that will replace the stack-based operations model. A management package is a self-contained deployment artifact that includes all the details for deploying, managing and upgrading a set of services bundled in the package. A third-party provider can also build their own management package containing their custom services. This eliminates the need to plug-in their services into a stack and also can define their own upgrade story for these custom services. A Hadoop operator will be able to deploy a Hadoop cluster with a mix of services across multiple packages instead of being limited to a single stack. For example, it would be possible to deploy a cluster with HDFS from HDP and NIFI from HDF.
Further, we will also discuss about the architectural changes needed to enable a multi instance architecture in future Ambari releases to support deploying multiple instances of a service in a cluster, deploying multiple instances of a component on a host as well as future proofing the Ambari architecture to leverage some of the advancements happening in the Hadoop community like YARN services (YARN-4692). We will wrap up the conversation with a brief overview of other improvements planned for future releases of Ambari.
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)Chris Nauroth Today's typical Apache Hadoop deployments use HDFS for persistent, fault-tolerant storage of big data files. However, recent emerging architectural patterns increasingly rely on cloud object storage such as S3, Azure Blob Store, GCS, which are designed for cost-efficiency, scalability and geographic distribution. Hadoop supports pluggable file system implementations to enable integration with these systems for use cases such as off-site backup or even complex multi-step ETL, but applications may encounter unique challenges related to eventual consistency, performance and differences in semantics compared to HDFS. This session explores those challenges and presents recent work to address them in a comprehensive effort spanning multiple Hadoop ecosystem components, including the Object Store FileSystem connector, Hive, Tez and ORC. Our goal is to improve correctness, performance, security and operations for users that choose to integrate Hadoop with Cloud Storage. We use S3 and S3A connector as case study.
Time-oriented event search. A new level of scale 
Time-oriented event search. A new level of scale DataWorks Summit/Hadoop Summit The document discusses Rocana Search, a system built by Rocana to enable large scale real-time collection, processing, and analysis of event data. It aims to provide higher indexing throughput and better horizontal scaling than general purpose search systems like Solr. Key features include fully parallelized ingest and query, dynamic partitioning of data, and assigning partitions to nodes to maximize parallelism and locality. Initial benchmarks show Rocana Search can index over 3 times as many events per second as Solr.
Apache Hadoop YARN: Present and Future
Apache Hadoop YARN: Present and FutureDataWorks Summit Apache Hadoop YARN is the modern Distributed Operating System. It enables the Hadoop compute layer to be a common resource-management platform that can host a wide variety of applications. Multiple organizations are able to leverage YARN in building their applications on top of Hadoop without themselves repeatedly worrying about resource management, isolation, multi-tenancy issues etc.
In this talk, we’ll first hit the ground with the current status of Apache Hadoop YARN – how it is faring today in deployments large and small. We will cover different types of YARN deployments, in different environments and scale.
We'll then move on to the exciting present & future of YARN – features that are further strengthening YARN as the first-class resource-management platform for datacenters running enterprise Hadoop. We’ll discuss the current status as well as the future promise of features and initiatives like – 10x scheduler throughput improvements, docker containers support on YARN, support for long running services (alongside applications) natively without any changes, seamless application upgrades, fine-grained isolation for multi-tenancy using CGroups on disk & network resources, powerful scheduling features like application priorities, intra-queue preemption across applications and operational enhancements including insights through Timeline Service V2, a new web UI and better queue management.
How do you decide where your customer was?
How do you decide where your customer was?DataWorks Summit/Hadoop Summit This document summarizes Burak IŞIKLI and Erkan HASPULAT's presentation on Turkcell's use of Hadoop for customer analytics. Turkcell processes 15TB of data and 6TB of logs daily using Hadoop on a 350-node cluster. They use Hadoop for ETL, analytics like fraud detection and recommendations, and a data lake. Location analysis joins call data to determine subscribers' locations, processing 1.5TB daily. As data volumes grew they upgraded Hadoop and expanded storage from 15TB to 698TB. They also developed indexes for movement analysis and industry comparisons using Hive and Spark.
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in HiveDataWorks Summit/Hadoop Summit LLAP enables sub-second analytical queries in Hive by running query fragments directly in memory on compute nodes using a long-running daemon process. It provides high performance scans and execution through an in-memory columnar cache shared across queries. LLAP queries are coordinated independently by Tez while utilizing Hive operators for processing and Tez for data transfers. It improves upon traditional MapReduce and Tez by keeping intermediate query results in memory rather than writing to disk.
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit This document provides an overview of stream processing with Apache Flink. It discusses the rise of stream processing and how it enables low-latency applications and real-time analysis. It then describes Flink's stream processing capabilities, including pipelining of data, fault tolerance through checkpointing and recovery, and integration with batch processing. The document also summarizes Flink's programming model, state management, and roadmap for further development.
Introduction to Apache NiFi And Storm
Introduction to Apache NiFi And StormJungtaek Lim This slide shares an introduction to Apache NiFi and Storm. For Apache Storm the slide also talks about future of Apache Storm quickly.
Hive2.0 sql speed-scale--hadoop-summit-dublin-apr-2016
Hive2.0 sql speed-scale--hadoop-summit-dublin-apr-2016alanfgates This document discusses new features in Apache Hive 2.0, including:
1) Adding procedural SQL capabilities through HPLSQL for writing stored procedures.
2) Improving query performance through LLAP which uses persistent daemons and in-memory caching to enable sub-second queries.
3) Speeding up query planning by using HBase as the metastore instead of a relational database.
4) Enhancements to Hive on Spark such as dynamic partition pruning and vectorized operations.
5) Default use of the cost-based optimizer and continued improvements to statistics collection and estimation.
LLAP: Building Cloud First BI
LLAP: Building Cloud First BIDataWorks Summit LLAP (Live Long and Process) is the newest query acceleration engine for Hive 2.0, which entered GA in 2017. LLAP brings into light a new set of trade-offs and optimizations that allows for efficient and secure multi-user BI systems on the cloud. In this talk, we discuss the specifics of building a modern BI engine within those boundaries, designed to be fast and cost-effective on the public cloud. The focus of the LLAP cache is to speed up common BI query patterns on the cloud, while avoiding most of the operational administration overheads of maintaining a caching layer, with an automatically coherent cache with intelligent eviction and support for custom file formats from text to ORC, and explore the possibilities of combining the cache with a transactional storage layer which supports online UPDATE and DELETES without full data reloads. LLAP by itself, as a relational data layer, extends the same caching and security advantages to any other data processing framework. We overview the structure of such a hybrid system, where both Hive and Spark use LLAP to provide SQL query acceleration on the cloud with new, improved concurrent query support and production-ready tools and UI.
Speaker
Sergey Shelukin, Member of Technical Staff, Hortonworks
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!Mich Talebzadeh (Ph.D.) This document discusses using Spark as an execution engine for Hive queries. It begins by explaining that Hive and Spark are both commonly used in the big data space, and that Hive on Spark uses the Hive optimizer with the Spark query engine, while Spark with a Hive context uses both the Catalyst optimizer and Spark engine. The document then covers challenges in deploying Hive on Spark, such as using a custom Spark JAR without Hive dependencies. It shows how the Hive EXPLAIN command works the same on Spark, and how the execution plan and stages differ between MapReduce and Spark. Overall, the document provides a high-level overview of using Spark as a query engine for Hive.
Advanced Visualization of Spark jobs
Advanced Visualization of Spark jobsDataWorks Summit/Hadoop Summit This document discusses an advanced visualization tool for Spark and Flink jobs. It collects fine-grained data about task execution, including data characteristics and block fetch information. This information is exposed through a REST API and used to visualize the physical execution plan, detect issues like data skew, and help developers optimize their applications. The tool aims to help understand distributed data processing systems and guide testing of adaptive partitioning techniques. It has been extended to support Flink visualization as well. Future plans include open-sourcing the framework and adding more visualization features and metrics.
Ingest and Stream Processing - What will you choose?
Ingest and Stream Processing - What will you choose?DataWorks Summit/Hadoop Summit This document discusses streaming data ingestion and processing options. It provides an overview of common streaming architectures including Kafka as an ingestion hub and various streaming engines. Spark Streaming is highlighted as a popular and full-featured option for processing streaming data due to its support for SQL, machine learning, and ease of transition from batch workflows. The document also briefly profiles StreamSets Data Collector as a higher-level tool for building streaming data pipelines.
Interactive Analytics at Scale in Apache Hive Using Druid
Interactive Analytics at Scale in Apache Hive Using DruidDataWorks Summit/Hadoop Summit Druid is an open-source analytics data store specially designed to execute OLAP queries on event data. Its speed, scalability and efficiency have made it a popular choice to power user-facing analytic applications, including multiple BI tools and dashboards. However, Druid does not provide important features requested by many of these applications, such as a SQL interface or support for complex operations such as joins. This talk presents our work on extending Druid indexing and querying capabilities using Apache Hive. In particular, our solution allows to index complex query results in Druid using Hive, query Druid data sources from Hive using SQL, and execute complex Hive queries on top of Druid data sources. We describe how we built an extension that brings benefits to both systems alike, leveraging Apache Calcite to overcome the challenge of transparently generating Druid JSON queries from the input Hive SQL queries. We conclude with a demo highlighting the performant and powerful integration of these projects.
From Device to Data Center to Insights
From Device to Data Center to InsightsDataWorks Summit/Hadoop Summit This document discusses key architectural considerations for Internet of Things (IoT) systems. It outlines three main tiers: origin, transport, and analytics. The origin tier includes sensors, devices, and gateways that generate IoT data. Common protocols at this tier are discussed. The transport tier orchestrates data flow and can perform transformations. Apache NiFi and minifi are presented as options. The analytics tier is where insights are derived from the data through streaming and batch processing. Apache Beam is highlighted as a framework that can unify both types of processing. The document also discusses firmware versions, parsers, schemas, and data ownership challenges.
Managing data workflows with Luigi
Managing data workflows with LuigiTeemu Kurppa A talk about data workflow tools in Metrics Monday Helsinki.
Both Custobar (https://custobar.com) and ŌURA (https://ouraring.com) are hiring talented developers. Contact me if you are interested in joining either of companies.
Reproducible Computational Pipelines with Docker and Nextflow
Reproducible Computational Pipelines with Docker and Nextflowinside-BigData.com This document summarizes a presentation about using Docker and Nextflow to create reproducible computational pipelines. It discusses two major challenges in computational biology being reproducibility and complexity. Containers like Docker help address these challenges by creating portable and standardized environments. Nextflow is introduced as a workflow framework that allows pipelines to run across platforms and isolates dependencies using containers, enabling fast prototyping. Examples are given of using Nextflow with Docker to run pipelines on different systems like HPC clusters in a scalable and reproducible way.
Ad
More Related Content
What's hot (20)
Scale-Out Resource Management at Microsoft using Apache YARN
Scale-Out Resource Management at Microsoft using Apache YARNDataWorks Summit/Hadoop Summit This document discusses Microsoft's use of Apache YARN for scale-out resource management. It describes how YARN is used to manage vast amounts of data and compute resources across many different applications and workloads. The document outlines some limitations of YARN and Microsoft's contributions to address those limitations, including Rayon for improved scheduling, Mercury and Yaq for distributed scheduling, and work on federation to scale YARN across multiple clusters. It provides details on the implementation and evaluation of these contributions through papers, JIRAs, and integration into Apache Hadoop releases.
Hive on spark is blazing fast or is it final
Hive on spark is blazing fast or is it finalHortonworks This presentation was given at the Strata + Hadoop World, 2015 in San Jose.
Apache Hive is the most popular and most widely used SQL solution for Hadoop. To keep pace with Hadoop’s increasingly vital role in the Enterprise, Hive has transformed from a batch-only, high-latency system into a modern SQL engine capable of both batch and interactive queries over large datasets. Hive’s momentum is accelerating: With Spark integration and a shift to in-memory processing on the horizon, Hive continues to expand the boundaries of Big Data.
In this talk the speakers examined Hive performance, past, present and future. In particular they looked at Hive’s origins as a petabyte scale SQL engine.
Through some numbers and graphs, they showed how Hive became 100x faster by moving beyond MapReduce, by vectorizing execution and by introducing a cost-based optimizer.
They detailed and discussed the challenges of scalable SQL on Hadoop.
The looked into Hive’s sub-second future, powered by LLAP and Hive on Spark.
And showed just how fast Hive on Spark really is.
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in HiveDataWorks Summit/Hadoop Summit We discuss the current state of LLAP (Live Long and Process) – the concurrent sub-second execution of analytical queries engine for Hive 2.0. LLAP is a hybrid execution model that enables performance improvement in and across queries, such as caching of columnar data with cache coherence and intelligent eviction for disaggregated storage models (like S3, Isilon, Azure), JIT-friendly operator pipelines, asynchronous I/O, data pre-fetching and multi-threaded processing. LLAP features robust machine and service failure tolerance achieved by building on top of the time-tested fault tolerant subsystems, as well as a concurrency-directed design that achieves high utilization with low latency via resource sharing, reducing overheads for multiple queries, and enabling the system to preempt tasks of lower priority without failing any query in-flight. The talk also aims to cover the novel deployment model required for hybrid execution. The elasticity demands of the system are served by a long-lived YARN service interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework for query execution. We discuss the current state of the project, performance numbers, deployment and usage strategy, as well as future work, including how LLAP fits into a unified secure DataFrame access layer.
Network for the Large-scale Hadoop cluster at Yahoo! JAPAN
Network for the Large-scale Hadoop cluster at Yahoo! JAPANDataWorks Summit/Hadoop Summit Yahoo Japan transitioned their Hadoop cluster network architecture over time to address problems and scale needs. They moved from a stack architecture to an L2 fabric to an IP CLOS architecture. The IP CLOS architecture improved scalability, high availability, and reduced operating costs by allowing over 10,000 nodes with 100-200Gbps uplinks per rack and an oversubscription ratio of 1.25:1. This solved problems around switch failures, BUM traffic loads, decommissioning limitations, and scale-out limits they previously faced.
Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in ProductionDataWorks Summit/Hadoop Summit This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
The Future of Apache Ambari
The Future of Apache AmbariDataWorks Summit Apache Ambari is an extensible framework that simplifies provisioning, managing and monitoring Hadoop clusters. Apache Ambari was built on a standardized stack-based operations model. Stacks wrap services of all shapes and sizes with a consistent definition and lifecycle-control layer; thereby providing a consistent approach for managing and monitoring the services. This also provided a natural extension point for operators and the community to bring in their own add-on services and “plug-in” the new services into the stack.
However, one of the fundamental limitations of the current Apache Ambari architecture has been that there is a strong one-on-one coupling between entities. For instance, a cluster is tied to a single stack and a Hadoop operator can only deploy services defined in that stack, a cluster can have only a single instance of a service and a host can have only a single instance of a component. Taking into consideration various use case scenarios that cannot be enabled due to these limitations there is a growing need to revamp the Ambari architecture.
In this talk, we propose a revamped Apache Ambari architecture that will open up the floodgates for a wide range of scenarios that wouldn’t have been possible thus far. We will focus the discussion on a new mpack-based operations model that will replace the stack-based operations model. A management package is a self-contained deployment artifact that includes all the details for deploying, managing and upgrading a set of services bundled in the package. A third-party provider can also build their own management package containing their custom services. This eliminates the need to plug-in their services into a stack and also can define their own upgrade story for these custom services. A Hadoop operator will be able to deploy a Hadoop cluster with a mix of services across multiple packages instead of being limited to a single stack. For example, it would be possible to deploy a cluster with HDFS from HDP and NIFI from HDF.
Further, we will also discuss about the architectural changes needed to enable a multi instance architecture in future Ambari releases to support deploying multiple instances of a service in a cluster, deploying multiple instances of a component on a host as well as future proofing the Ambari architecture to leverage some of the advancements happening in the Hadoop community like YARN services (YARN-4692). We will wrap up the conversation with a brief overview of other improvements planned for future releases of Ambari.
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)Chris Nauroth Today's typical Apache Hadoop deployments use HDFS for persistent, fault-tolerant storage of big data files. However, recent emerging architectural patterns increasingly rely on cloud object storage such as S3, Azure Blob Store, GCS, which are designed for cost-efficiency, scalability and geographic distribution. Hadoop supports pluggable file system implementations to enable integration with these systems for use cases such as off-site backup or even complex multi-step ETL, but applications may encounter unique challenges related to eventual consistency, performance and differences in semantics compared to HDFS. This session explores those challenges and presents recent work to address them in a comprehensive effort spanning multiple Hadoop ecosystem components, including the Object Store FileSystem connector, Hive, Tez and ORC. Our goal is to improve correctness, performance, security and operations for users that choose to integrate Hadoop with Cloud Storage. We use S3 and S3A connector as case study.
Time-oriented event search. A new level of scale
Time-oriented event search. A new level of scale DataWorks Summit/Hadoop Summit The document discusses Rocana Search, a system built by Rocana to enable large scale real-time collection, processing, and analysis of event data. It aims to provide higher indexing throughput and better horizontal scaling than general purpose search systems like Solr. Key features include fully parallelized ingest and query, dynamic partitioning of data, and assigning partitions to nodes to maximize parallelism and locality. Initial benchmarks show Rocana Search can index over 3 times as many events per second as Solr.
Apache Hadoop YARN: Present and Future
Apache Hadoop YARN: Present and FutureDataWorks Summit Apache Hadoop YARN is the modern Distributed Operating System. It enables the Hadoop compute layer to be a common resource-management platform that can host a wide variety of applications. Multiple organizations are able to leverage YARN in building their applications on top of Hadoop without themselves repeatedly worrying about resource management, isolation, multi-tenancy issues etc.
In this talk, we’ll first hit the ground with the current status of Apache Hadoop YARN – how it is faring today in deployments large and small. We will cover different types of YARN deployments, in different environments and scale.
We'll then move on to the exciting present & future of YARN – features that are further strengthening YARN as the first-class resource-management platform for datacenters running enterprise Hadoop. We’ll discuss the current status as well as the future promise of features and initiatives like – 10x scheduler throughput improvements, docker containers support on YARN, support for long running services (alongside applications) natively without any changes, seamless application upgrades, fine-grained isolation for multi-tenancy using CGroups on disk & network resources, powerful scheduling features like application priorities, intra-queue preemption across applications and operational enhancements including insights through Timeline Service V2, a new web UI and better queue management.
How do you decide where your customer was?
How do you decide where your customer was?DataWorks Summit/Hadoop Summit This document summarizes Burak IŞIKLI and Erkan HASPULAT's presentation on Turkcell's use of Hadoop for customer analytics. Turkcell processes 15TB of data and 6TB of logs daily using Hadoop on a 350-node cluster. They use Hadoop for ETL, analytics like fraud detection and recommendations, and a data lake. Location analysis joins call data to determine subscribers' locations, processing 1.5TB daily. As data volumes grew they upgraded Hadoop and expanded storage from 15TB to 698TB. They also developed indexes for movement analysis and industry comparisons using Hive and Spark.
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in HiveDataWorks Summit/Hadoop Summit LLAP enables sub-second analytical queries in Hive by running query fragments directly in memory on compute nodes using a long-running daemon process. It provides high performance scans and execution through an in-memory columnar cache shared across queries. LLAP queries are coordinated independently by Tez while utilizing Hive operators for processing and Tez for data transfers. It improves upon traditional MapReduce and Tez by keeping intermediate query results in memory rather than writing to disk.
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit This document provides an overview of stream processing with Apache Flink. It discusses the rise of stream processing and how it enables low-latency applications and real-time analysis. It then describes Flink's stream processing capabilities, including pipelining of data, fault tolerance through checkpointing and recovery, and integration with batch processing. The document also summarizes Flink's programming model, state management, and roadmap for further development.
Introduction to Apache NiFi And Storm
Introduction to Apache NiFi And StormJungtaek Lim This slide shares an introduction to Apache NiFi and Storm. For Apache Storm the slide also talks about future of Apache Storm quickly.
Hive2.0 sql speed-scale--hadoop-summit-dublin-apr-2016
Hive2.0 sql speed-scale--hadoop-summit-dublin-apr-2016alanfgates This document discusses new features in Apache Hive 2.0, including:
1) Adding procedural SQL capabilities through HPLSQL for writing stored procedures.
2) Improving query performance through LLAP which uses persistent daemons and in-memory caching to enable sub-second queries.
3) Speeding up query planning by using HBase as the metastore instead of a relational database.
4) Enhancements to Hive on Spark such as dynamic partition pruning and vectorized operations.
5) Default use of the cost-based optimizer and continued improvements to statistics collection and estimation.
LLAP: Building Cloud First BI
LLAP: Building Cloud First BIDataWorks Summit LLAP (Live Long and Process) is the newest query acceleration engine for Hive 2.0, which entered GA in 2017. LLAP brings into light a new set of trade-offs and optimizations that allows for efficient and secure multi-user BI systems on the cloud. In this talk, we discuss the specifics of building a modern BI engine within those boundaries, designed to be fast and cost-effective on the public cloud. The focus of the LLAP cache is to speed up common BI query patterns on the cloud, while avoiding most of the operational administration overheads of maintaining a caching layer, with an automatically coherent cache with intelligent eviction and support for custom file formats from text to ORC, and explore the possibilities of combining the cache with a transactional storage layer which supports online UPDATE and DELETES without full data reloads. LLAP by itself, as a relational data layer, extends the same caching and security advantages to any other data processing framework. We overview the structure of such a hybrid system, where both Hive and Spark use LLAP to provide SQL query acceleration on the cloud with new, improved concurrent query support and production-ready tools and UI.
Speaker
Sergey Shelukin, Member of Technical Staff, Hortonworks
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!Mich Talebzadeh (Ph.D.) This document discusses using Spark as an execution engine for Hive queries. It begins by explaining that Hive and Spark are both commonly used in the big data space, and that Hive on Spark uses the Hive optimizer with the Spark query engine, while Spark with a Hive context uses both the Catalyst optimizer and Spark engine. The document then covers challenges in deploying Hive on Spark, such as using a custom Spark JAR without Hive dependencies. It shows how the Hive EXPLAIN command works the same on Spark, and how the execution plan and stages differ between MapReduce and Spark. Overall, the document provides a high-level overview of using Spark as a query engine for Hive.
Advanced Visualization of Spark jobs
Advanced Visualization of Spark jobsDataWorks Summit/Hadoop Summit This document discusses an advanced visualization tool for Spark and Flink jobs. It collects fine-grained data about task execution, including data characteristics and block fetch information. This information is exposed through a REST API and used to visualize the physical execution plan, detect issues like data skew, and help developers optimize their applications. The tool aims to help understand distributed data processing systems and guide testing of adaptive partitioning techniques. It has been extended to support Flink visualization as well. Future plans include open-sourcing the framework and adding more visualization features and metrics.
Ingest and Stream Processing - What will you choose?
Ingest and Stream Processing - What will you choose?DataWorks Summit/Hadoop Summit This document discusses streaming data ingestion and processing options. It provides an overview of common streaming architectures including Kafka as an ingestion hub and various streaming engines. Spark Streaming is highlighted as a popular and full-featured option for processing streaming data due to its support for SQL, machine learning, and ease of transition from batch workflows. The document also briefly profiles StreamSets Data Collector as a higher-level tool for building streaming data pipelines.
Interactive Analytics at Scale in Apache Hive Using Druid
Interactive Analytics at Scale in Apache Hive Using DruidDataWorks Summit/Hadoop Summit Druid is an open-source analytics data store specially designed to execute OLAP queries on event data. Its speed, scalability and efficiency have made it a popular choice to power user-facing analytic applications, including multiple BI tools and dashboards. However, Druid does not provide important features requested by many of these applications, such as a SQL interface or support for complex operations such as joins. This talk presents our work on extending Druid indexing and querying capabilities using Apache Hive. In particular, our solution allows to index complex query results in Druid using Hive, query Druid data sources from Hive using SQL, and execute complex Hive queries on top of Druid data sources. We describe how we built an extension that brings benefits to both systems alike, leveraging Apache Calcite to overcome the challenge of transparently generating Druid JSON queries from the input Hive SQL queries. We conclude with a demo highlighting the performant and powerful integration of these projects.
From Device to Data Center to Insights
From Device to Data Center to InsightsDataWorks Summit/Hadoop Summit This document discusses key architectural considerations for Internet of Things (IoT) systems. It outlines three main tiers: origin, transport, and analytics. The origin tier includes sensors, devices, and gateways that generate IoT data. Common protocols at this tier are discussed. The transport tier orchestrates data flow and can perform transformations. Apache NiFi and minifi are presented as options. The analytics tier is where insights are derived from the data through streaming and batch processing. Apache Beam is highlighted as a framework that can unify both types of processing. The document also discusses firmware versions, parsers, schemas, and data ownership challenges.
Viewers also liked (11)
Managing data workflows with Luigi
Managing data workflows with LuigiTeemu Kurppa A talk about data workflow tools in Metrics Monday Helsinki.
Both Custobar (https://custobar.com) and ŌURA (https://ouraring.com) are hiring talented developers. Contact me if you are interested in joining either of companies.
Reproducible Computational Pipelines with Docker and Nextflow
Reproducible Computational Pipelines with Docker and Nextflowinside-BigData.com This document summarizes a presentation about using Docker and Nextflow to create reproducible computational pipelines. It discusses two major challenges in computational biology being reproducibility and complexity. Containers like Docker help address these challenges by creating portable and standardized environments. Nextflow is introduced as a workflow framework that allows pipelines to run across platforms and isolates dependencies using containers, enabling fast prototyping. Examples are given of using Nextflow with Docker to run pipelines on different systems like HPC clusters in a scalable and reproducible way.
Building cloud-enabled genomics workflows with Luigi and Docker
Building cloud-enabled genomics workflows with Luigi and DockerJacob Feala Talk given at Bio-IT 2016, Cloud Computing track
Abstract:
As bioinformatics scientists, we tend to write custom tools for managing our workflows, even when viable, open-source alternatives are available from the tech community. Our field has, however, begun to adopt Docker containers to stabilize compute environments. In this talk, I will introduce Luigi, a workflow system built by engineers at Spotify to manage long-running big data processing jobs with complex dependencies. Focusing on a case study of next generation sequencing analysis in cancer genomics research, I will show how Luigi can connect simple, containerized applications into complex bioinformatics pipelines that can be easily integrated with compute, storage, and data warehousing on the cloud.
Luigi presentation NYC Data Science
Luigi presentation NYC Data ScienceErik Bernhardsson Luigi is a workflow management system that allows users to build complex data pipelines. It provides tools to define dependencies between tasks, run workflows on Hadoop, and visualize data flows. The speaker describes how they developed Luigi at Spotify to manage thousands of Hadoop jobs run daily for music recommendations and other applications. Key features of Luigi include defining Python tasks, easy command line execution, automatic dependency resolution, and failure recovery through atomic file operations. The speaker demonstrates how Luigi can run multi-step workflows on the command line, including a music recommendation example involving feature extraction, model training, and evaluation.
A Beginner's Guide to Building Data Pipelines with Luigi
A Beginner's Guide to Building Data Pipelines with LuigiGrowth Intelligence In this talk, we provide an introduction to Python Luigi via real life case studies showing you how you can break large, multi-step data processing task into a graph of smaller sub-tasks that are aware of the state of their interdependencies.
Growth Intelligence tracks the performance and activity of all the companies in the UK economy using their data ‘footprint’. This involves tracking numerous unstructured data points from multiple sources in a variety of formats and transforming them into a standardised feature set we can use for building predictive models for our clients.
In the past, this data was collected by in a somewhat haphazard fashion: combining manual effort, ad hoc scripting and processing which was difficult to maintain. In order to streamline the data flows, we’re using an open-source Python framework from Spotify called Luigi. Luigi was created for managing task dependencies, monitoring the progress of the data pipeline and providing frameworks for common batch processing tasks.
認識那條鯨魚 Docker 初探
認識那條鯨魚 Docker 初探仲昀 王 介紹 PyCV-time / TechTree
docker 操作基本功
在 docker container 中使用 GUI 介面 / webcam / GPU 前要做的處理
Docker 初探，實驗室中的運貨鯨
Docker 初探，實驗室中的運貨鯨Ruoshi Ling This document provides instructions for getting started with Docker and Docker Compose. It explains how to install Docker and Docker Compose, basic Docker commands like running containers and viewing logs, mapping ports, and using Docker Compose to define and run multi-container applications.
Structured Container Delivery by Oscar Renalias, Accenture
Structured Container Delivery by Oscar Renalias, AccentureDocker, Inc. With tools like Docker Toolbox, the entry barrier to Docker and containers is rather low. However, it takes a lot more to design, build and run an entire container platform, at scale, for production applications.
This talk will focus on why it is important to have a well-defined reference model for building container platforms that guides container engineers and architects through the process of identifying platform concerns, patterns, components as well as the interactions between them in order to deliver a set of platform capabilities (service discovery, load balancing, security, and others) to support containerized applications using existing tooling.
As part of this session will also see how a container architecture has enabled real projects in their delivery of container platforms.
Why Docker
Why DockerdotCloud Configuration management tools like Chef, Puppet, and Ansible aim to reduce inconsistencies by imposing and managing consistent configurations across environments. However, they do not fully address issues related to dependencies, isolation, and portability. Docker containers build on these tools by adding standard interfaces and a lightweight virtualization layer that encapsulates code and dependencies, allowing applications and their environments to be packaged together and run consistently on any infrastructure while also providing isolation.
Docker introduction
Docker introductiondotCloud Docker is a system for running applications in isolated containers. It addresses issues with traditional virtual machines by providing lightweight containers that share resources and allow applications to run consistently across different environments. Docker eliminates inconsistencies in development, testing and production environments. It allows applications and their dependencies to be packaged into a standardized unit called a container that can run on any Linux server. This makes applications highly portable and improves efficiency across the entire development lifecycle.
Ad
Similar to Migrating pipelines into Docker (20)
DevOps and OpenStack December 2012
DevOps and OpenStack December 2012Matt Ray This document discusses OpenStack, an open source cloud computing platform. It provides OpenStack's mission statement to create a simple and scalable cloud platform. OpenStack controls compute, storage, and networking through projects like Nova (compute), Cinder (block storage), Quantum (networking), Glance (image registry), Keystone (identity), Swift (object storage), and Horizon (dashboard). The document outlines OpenStack's architecture, community involvement, code repositories, distributions, and the upcoming Grizzly release.
State of Big Data on ARM64 / AArch64 - Apache Bigtop
State of Big Data on ARM64 / AArch64 - Apache BigtopGanesh Raju This document discusses Apache Bigtop, an open source project that provides tools for building, deploying, and testing big data software stacks across multiple platforms. It summarizes Bigtop's components and goals, contributions from ARM to support AArch64, challenges faced in the porting process, and the roadmap for further improving Bigtop including adding Kubernetes support and predefined sample stacks. It also demonstrates Bigtop's sandbox feature for building and running pseudo clusters using Docker.
R meetup 20161011v2
R meetup 20161011v2Niels Ole Dam Presentation from the CopenhagenR - useR Group Meetup at
IT University of Copenhagen on Oct. 11 2016 on how to automatically deploy web applications built in R to a Cloud server (here DigitalOcean) using open source Docker with GitHub and basic Continuous Integration (here CircleCI) for automated testing and deployment.
Presenter:
Niels Ole Dam, Things in Flow
Excerpt from the invitation to the meetup:
Niels will talk about his favorite R-setup and will demonstrate how R, combined with some nice DockeR and Github tricks, can help even small teams and companies leverage the power of modern cloud computing. Niels uses R on a daily basis in his work as an independent consultant and he will share his thoughts on DockeR at the next meetup.
Subjects covered:
- How to setup and use RStudio, Docker, Docker Compose locally and with GitHub intgration.
- How to setup and use Continuous Integration (CI) with automated testing and deployment to DigitalOcean using CircelCI and with reuse of the same docker-compose.yml file locally and remotely.
- Tips and tricks on how to setup a good workflow.
- Introduction to all the technologies and tools used.
There are lots of clickable links in the pdf-version of the slides.
Code for the setup demonstrated can be found at:
https://github.com/thingsinflow/r-docker-workflow
An accompanying clickable flowdiagram can be found at:
http://bit.ly/R-Docker-workflow
Enjoy!
:-)
Hambug R Meetup - Intro to H2O
Hambug R Meetup - Intro to H2OSri Ambati This document summarizes a presentation about H2O's machine learning platform and Deep Water distributed deep learning capabilities. The presentation introduces H2O, its open source in-memory machine learning platform, performance advantages, and interfaces for R, Python and Flow. Deep Water is introduced as H2O's integration with TensorFlow, MXNet and Caffe that provides a unified interface for distributed deep learning on GPUs. Examples are shown training convolutional neural networks on image datasets using Deep Water with different backends.
'DOCKER' & CLOUD: ENABLERS For DEVOPS
'DOCKER' & CLOUD: ENABLERS For DEVOPSACA IT-Solutions Sprekers: Stijn Van den Enden & Stijn Wijndaele (ACA IT-Solutions) DevOps is gericht op het tot stand brengen van een cultuur binnen organisaties waardoor het ontwikkelen, valideren en releasen van software sneller, meer betrouwbaar en frequenter kan verlopen. Om dit te realiseren staan het automatiseren van het 'software delivery process' en de bijhorende infrastructurele veranderingen centraal. Door de opkomst van 'Microservice Architecture' neemt het belang hiervan nog verder toe.
In deze avondconferentie werd, na een korte toelichting over DevOps, nagegaan wat Docker en de Cloud kunnen betekenen voor uw business, en hoe zij als enablers kunnen dienen voor het tot stand brengen van een DevOps-cultuur. Het container-landschap waarvan tools zoals Kubernetes, Docker Swarm, ...een belangrijk onderdeel vormen, wordt toegelicht en er wordt ingegaan op de wijze waarop deze tools aangewend kunnen worden om 'development' en 'operations' efficiënt te laten samenwerken.
Docker and Cloud - Enables for DevOps - by ACA-IT
Docker and Cloud - Enables for DevOps - by ACA-ITStijn Wijndaele DevOps is gericht op het tot stand brengen van een cultuur binnen organisaties waardoor het ontwikkelen, valideren en releasen van software sneller, meer betrouwbaar en frequenter kan verlopen. Om dit te realiseren staan het automatiseren van het 'software delivery process' en de bijhorende infrastructurele veranderingen centraal. Door de opkomst van 'Microservice Architecture' neemt het belang hiervan nog verder toe.
Openstack and Reddwarf Overview
Openstack and Reddwarf OverviewCraig Vyvial This document provides an overview of OpenStack and Reddwarf architectures. OpenStack is an open source cloud computing platform consisting of services like identity (Keystone), compute (Nova), image (Glance), networking (Quantum), and block storage (Cinder). Reddwarf builds on OpenStack to provide a platform as a service for database instances using a REST API, task manager, and guest agent to manage MySQL instances. The presentation also covers developer vs deployer roles and links to project resources.
Manchester geek night pcf 101
Manchester geek night pcf 101Sufyaan Kazi Pivotal Cloud Foundry 101 given at Manchester Geek night
Original Content: https://www.youtube.com/watch?v=dHSbR9cm-_0&feature=youtu.be
data.table and H2O at LondonR with Matt Dowle
data.table and H2O at LondonR with Matt DowleSri Ambati The document summarizes a presentation given by Matt Dowle on news from the data.table package version 1.9 and H2O.ai. It begins by discussing past presentations given by Dowle on data.table and addresses a comment about dplyr killing off data.table. It then demonstrates the speed and functionality of data.table, such as automatic indexing, and compares it to dplyr and base R. The document concludes by introducing H2O.ai for machine learning on large datasets and provides instructions for installing and using H2O in R.
Cloud Native Landscape (CNCF and OCI)
Cloud Native Landscape (CNCF and OCI)Chris Aniszczyk Presentation given at Open Source Summit Japan 2016 about the state of the cloud native technology (Cloud Native Computing Foundation) and the standardization of container technology (Open Container Initiative)
GCP Meetup #3 - Approaches to Cloud Native Architectures
GCP Meetup #3 - Approaches to Cloud Native Architecturesnine Talk by Daniel Leahy and Nic Gibson, given at the Google Cloud Meetup on March 3, 2020, hosted by Nine Internet Solutions AG - Your Swiss Managed Cloud Service Provider.
Xen in the Cloud
Xen in the CloudThe Linux Foundation Globo.com is the internet branch of Globo Organizations, the largest media conglomerate in Latin America. Globo.com has built a cloud platform called Project Orquestra using the Xen hypervisor to virtualize servers and reduce costs associated with power consumption and data center space. Project Orquestra uses queuing, authorization, and network services to manage virtual machine and resource requests across datacenters. Globo.com plans to contribute Project Orquestra to OpenStack and make it open source to help the OpenStack project.
Xen Summit 2011 - Xen in the Cloud - globo.com
Xen Summit 2011 - Xen in the Cloud - globo.comMarco Sinhoreli Globo.com is the internet branch of Globo Organizations, the largest media conglomerate in Latin America. Globo.com has built a cloud platform called Project Orquestra using the Xen hypervisor to virtualize servers and reduce costs associated with power consumption and data center space. Project Orquestra uses queuing, network management, and identity services to provide a self-service interface for users to provision virtual machines and manage cloud resources according to their assigned permissions. Globo.com plans to contribute Project Orquestra to the open source community and integrate it with OpenStack.
Docker Training - June 2015
Docker Training - June 2015{code} Docker provides an open platform for building, shipping, and running distributed applications. The Docker Engine and Docker Hub provide the core open source software and cloud services. Docker allows developers to build applications once and run them anywhere by packaging applications into lightweight executable containers that can operate consistently regardless of infrastructure. This enables faster delivery, easier deployment and scaling, and running more workloads.
[db tech showcase Tokyo 2017] C13:There and back again or how to connect Orac...![[db tech showcase Tokyo 2017] C13:There and back again or how to connect Orac...](https://cdn.slidesharecdn.com/ss_thumbnails/thereandbackagainoracleandbigdata16x9-170911060821-thumbnail.jpg?width=560&fit=bounds)
[db tech showcase Tokyo 2017] C13:There and back again or how to connect Orac...Insight Technology, Inc. Gleb Otochkin presented on connecting Oracle databases to Big Data platforms. He discussed using Oracle GoldenGate for real-time replication from Oracle to HDFS, Kafka, Flume, HBase, MongoDB and Cassandra. For batch loads, tools include Sqoop and Oracle Loader for Hadoop. Oracle Data Integrator and Oracle SQL Connector for HDFS can query Big Data sources. Business cases for integration include growing data volumes, unstructured data, data retention policies and cost savings.
There and back_again_oracle_and_big_data_16x9
There and back_again_oracle_and_big_data_16x9Gleb Otochkin Gleb Otochkin presented ways to connect Oracle databases to Big Data platforms. Real-time replication from Oracle to HDFS, Kafka, Flume, and HBase can be done using Oracle GoldenGate. Batch loading is possible using Sqoop. Data from Big Data sources can be accessed from Oracle using tools like Oracle Data Integrator, Oracle SQL Connector for HDFS, and Oracle Loader for Hadoop.
Mad scalability: Scaling when you are not Google
Mad scalability: Scaling when you are not GoogleAbel Muíño A brief history of how Cabify grew during the last couple of years without using cloud platforms like AWS or Google Cloud
I gave this talk at the madScalability meet up on Madrid, March 2017.
Latest Developments in H2O
Latest Developments in H2OSri Ambati This document provides an overview of latest developments at H2O.ai, including enhancements to their machine learning platform H2O such as Deep Water for distributed deep learning using GPUs, xgboost integration, stacked ensembles, automatic machine learning, and model interpretation capabilities. It also discusses H2O's community involvement through meetup groups and conferences like PyData.
Lambda Architectures in Practice
Lambda Architectures in PracticeC4Media Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1FQYcP0.
Gian Merlino presents the advantages, challenges, and best practices to deploying and maintaining lambda architectures in the real world, using the infrastructure at Metamarkets as a case study. Filmed at qconsf.com.
Gian Merlino is a senior software engineer at Metamarkets, responsible for the infrastructure behind its data ingestion pipelines and is a committer on the Druid project.
Docker and containers - For Boston Docker Meetup Workshop in March 2015
Docker and containers - For Boston Docker Meetup Workshop in March 2015Jonas Rosland Docker provides an open platform consisting of Docker Engine, Docker Hub, and an API for building, shipping and running distributed applications across any infrastructure. The Docker Engine is open source software that provides lightweight containers to package applications securely and consistently across any infrastructure from physical to virtual cloud servers. Docker Hub is a cloud-based registry where public, private and curated container images can be stored, shared and deployed. This allows applications to be built once and run anywhere while simplifying deployment, management and scaling of applications across infrastructure.
[db tech showcase Tokyo 2017] C13:There and back again or how to connect Orac...
[db tech showcase Tokyo 2017] C13:There and back again or how to connect Orac...Insight Technology, Inc.
Ad
More from DataWorks Summit/Hadoop Summit (20)
State of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache ZeppelinDataWorks Summit/Hadoop Summit This document discusses Spark security and provides an overview of authentication, authorization, encryption, and auditing in Spark. It describes how Spark leverages Kerberos for authentication and uses services like Ranger and Sentry for authorization. It also outlines how communication channels in Spark are encrypted and some common issues to watch out for related to Spark security.
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache RangerDataWorks Summit/Hadoop Summit The document discusses the Virtual Data Connector project which aims to leverage Apache Atlas and Apache Ranger to provide unified metadata and access governance across data sources. Key points include:
- The project aims to address challenges of understanding, governing, and controlling access to distributed data through a centralized metadata catalog and policies.
- Apache Atlas provides a scalable metadata repository while Apache Ranger enables centralized access governance. The project will integrate these using a virtualization layer.
- Enhancements to Atlas and Ranger are proposed to better support the project's goals around a unified open metadata platform and metadata-driven governance.
- An initial minimum viable product will be built this year with the goal of an open, collaborative ecosystem around shared
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science PlatformDataWorks Summit/Hadoop Summit This document discusses using a data science platform to enable digital diagnostics in healthcare. It provides an overview of healthcare data sources and Yale/YNHH's data science platform. It then describes the data science journey process using a clinical laboratory use case as an example. The goal is to use big data and machine learning to improve diagnostic reproducibility, throughput, turnaround time, and accuracy for laboratory testing by developing a machine learning algorithm and real-time data processing pipeline.
Revolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and ZeppelinDataWorks Summit/Hadoop Summit This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSenseDataWorks Summit/Hadoop Summit This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.
Hadoop Crash Course
Hadoop Crash CourseDataWorks Summit/Hadoop Summit The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.
Data Science Crash Course
Data Science Crash CourseDataWorks Summit/Hadoop Summit This document provides an overview of data science and machine learning. It discusses what data science and machine learning are, including extracting insights from data and computers learning without being explicitly programmed. It also covers Apache Spark, which is an open source framework for large-scale data processing. Finally, it discusses common machine learning algorithms like regression, classification, clustering, and dimensionality reduction.
Apache Spark Crash Course
Apache Spark Crash CourseDataWorks Summit/Hadoop Summit This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataWorks Summit/Hadoop Summit This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
Schema Registry - Set you Data Free
Schema Registry - Set you Data FreeDataWorks Summit/Hadoop Summit Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats.
SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc.
In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time.
The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and MLDataWorks Summit/Hadoop Summit QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful.
At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.
How Hadoop Makes the Natixis Pack More Efficient 
How Hadoop Makes the Natixis Pack More Efficient DataWorks Summit/Hadoop Summit Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together.
This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear:
• How and why the business and IT requirements originated
• How we leverage the platform to fulfill security and production requirements
• How we organize a community to:
o Guard all the players, no one gets left on the ground!
o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead)
• What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community
We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match!
DETAILS
This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.
HBase in Practice 
HBase in Practice DataWorks Summit/Hadoop Summit HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases.
In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopDataWorks Summit/Hadoop Summit In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs.
Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.
Backup and Disaster Recovery in Hadoop 
Backup and Disaster Recovery in Hadoop DataWorks Summit/Hadoop Summit While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.
Scaling HDFS to Manage Billions of Files with Distributed Storage Schemes
Scaling HDFS to Manage Billions of Files with Distributed Storage SchemesDataWorks Summit/Hadoop Summit Hadoop Distributed File System (HDFS) evolves from a MapReduce-centric storage system to a generic, cost-effective storage infrastructure where HDFS stores all data of inside the organizations. The new use case presents a new sets of challenges to the original HDFS architecture. One challenge is to scale the storage management of HDFS - the centralized scheme within NameNode becomes a main bottleneck which limits the total number of files stored. Although a typical large HDFS cluster is able to store several hundred petabytes of data, it is inefficient to handle large amounts of small files under the current architecture.
In this talk, we introduce our new design and in-progress work that re-architects HDFS to attack this limitation. The storage management is enhanced to a distributed scheme. A new concept of storage container is introduced for storing objects. HDFS blocks are stored and managed as objects in the storage containers instead of being tracked only by NameNode. Storage containers are replicated across DataNodes using a newly-developed high-throughput protocol based on the Raft consensus algorithm. Our current prototype shows that under the new architecture the storage management of HDFS scales 10x better, demonstrating that HDFS is capable of storing billions of files.
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit
Scaling HDFS to Manage Billions of Files with Distributed Storage Schemes
Scaling HDFS to Manage Billions of Files with Distributed Storage SchemesDataWorks Summit/Hadoop Summit
Recently uploaded (20)
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityriccardosl1 Cyber awareness training educates employees on risk associated with internet and malicious emails
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBrian McKeiver May 2nd, 2025 talk at StirTrek 2025 Conference.
HCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/
HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client.
Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience.
In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including
- Accessing the console
- Locating and interpreting log files
- Accessing the data folder within the browser’s cache (using OPFS)
- Understand the difference between single- and multi-user scenarios
- Utilizing Client Clocking
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.
How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?Daniel Lehner 𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨?
Everyone’s talking about AI but is anyone really using it to create real value?
Most companies want to leverage AI. Few know 𝗵𝗼𝘄.
✅ What exactly should you ask to find real AI opportunities?
✅ Which AI techniques actually fit your business?
✅ Is your data even ready for AI?
If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.
Semantic Cultivators : The Critical Future Role to Enable AI
Semantic Cultivators : The Critical Future Role to Enable AIartmondano By 2026, AI agents will consume 10x more enterprise data than humans, but with none of the contextual understanding that prevents catastrophic misinterpretations.
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025
https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/
Is AI just another technology, or does it fundamentally change the way we live and think?
Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater.
At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts.
At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.
Technology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data AnalyticsInData Labs At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including:
-Artificial Intelligence Market Overview
-Strategies for AI Adoption in 2025
-Anticipated drivers of AI adoption and transformative technologies
-Benefits of AI and Big data for your business
-Tips on how to prepare your business for innovation
-AI and data privacy: Strategies for securing data privacy in AI models, etc.
Download your free copy nowand implement the key findings to improve your business.
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv IEDM 2024 Tutorial2
Procurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptxJon Hansen Procurement Insights integrated Historic Procurement Industry Archives, serves as a powerful complement — not a competitor — to other procurement industry firms. It fills critical gaps in depth, agility, and contextual insight that most traditional analyst and association models overlook.
Learn more about this value- driven proprietary service offering here.
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how:
• Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules.
• Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance.
• Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity.
• Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications.
• Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market.
With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications.
Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market
For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB Want to learn practical tips for designing systems that can scale efficiently without compromising speed?
Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development.
As you explore key principles of designing low-latency systems with Rust, you will learn how to:
- Create and compile a real-world app with Rust
- Connect the application to ScyllaDB (NoSQL data store)
- Negotiate tradeoffs related to data modeling and querying
- Manage and monitor the database for consistently low latencies
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok In today's fast-paced retail environment, efficiency is key. Every minute counts, and every penny matters. One tool that can significantly boost your store's efficiency is a well-executed planogram. These visual merchandising blueprints not only enhance store layouts but also save time and money in the process.
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john Analyze the growth of meme coins from mere online jokes to potential assets in the digital economy. Explore the community, culture, and utility as they elevate themselves to a new era in cryptocurrency.
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies In late April 2025, a significant portion of Europe, particularly Spain, Portugal, and parts of southern France, experienced widespread, rolling power outages that continue to affect millions of residents, businesses, and infrastructure systems.
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell With expertise in data architecture, performance tracking, and revenue forecasting, Andrew Marnell plays a vital role in aligning business strategies with data insights. Andrew Marnell’s ability to lead cross-functional teams ensures businesses achieve sustainable growth and operational excellence.
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat
Migrating pipelines into Docker
- 1. Migrating pipelines into Docker Noa Resare, Spotify @blippie
- 2. Welcome! ‣(let’s define pipeline) ‣Background ‣Docker improving engineering experience ‣Docker piece of puzzle to handle growth ‣Practical advice
- 3. Spotify & me ‣Spotify Streaming music Celebrates 10 years this summer 30m subscribers, most users on free tier millions of concurrent users ‣..me at Spotify for 6 years. less than 50 engineers, now more than 1000 operations engineering backend development Free Software Data Infrastructure
- 5. Humble beginnings ‣Counting stream playbacks ‣Stack of servers in the Fußball-room ‣Streaming Hadoop, python ‣Quick excursion to Amazon 2012
- 6. The new cluster ‣One large cluster, early 2012 ‣60 nodes! ‣Luigi development starts
- 7. More technologies ‣python code ‣pure java map/reduce ‣apache scrub ‣scala, scalding
- 8. Spotify engineering org ‣A lot of autonomy ‣Big data touches a many different teams Finance Analytics Feature development (A/B testing) Recommendations Payments and fraud
- 9. Shared resources, packaging ‣Started out with some shared edge nodes chaos ensued ‣More edge nodes! more chaos? more chaos! ‣Shared execution environment from .deb to .jar still a lot of one off edge nodes
- 11. Brief introduction to docker ‣Containers seem like virtual machines ‣docker run -it <image_name> ‣Filesystem reset between invocations ‣Typically built using a docker file ‣Image inheritance
- 12. Docker at Spotify ‣Big bet on docker for services: helios ‣Lots of useful infrastructure ‣Solves some immediate packaging problems
- 13. What does Docker provide? ‣Useful abstraction to reason about ‣an incremental way out of dependency hell ‣Artefact distribution, caching ‣Image inheritance mechanism for sharing infrastructure
- 14. Switching to docker in practice ‣Previously maven project with java, python, cron file build step to upload resulting jar to artifactory build step to copy cron file to execution cluster ‣Now add Dockerfile, data infrastructure base image build step to build and upload image
- 15. Problems with cron cluster execution ‣Implicit deployment via CI/CD declaration ‣Status reported via output materialising ‣Who / what triggered job X? ‣Where does it run? ‣Debugging is a pain
- 16. Our solution: execution as a service ‣Restful API for pipeline execution ‣List your job invocations ‣Explicitly schedule execution on node ‣Don’t rerun successful execution ‣Interface: docker image
- 17. Data growth, or cluster day of doom
- 18. Scaling is hard ‣2000 nodes ‣100 PB storage ‣800 000 000 files in HDFS ‣180GB heap, 10G young generation ‣Adding 100TB data per day
- 19. Docker as vehicle for migration ‣Our path forward: Google Cloud ‣Decouple storage from compute ‣Transparent switch from on premise Hadoop to DataProc and Cloud Storage ‣Entry point executable in base image ‣Auth, config, dynamic cluster allocation
- 20. Where are we now? ‣Two squads are using dockerized pipelines in production ‣Still using luigi, pull based dependencies ‣Styx, execution as service soon in prod ‣Google cloud migration as we speak ‣Docker drives transparent migration
- 21. Some practical docker advice ‣Reproducible normalised builds ‣Explicit versioning ‣Split code, configuration, secrets ‣github.com/spotify/dockerfile-maven
- 22. Thank you! Don’t be a stranger [email protected] @blippie