Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack

Feb 28, 20173 likes481 views

This document proposes implementing a real-time centralized logging system using the Elastic Stack. It introduces Elastic Stack components like Filebeat, Elasticsearch, and Kibana. It then provides a use case of converting log timestamps to a standard sort format using Logstash filters like grok and date. The presenter works at WeMo Scooter, an electric scooter rental startup aiming to reduce emissions. He is interested in technologies like Elastic Stack, PostgreSQL, and Spark.

Implement Real-time Centralized logging System
by Elastic Stack
Len Chang, WeMo Scooter
1

Who am I
● 軟體工程師, 資料庫架構師, 資料科學家, HadoopCon 2015 Speaker
● 喜歡發呆, 做瑜珈, 跳舞, 和各種球類運動。當然，寫程式已經是生活的一部分
● 目前較關注和熟悉的技術為: C#, Python, Elastic Stack, PostgreSQL, Spark
● 目前任職於 WeMo Scooter, 我們是一間很有趣的小新創公司, 目標是希望推廣城市內的電動機車租貸服務，進而
減少廢氣排放與機車數量。
● Linkedin: https://tw.linkedin.com/in/huailunchang 2

Agenda
● Elastic stack
○ Arch.
■ Open Source
■ License
● How to establish a simple Elastic Stack ?
○ filebeat
○ elasticsearch
○ kibana
● Use Case: How to convert “log timestamp” to be “sort standard”
○ logstash
● Q & A
● WeMo Scooter
3

Overview
● Elastic's open source solutions solve a growing list of search and log analysis.
● Helps you take data from any source, any format and search, analyze, and
visualize it in real time.
5

Architecture
DB
Application
sys log
sys log
Web Console
Restful API
6

How to establish a simple Elastic Stack ?
9

https://www.elastic.co/downloads/beats
Beats - filebeat
11

filebeat - Index Template Setting
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html
13

Logs
filebeat - Index Template Description
hostname: string (No Analyze)
msg: string (Analyze)
msg_count: number
...
template.json
14

filebeat - tab && space
17
Remember to use
“space”

Elasticsearch
DB
Application
sys log
sys log
18

Elasticsearch - Deploy by RPM
Category Explanation Destination
conf Configuration files elasticsearch.yml and logging.yml. /etc/elasticsearch
conf Environment variables including heap size, file descriptors. /etc/sysconfig/elasticsearch
19

Elasticsearch - Heap Tuning
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
● Give (less than) Half Your Memory to Lucene
○ Lucene need memory to interact with the OS.
● Don’t Cross 32 GB!
○ Compressed oops(Ordinary object pointers) have a upper boundary (~ 32 GB)
■ 32-bit pointer can reference four billion objects, rather than four billion bytes
20

Kibana
DB
Application
sys log
sys log
21

<Use Case>
How to convert “log timestamp” to be
“sort standard”
22

Why we can’t use Beats to do it ?
DB
Application
sys log
sys log
25

Logstash
DB
Application
sys log
sys log
27

Logstash - Filter plugins
28
grok
● https://www.elastic.co/guide/en/logstash/current/p
lugins-filters-grok.html
● Parse arbitrary text and structure it.
● Grok is currently the best way in logstash to
parse crappy unstructured log data into
something structured and queryable.
date
● https://www.elastic.co/guide/en/logstash/current/p
lugins-filters-date.html
● The date filter is used for parsing dates from
fields, and then using that date or timestamp as
the logstash timestamp for the event.

$Logstash - Filter of Config 29 PREFIX_TIMESTAMP ^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}$

WeMo Scooter
32
Official Website
● http://www.wemoscooter.com/
Video Introduction
● https://www.youtube.com/watch?v=Ne1kg3KeoRs
If you want to be a software engineer with us….
● len.chang@wemoscooter.com
○ Assistant software engineer / software engineer
■ Django / Python
■ ASP.NET MVC 5 / C#
■ Others...

Charlie Reverte, VP of Engineering at AddThis, discusses lessons learned from processing large-scale web data. AddThis processes data from 14 million domains, including 100 billion monthly page views and 50,000 events per second. Reverte outlines challenges around distributed ID generation, counting unique values, joining distributed data, sampling large datasets, and deploying systems that invalidate over 1.4 billion browser caches. He advocates for loose coupling between systems using approaches like Kafka for asynchronous event logging. Reverte also discusses techniques for columnar compression, tunable quality of service, and open sourcing Hydra, AddThis' custom processing system optimized for real-time data.

Big data @ Hootsuite analtyicsClaudiu Coman

Redis OverviewHoang Long

This document provides an overview of Redis, including its creation, features, and persistence methods. Redis is an open-source, in-memory key-value store that was created by Salvatore Sanfilippo to solve the problem of storing real-time page view data from multiple websites. It features different data types that can be stored as keys, master-slave replication for availability, and two persistence methods: RDB, which takes periodic snapshots of the dataset, and AOF, which logs all write operations to reconstruct the dataset from disk on startup.

Introduction to new high performance storage engines in mongodb 3.0Henrik Ingo

This document provides an introduction to new high performance storage engines being introduced in MongoDB 2.8, including WiredTiger. It discusses how WiredTiger provides improved performance over MMAPv1 for both read and write heavy workloads through features like document level locking, write optimized data structures, and compression. The document also outlines different configuration options and tunables for WiredTiger to optimize performance based on factors like fitting the working data set in cache or disk.

RIPE74 - ARouteServer - IXP Automation Made EasyPier Carlo Chiodi

Speed Up Uber's Presto with AlluxioAlluxio, Inc.

Redis IUIsaiah Edem

Redis is an open-source in-memory key-value data store that is optimized for speed. It uses various data structures like strings, lists, sets, and hashes to store data. MongoDB is a document-oriented NoSQL database where data is stored in flexible, JSON-like documents within collections, similar to tables in a relational database. Both Redis and MongoDB can be useful alternatives to SQL databases for applications that require speed, large amounts of data, or flexible schemas.

FleetDBDiego Pacheco

FleetDB is a NoSQL document database that uses a schema-free and document-based data model. It stores all data in memory for high performance and persistence. FleetDB supports rich data structures, single document access across multiple tables, clear path to horizontal scalability without migrations, multi-record transactions, and excellent concurrency. Client libraries are available for Java, Python, and Clojure.

Lessons Learned Migrating 2+ Billion Documents at CraigslistJeremy Zawodny

Lessons Learned from Migrating 2+ Billion Documents at Craigslist outlines Craigslist's migration from MySQL to MongoDB. Some key lessons include: knowing your hardware limitations, that replica sets provide high availability during reboots, understanding your data types and sizes, and being aware of limitations with sharding and replica set re-sync processes. The migration addressed issues with their archive data storage and provided a more scalable and performant system.

Small intro to Big Data - Old versionSoftwareMill

Presented on Codemotion Warsaw 2016 and JDD 2016. Pig, Hive, Flink, Kafka, Zeppelin... if you now wonder if someone just tried to offend you or are those just Pokemon names, then this talk is just for you! Big Data is everywhere and new tools for it are released almost at the speed of new JavaScript frameworks. During this entry level presentation we will walk though the challenges which Big Data presents, reflect how big is big and introduce currently most fancy and popular (mostly open source) tools. We'll try to spark off interest in Big Data by showing application areas and by throwing ideas where you can later dive into.

MongoDB SF RubyMike Dirolf

Behind the Scenes at Coolblue - Feb 2017Pat Hermens

The Next Generation Software Stack: MeteorMongoDB

Meteor is the next take on agile development on the full JavaScript stack. Based on established JavaScript tools like Node, JQuery and Underscore, it still brings a fresh and integrated approach. And MongoDB is very much its heart: Minimongo implements a client side MongoDB API for manipulating your data model; Transparent replication of data between client and server; Using WebSockets, MongoDB oplog events replicate immediately to all clients, making it simple to do distributed applications "Google Docs style."

Mongo db intro newAbhinav Dhasmana

MongoDB is a non-SQL document-oriented database where data is stored in BSON format. It is well-suited for write-intensive applications, geo-spatial queries, flexible schemas where requirements may change, and scaling through sharding. Key advantages include very fast writes, no need for migrations when changing schemas, and built-in support for features like sharding, geospatial indexing, and logging to capped collections.

MongoDB for Oracle Experts - OUGF Harmony 2014 Henrik Ingo

KDB+ LiteSayanosauras

Kdb+ is a database and analytics software designed for processing large, diverse financial and market data in real-time. It uses the q programming language which allows for less code and faster execution compared to other languages. Kdb+ includes components for ingesting data from multiple sources, storing real-time and historical data, performing analytics using triggers without slowing performance, and returning query results in various formats. The q language provides SQL-like and time series querying capabilities along with built-in functions to minimize programming and data transferred over networks for efficient analytics. Users can get started with kdb+ by downloading and installing it, then executing queries from the command line or IDE.

Using MongoDB For BigData in 20 MinutesAndrás Fehér

This document provides an overview of using MongoDB for big data applications. It describes what MongoDB is, including that it is a open-source, document-oriented NoSQL database. It then covers some key MongoDB concepts and features like replica sets for high availability, sharding for horizontal scaling, and using MapReduce and Apache Spark with MongoDB. The document concludes that MongoDB is a fast, flexible, and scalable database solution for big data, though deploying sharded clusters requires planning.

Realtime Search Infrastructure at Craigslist (OpenWest 2014)Jeremy Zawodny

Bringing spatial love to your python applicationShekhar Gulati

This document summarizes a workshop presented by Shekhar Gulati on building a location-aware job search application using Flask and MongoDB and deploying it to OpenShift. The agenda includes learning about Flask and MongoDB basics, building the application, and deploying it to OpenShift in about 50 minutes. The document provides information on the tools used, includes code examples, and outlines the steps to create accounts, import data, and deploy the application to OpenShift.

KDB database (EPAM tech talks, Sofia, April, 2015)Martin Toshev

KDB is an in-memory column-oriented database that provides high-performance for real-time and historical large volumes of data. It is used widely in the financial industry. KDB supports the Q programming language for querying and manipulating data, and can be deployed in a distributed environment. The Java API provides simple connection and query methods to access a KDB database. KDB is well-suited for use cases involving capturing market data feeds and analyzing FIX messages.

Spry 2017Göran Krampe

Logging for ContainersEduardo Silva Pereira

Monogo db in-actionChi Lee

This document provides an overview and agenda for a MongoDB workshop. It begins with introductions and background on MongoDB. The agenda then covers basic CRUD operations and aggregation. It also discusses replication, sharding, and development uses for MongoDB. Replication in MongoDB uses replica sets for high availability and fault tolerance. Sharding partitions data across multiple servers to support large data volumes and high throughput. The document provides examples of shard key strategies and the production sharding architecture in MongoDB.

«Scrapy internals» Александр Сибиряков, Scrapinghubit-people

- Scrapy is a framework for web scraping that allows for extraction of structured data from HTML/XML through selectors like CSS and XPath. It provides features like an interactive shell, feed exports, encoding support, and more. - Scrapy is built on top of the Twisted asynchronous networking framework, which provides an event loop and deferreds. It handles protocols and transports like TCP, HTTP, and more across platforms. - Scrapy architecture includes components like the downloader, scraper, and item pipelines that communicate internally. Flow control is needed between these to limit memory usage and scheduling through techniques like concurrent item limits, memory limits, and delays between calls.

Introduction of search engineJinglun Li

This document provides an overview of the major components of a search engine, including spiders that crawl websites, analysis of crawled pages for structuring and ranking, indexing of page content, and searching of the index. It describes common problems search engines face like duplication and optimization of downloads, as well as solutions involving depth-first versus breadth-first crawling and use of robots.txt files. Methods for word segmentation, duplicate detection, and calculating page importance through PageRank are also outlined. The document concludes with discussions of classic search models and ways to optimize merging of indexes.

MongoDB - Warehouse and Aggregator of EventsMaxim Ligus

This document discusses using MongoDB to warehouse and aggregate events from different sources. MongoDB can scale simply to handle large volumes of event data, provide 99.999% uptime, and integrate smoothly with other infrastructure components. It describes how MongoDB can distribute data across multiple shards to improve performance and scale to handle large workloads of event data over long retention periods in a cost effective manner using reasonable hardware requirements. The document compares MongoDB to Elasticsearch and provides an overview of how event data would flow through the system from ingestion to storage to retrieval.

MongoSF - Spatial MongoDB in OpenShift - script fileSteven Pousty

This document provides instructions for adding spatial data to MongoDB. It details how to import a JSON file containing coordinate data into a MongoDB database hosted on OpenShift, build a 2D index on the coordinate field, perform spatial queries to find documents near a given location, insert new documents with location data to new and existing collections, and update the notes field of an existing document.

Fluentd Intro for OpenShift Commons BriefingEduardo Silva Pereira

This document discusses logging in cloud native environments and introduces Fluentd and Fluent Bit as solutions. Fluentd is well-suited for log aggregation due to its pluggable architecture, reliability, and integration with Docker and Kubernetes. Fluent Bit is designed as a lightweight log forwarder that can send logs from edge nodes to Fluentd for aggregation. Using Fluent Bit for forwarding reduces resource usage on edge nodes. Both support features important for cloud environments like backpressure handling.

Tugas 4 0317-imelda felicia-1412510545imeldafelicia

Dokumen tersebut membahas tentang database, termasuk pengertian database, jenis database, dan perbedaan antara database relasional dan non-relasional (NoSQL). Database dijelaskan sebagai kumpulan informasi yang disimpan secara sistematis untuk memperoleh informasi, sedangkan database relasional menyimpan data dalam bentuk tabel yang saling berhubungan dan NoSQL menyederhanakan proses database dengan menghilangkan redudansi data.

ServerlessYoung Yang

More Related Content

What's hot (20)

Lessons Learned Migrating 2+ Billion Documents at CraigslistJeremy Zawodny

Small intro to Big Data - Old versionSoftwareMill

MongoDB SF RubyMike Dirolf

Behind the Scenes at Coolblue - Feb 2017Pat Hermens

The Next Generation Software Stack: MeteorMongoDB

Mongo db intro newAbhinav Dhasmana

MongoDB for Oracle Experts - OUGF Harmony 2014 Henrik Ingo

KDB+ LiteSayanosauras

Using MongoDB For BigData in 20 MinutesAndrás Fehér

Realtime Search Infrastructure at Craigslist (OpenWest 2014)Jeremy Zawodny

Bringing spatial love to your python applicationShekhar Gulati

KDB database (EPAM tech talks, Sofia, April, 2015)Martin Toshev

Spry 2017Göran Krampe

Logging for ContainersEduardo Silva Pereira

Monogo db in-actionChi Lee

«Scrapy internals» Александр Сибиряков, Scrapinghubit-people

Introduction of search engineJinglun Li

MongoDB - Warehouse and Aggregator of EventsMaxim Ligus

MongoSF - Spatial MongoDB in OpenShift - script fileSteven Pousty

Fluentd Intro for OpenShift Commons BriefingEduardo Silva Pereira

Lessons Learned Migrating 2+ Billion Documents at CraigslistJeremy Zawodny

Small intro to Big Data - Old versionSoftwareMill

MongoDB SF RubyMike Dirolf

Behind the Scenes at Coolblue - Feb 2017Pat Hermens

The Next Generation Software Stack: MeteorMongoDB

Mongo db intro newAbhinav Dhasmana

MongoDB for Oracle Experts - OUGF Harmony 2014 Henrik Ingo

KDB+ LiteSayanosauras

Using MongoDB For BigData in 20 MinutesAndrás Fehér

Realtime Search Infrastructure at Craigslist (OpenWest 2014)Jeremy Zawodny

Bringing spatial love to your python applicationShekhar Gulati

KDB database (EPAM tech talks, Sofia, April, 2015)Martin Toshev

Spry 2017Göran Krampe

Logging for ContainersEduardo Silva Pereira

Monogo db in-actionChi Lee

«Scrapy internals» Александр Сибиряков, Scrapinghubit-people

Introduction of search engineJinglun Li

MongoDB - Warehouse and Aggregator of EventsMaxim Ligus

MongoSF - Spatial MongoDB in OpenShift - script fileSteven Pousty

Fluentd Intro for OpenShift Commons BriefingEduardo Silva Pereira

Viewers also liked (20)

Tugas 4 0317-imelda felicia-1412510545imeldafelicia

ServerlessYoung Yang

2015 Internet Trends ReportIQbal KHan

The document summarizes key trends from the 2015 Internet Trends report by Mary Meeker. It outlines that while global internet and smartphone user growth is still solid, the growth rate is slowing as adoption increases. It also notes that incremental users will be harder to obtain as adoption depends more on developing markets. Internet usage and engagement growth remains strong, especially for mobile video. Mobile advertising is growing faster than desktop but still lags in share of total internet advertising spending. The document also highlights new advertising formats and payment options optimized for mobile usage as well as the rise of vertical video viewing. Finally, it discusses how enterprise technology startups are reimagining business processes by addressing prior pain points in areas like communications, payments, analytics and

Comparing 30 MongoDB operations with Oracle SQL statementsLucas Jellema

Tracxn Research - Chatbots Landscape, February 2017Tracxn

Startup & VC Tech Trends Dave McClure

Dave McClure, a founding partner at 500 Startups, gave a presentation on technology trends in 2017. He discussed how startups have become cheaper, faster, and better. He also talked about how VCs are making lots of small bets through many new, small funds. McClure highlighted several investment areas including fintech, AI, AR/VR, blockchain, and other new technologies. He emphasized 500 Startups' strategy of making many small investments to find the next big winners.

Tracxn Research - Mobile Advertising Landscape, February 2017Tracxn

Tracxn Research - Construction Tech Landscape, February 2017Tracxn

The document provides an overview of investment trends in the construction technology sector from 2008 to 2016. It finds that the number of startups founded and funding rounds increased year over year, with total funding reaching $491 million in 2016. Early stage funding amounts and average ticket sizes also increased over time, with average early stage deals reaching $11.9 million in 2016. The report also analyzes subsectors of construction tech and provides examples of interesting startups.

Introduction to Data ScienceANOOP V S

This document provides an introduction to data science and analytics. It discusses why data science jobs are in high demand, what skills are needed for these roles, and common types of analytics including descriptive, predictive, and prescriptive. It also covers topics like machine learning, big data, structured vs unstructured data, and examples of companies that utilize data and analytics like Amazon and Facebook. The document is intended to explain key concepts in data science and why attending a talk on this topic would be beneficial.

#askSAP: Journey to the Cloud: SAP Strategy and Roadmap for Cloud and Hybrid ...SAP Analytics

www.sap.com/businessobjects-cloud. The momentum of customers moving to the SAP BusinessObjects Cloud is rapidly accelerating – and so are the innovations being introduced by SAP. New features and functionality for cloud and on premise with SAP BusinessObjects Enterprise offer hybrid use cases that organizations can take advantage of as they embark on their journey to the cloud. View the webinar reply at http://webinars.sap.com/asksap-webinar-series/en/home#section_3.

Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...Lucas Jellema

Tracxn Research - Insurance Tech Landscape, February 2017Tracxn

This document provides an overview of investment trends in the insurance technology sector from 2010 to 2016. It finds that the number of companies founded, funding rounds completed, and total funding amount have all increased significantly year-over-year in this period. The largest investments in the last year went to Oscar, Clover Health, Metromile, and Gusto. The report also analyzes trends by funding stage and subsectors within insurtech.

Enabling the Real Time Analytical EnterpriseHortonworks

This document discusses enabling real-time analytics in the enterprise. It begins with an overview of the challenges of real-time analytics due to non-integrated systems, varied data types and volumes, and data management complexity. A case study on real-time quality analytics in automotive is presented, highlighting the need to analyze varied data sources quickly to address issues. The Hortonworks/Attunity solution is then introduced using Attunity Replicate to integrate data from various sources in real-time into Hortonworks Data Platform for analysis. A brief demonstration of data streaming from a database into Kafka and then Hortonworks Data Platform is shown.

AWSでアプリ開発するなら知っておくべことKeisuke Nishitani

Slides for RFID-MIMO Prototype based on GnuRadioAmelia Jiménez Sánchez

This document discusses the development of an RFID-MIMO prototype using GNU Radio. The goal is to set up a testbed using USRP devices to implement MIMO for UHF RFID systems and compare performance to single antenna solutions. It describes implementing a single antenna listener first, then adding selection combining as a diversity combining technique with two antennas. Results show selection combining improves read range and listener success ratio over a single antenna. The document concludes by proposing implementing maximal ratio combining next to further improve performance.

Elasticsearch 5.0 les nouveautésMathieu Elie

Hadoop con 2016_9_10_王經篤(Jing-Doo Wang)Jing-Doo Wang

This document summarizes a presentation on potential applications using the class frequency distribution of maximal repeats from tagged sequential data. It discusses using maximal repeat patterns and their frequency distributions over time to analyze trends in topic histories from literature, detect anomalies in manufacturing processes for quality control, and identify distinguishing patterns in genomic sequences. Potential applications discussed include text mining historical archives, individualized learning based on topic histories, detecting changes in language for elderly assessment, monitoring new word adoption, and integrating IoT sensor data with product traceability systems for industrial quality assurance.

Yarn Resource Management Using Machine Learningojavajava

How to plan a hadoop cluster for testing and production environmentAnna Yen

Introduction au langage GoSylvain Wallez

Tugas 4 0317-imelda felicia-1412510545imeldafelicia

ServerlessYoung Yang

2015 Internet Trends ReportIQbal KHan

Comparing 30 MongoDB operations with Oracle SQL statementsLucas Jellema

Tracxn Research - Chatbots Landscape, February 2017Tracxn

Startup & VC Tech Trends Dave McClure

Tracxn Research - Mobile Advertising Landscape, February 2017Tracxn

Tracxn Research - Construction Tech Landscape, February 2017Tracxn

Introduction to Data ScienceANOOP V S

#askSAP: Journey to the Cloud: SAP Strategy and Roadmap for Cloud and Hybrid ...SAP Analytics

Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...Lucas Jellema

Tracxn Research - Insurance Tech Landscape, February 2017Tracxn

Enabling the Real Time Analytical EnterpriseHortonworks

AWSでアプリ開発するなら知っておくべことKeisuke Nishitani

Slides for RFID-MIMO Prototype based on GnuRadioAmelia Jiménez Sánchez

Elasticsearch 5.0 les nouveautésMathieu Elie

Hadoop con 2016_9_10_王經篤(Jing-Doo Wang)Jing-Doo Wang

Yarn Resource Management Using Machine Learningojavajava

How to plan a hadoop cluster for testing and production environmentAnna Yen

Introduction au langage GoSylvain Wallez

Similar to Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack (20)

Kibana+ElasticSearch+LogStash to handle Log messages on Prod serversHYS Enterprise

The document provides an overview of logging and the Elastic Stack (Elasticsearch, Logstash, Kibana) for log analysis. It discusses what logs are, popular logging processors like Loggly and Splunk, and the key components of the Elastic Stack - Elasticsearch for storage, Logstash for processing, and Kibana for visualization. It also gives a live demo example of using Logstash to ingest application logs and explore them in Kibana.

Experiences sharing about Lambda, Kinesis, and PostgresqlOkis Chuang

Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Hernan Costante

Nowadays in an increasingly more complex and dynamic network its not enough to be a regex ninja and storing only the logs you think you might need. From network traffic to custom logs you won't know which logs will be crucial to stop the next attacker, and if you are not planning to spend a half of your security budget in a commercial solution we will show you a way to building you own SIEM with open source. The talk will go from how to build a powerful logging environment for your organization to scaling on the cloud and storing everything forever. We will walk through how to build such a system with open source solutions as Elasticsearch and Hadoop, and creating your own custom monitoring rules to monitor everything you need. The talk will also include how to secure the environment and allow restricted access to other teams as well as avoiding common pitfalls and ensuring compliance standards.

The Parquet Format and Performance Optimization OpportunitiesDatabricks

The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads. As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is ‘many small files’, and will discuss the open-source Delta Lake format in relation to this and Parquet in general. This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.

Big Data in 200 km/h | AWS Big Data Demystified #1.3 Omid Vahdaty

What we're about A while ago I entered the challenging world of Big Data. As an engineer, at first, I was not so impressed with this field. As time went by, I realised more and more, The technological challenges in this area are too great to master by one person. Just look at the picture in this articles, it only covers a small fraction of the technologies in the Big Data industry… Consequently, I created a meetup detailing all the challenges of Big Data, especially in the world of cloud. I am using AWS infrastructure to answer the basic questions of anyone starting their way in the big data world. how to transform data (TXT, CSV, TSV, JSON) into Parquet, ORCwhich technology should we use to model the data ? EMR? Athena? Redshift? Spectrum? Glue? Spark? SparkSQL?how to handle streaming?how to manage costs?Performance tips?Security tip?Cloud best practices tips? Some of our online materials: Website: https://big-data-demystified.ninja/ Youtube channels: https://www.youtube.com/channel/UCzeGqhZIWU-hIDczWa8GtgQ?view_as=subscriber https://www.youtube.com/channel/UCMSdNB0fGmX5dXI7S7Y_LFA?view_as=subscriber Meetup: https://www.meetup.com/AWS-Big-Data-Demystified/ https://www.meetup.com/Big-Data-Demystified Facebook Group : https://www.facebook.com/groups/amazon.aws.big.data.demystified/ Facebook page (https://www.facebook.com/Amazon-AWS-Big-Data-Demystified-1832900280345700/) Audience: Data Engineers Data Science DevOps Engineers Big Data Architects Solution Architects CTO VP R&D

Serverless for High Performance ComputingLuciano Mammino

Serverless is great for web applications and APIs, but this does not mean it cannot be used successfully for other use cases. In this talk, we will discuss a successful application of serverless in the field of High Performance Computing. Specifically we will discuss how Lambda, Fargate, Kinesis and other serverless technologies are being used to run sophisticated financial models at one of the major reinsurance companies in the World. We we learn about the architecture, the tradeoffs, some challenges and some unresolved pain points. Most importantly, we'll find out if serverless can be a great fit for HPC and if we can finally stop managing those boring EC2 instances!

High performance json- postgre sql vs. mongodbWei Shan Ang

PostgreSQL and MongoDB were benchmarked for performance on common operations like inserts, updates, and selects using a JSON document format. The key findings were: 1) PostgreSQL generally had lower latency but required extensive tuning to achieve high performance, while MongoDB delivered reasonable performance out of the box. 2) MongoDB showed unstable throughput and latency over time due to a cache eviction bug. 3) PostgreSQL did not scale well to large connection loads without connection pooling, while MongoDB scaled horizontally more easily. 4) Both databases had pros and cons for their data models, query capabilities, and upgrade processes. The optimal choice depends on an application's specific requirements.

AWS Big Data Demystified #1: Big data architecture lessons learned Omid Vahdaty

AWS Big Data Demystified #1: Big data architecture lessons learned . a quick overview of a big data techonoligies, which were selected and disregard in our company The video: https://youtu.be/l5KmaZNQxaU dont forget to subcribe to the youtube channel The website: https://amazon-aws-big-data-demystified.ninja/ The meetup : https://www.meetup.com/AWS-Big-Data-Demystified/ The facebook group : https://www.facebook.com/Amazon-AWS-Big-Data-Demystified-1832900280345700/

Security Monitoring for big Infrastructures without a Million Dollar budgetJuan Berner

SOSCON 2016 JerryScriptSamsung Open Source Group

JerryScript is a lightweight JavaScript engine optimized for microcontrollers and IoT devices. It has a small base footprint of only 3KB of RAM and implements ECMAScript 5.1. JerryScript parses code directly to compact bytecode without an intermediate representation, uses compressed pointers and value representations to reduce memory usage, and is highly portable across platforms. A demo of a Pong game showed JerryScript running on a microcontroller driving an LED matrix with gameplay shared across devices. Future work will focus on additional optimizations, debugging support, and selected ES6 features.

Data Science in the Cloud @StitchFixC4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lGNybu. Stefan Krawczyk discusses how his team at StitchFix use the cloud to enable over 80 data scientists to be productive. He also talks about prototyping ideas, algorithms and analyses, how they set up & keep schemas in sync between Hive, Presto, Redshift & Spark and make access easy for their data scientists, etc. Filmed at qconsf.com.. Stefan Krawczyk is Algo Dev Platform Lead at StitchFix, where he’s leading development of the algorithm development platform. He spent formative years at Stanford, LinkedIn, Nextdoor & Idibon, working on everything from growth engineering, product engineering, data engineering, to recommendation systems, NLP, data science and business intelligence.

Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...javier ramirez

Los sistemas distribuidos son difíciles. Los sistemas distribuidos de alto rendimiento, más. Latencias de red, mensajes sin confirmación de recibo, reinicios de servidores, fallos de hardware, bugs en el software, releases problemáticas, timeouts... hay un montón de motivos por los que es muy difícil saber si un mensaje que has enviado se ha recibido y procesado correctamente en destino. Así que para asegurar mandas el mensaje otra vez.. y otra... y cruzas los dedos para que el sistema del otro lado tenga tolerancia a los duplicados. QuestDB es una base de datos open source diseñada para alto rendimiento. Nos queríamos asegurar de poder ofrecer garantías de "exactly once", deduplicando mensajes en tiempo de ingestión. En esta charla, te cuento cómo diseñamos e implementamos la palabra clave DEDUP en QuestDB, permitiendo deduplicar y además permitiendo Upserts en datos en tiempo real, añadiendo solo un 8% de tiempo de proceso, incluso en flujos con millones de inserciones por segundo. Además, explicaré nuestra arquitectura de log de escrituras (WAL) paralelo y multithread. Por supuesto, todo esto te lo cuento con demos, para que veas cómo funciona en la práctica.

AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | EnglishOmid Vahdaty

PL22 - Backup and Restore Performance.pptxVinicius M Grippa

This document compares the performance of different MySQL backup and restore tools including mysqldump, mydumper, mysqlpump, Xtrabackup, and MySQL shell. It describes benchmark tests conducted on a 96GB MySQL database using these tools under various compression options. The results show that Xtrabackup offers the best balance of backup speed and size when compression is used. mydumper/myloader and MySQL shell provide the fastest logical backups while mysqlpump has high backup capacity but slow restores due to lack of parallelism. In conclusion, compression does not significantly impact performance but saves disk space, and parallelism provides a major boost that is limited by I/O capacity. For routine backups, the presenter

Benchmarking for postgresql workloads in kubernetesDoKC

ABSTRACT OF THE TALK 6 months have passed since our last DoK webinar about benchmarking PostgreSQL workloads in a Kubernetes environment. In the meantime, many things have happened at EDB, and we’re happy to share what we’ve learned in this timeframe. We’ll use cnp-bench and cnp-sandbox to help us describe some of the challenges we might face when running PostgreSQL workloads, how to spot them, and what actions to take to make your databases healthier and more longeve. cnp-bench is a collection of Helm charts that help run storage and database benchmarks, using popular open source tools like fio, pgbench, and HammerDB. cnp-sandbox is a Helm chart that sets up a Prometheus/Grafana stack, including basic metrics and dashboards for Cloud Native PostgreSQL, the Kubernetes operator developed by EDB. Both cnp-sandbox and cnp-bench are open source and recommended for development, testing, and pre-production environments only. BIO A long time open-source programmer and entrepreneur, Gabriele has a degree in Statistics from the University of Florence. After having consistently contributed to the growth of 2ndQuadrant and its members through nurturing a lean and devops culture, he is now leading the Cloud Native initiative at EDB. Gabriele lives in Prato, a small but vibrant city located in the northern part of Tuscany, Italy - famous for having hosted the first European PostgreSQL conferences. His second home is Melbourne, Australia, where he studied at Monash University and worked in the ICT sector. He loves playing the Blues with his Fender Stratocaster, but his major passions are called Elisabeth and Charlotte! KEY TAKE-AWAYS FROM THE TALK - A methodology for benchmarking a PostgreSQL database in Kubernetes - Open source set of tools for benchmarking a PostgreSQL database in Kubernetes - Reasons why benchmarking both the storage and the database is important https://github.com/EnterpriseDB/cnp-sandbox https://github.com/EnterpriseDB/cnp-bench

Dfrws eu 2014 rekall workshopTamas K Lengyel

Log forwarding at ScaleEduardo Silva Pereira

VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics

VictoriaLogs Preview - Aliaksandr Valialkin * Existing open source log management systems - ELK (ElasticSearch) stack: Pros & Cons - Grafana Loki: Pros & Cons * What is VictoriaLogs - Open source log management system from VictoriaMetrics - Easy to setup and operate - Scales vertically and horizontally - Optimized for low resource usage (CPU, RAM, disk space) - Accepts data from Logstash and Fluentbit in Elasticsearch format - Accepts data from Promtail in Loki format - Supports stream concept from Loki - Provides easy to use yet powerful query language - LogsQL * LogsQL Examples - Search by time - Full-text search - Combining search queries - Searching arbitrary labels * Log Streams - What is a log stream? - LogsQL examples: querying log streams - Stream labels vs log labels * LogsQL: stats over access logs * VictoriaLogs: CLI Integration * VictoriaLogs Recap

Fluent Bit: Log Forwarding at ScaleEduardo Silva Pereira

This document summarizes a presentation about log forwarding at scale. It discusses how logging works internally and requires understanding the logging pipeline of parsing, filtering, buffering and routing logs. It then introduces Fluent Bit as a lightweight log forwarder that can be used to cheaply forward logs from edge nodes to log aggregators in a scalable way, especially in cloud native environments like Kubernetes. Hands-on demos show how Fluent Bit can parse and add metadata to Kubernetes logs.

OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...NETWAYS

At Uber we use high cardinality monitoring to observe and detect issues with our 4,000 microservices running on Mesos and across our infrastructure systems and servers. We’ll cover how we put the resulting 6 billion plus time series to work in a variety of different ways, auto-discovering services and their usage of other systems at Uber, setting up and tearing down alerts automatically for services, sending smart alert notifications that rollup different failures into individual high level contextual alerts, and more. We’ll also talk about how we accomplish all this with a global view of our systems with M3, our open source metrics platform. We’ll take a deep dive look at how we use M3DB, now available as an open source Prometheus long term storage backend, to horizontally scale our metrics platform in a cost efficient manner with a system that’s still sane to operate with petabytes of metrics data.

Kibana+ElasticSearch+LogStash to handle Log messages on Prod serversHYS Enterprise

Experiences sharing about Lambda, Kinesis, and PostgresqlOkis Chuang

Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Hernan Costante

The Parquet Format and Performance Optimization OpportunitiesDatabricks

Big Data in 200 km/h | AWS Big Data Demystified #1.3 Omid Vahdaty

Serverless for High Performance ComputingLuciano Mammino

High performance json- postgre sql vs. mongodbWei Shan Ang

AWS Big Data Demystified #1: Big data architecture lessons learned Omid Vahdaty

Security Monitoring for big Infrastructures without a Million Dollar budgetJuan Berner

SOSCON 2016 JerryScriptSamsung Open Source Group

Data Science in the Cloud @StitchFixC4Media

Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...javier ramirez

AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | EnglishOmid Vahdaty

PL22 - Backup and Restore Performance.pptxVinicius M Grippa

Benchmarking for postgresql workloads in kubernetesDoKC

Dfrws eu 2014 rekall workshopTamas K Lengyel

Log forwarding at ScaleEduardo Silva Pereira

VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics

Fluent Bit: Log Forwarding at ScaleEduardo Silva Pereira

OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...NETWAYS

More from Len Chang (8)

The issue discussion between dml and ddl deploymentLen Chang

COSCUP 2019 - The discussion between Knex.js and PostgreSQLLen Chang

This document discusses using knex.js to solve schema change problems. Knex.js is a SQL query builder that can be used with Node.js and relational databases. It features query building, pooling, transactions, migrations, and seed files. Migrations allow defining schema changes, and seed files populate data. Testing, automatic deployment, and using only JavaScript helps avoid crashes during schema updates and reduces operation errors.

DevOps Taiwan meetup #19Len Chang

這次我主要會分享 bitbucket pipeline & gitlab-ci 的一些設定和操作流程，以及我自己和同事在踩雷時的一些心得分享。 bitbucket pipeline & gitlab-ci 這兩款都算是相對知名的 CI/CD Tools，雖然一般來說大家可能比較常用類似 Jenkins & Travis CI 之類的產品。但其實我個人覺得，以小團隊的使用情境來說 bitbucket & gitlab 是相當不錯的，建議大家都可以去玩玩看。另外，這場分享中間可能也會穿插一些使用 Elastic 的 logstash 做 monitor 碰到的一些雷區和解法，基本上就看時間夠不夠講嘍～敬請期待！

BI in XuennLen Chang

This document provides an overview of a business intelligence (BI) system architecture. It includes a product database using Attunity for change data capture fed into a Teradata data warehouse. An ETL system extracts and transforms the data from the warehouse for analysis in Tableau, a BI reporting tool. Centralized logging of the database, applications, and web console are stored in a separate logging database.

2014 Pixnet Hackathonh - EXIF MiningLen Chang

Agile scrum in startup Len Chang

Hadoop Con2015 - The Data Scientist’s ToolboxLen Chang

Spam user detection reportLen Chang

The document discusses spam user detection through machine learning tools from AWS and Azure. It proposes two algorithms: detecting spam users by their time frequency of actions, flagging those with over 1 million actions; and detecting spam users by analyzing correlations between the articles they interact with, flagging those with high correlations between article titles and counts. The presentation thanks the audience at the end.

The issue discussion between dml and ddl deploymentLen Chang

COSCUP 2019 - The discussion between Knex.js and PostgreSQLLen Chang

DevOps Taiwan meetup #19Len Chang

BI in XuennLen Chang

2014 Pixnet Hackathonh - EXIF MiningLen Chang

Agile scrum in startup Len Chang

Hadoop Con2015 - The Data Scientist’s ToolboxLen Chang

Spam user detection reportLen Chang

Recently uploaded (20)

Not So Common Memory Leaks in Java WebinarTier1 app

This SlideShare presentation is from our May webinar, “Not So Common Memory Leaks & How to Fix Them?”, where we explored lesser-known memory leak patterns in Java applications. Unlike typical leaks, subtle issues such as thread local misuse, inner class references, uncached collections, and misbehaving frameworks often go undetected and gradually degrade performance. This deck provides in-depth insights into identifying these hidden leaks using advanced heap analysis and profiling techniques, along with real-world case studies and practical solutions. Ideal for developers and performance engineers aiming to deepen their understanding of Java memory management and improve application stability.

Interactive odoo dashboards for sales, CRM , Inventory, Invoice, Purchase, Pr...AxisTechnolabs

Interactive Odoo Dashboard for various business needs can provide users with dynamic, visually appealing dashboards tailored to their specific requirements. such a module that could support multiple dashboards for different aspects of a business ✅Visit And Buy Now : https://bit.ly/3VojWza ✅This Interactive Odoo dashboard module allow user to create their own odoo interactive dashboards for various purpose. App download now : Odoo 18 : https://bit.ly/3VojWza Odoo 17 : https://bit.ly/4h9Z47G Odoo 16 : https://bit.ly/3FJTEA4 Odoo 15 : https://bit.ly/3W7tsEB Odoo 14 : https://bit.ly/3BqZDHg Odoo 13 : https://bit.ly/3uNMF2t Try Our website appointment booking odoo app : https://bit.ly/3SvNvgU 👉Want a Demo ?📧 [email protected] ➡️Contact us for Odoo ERP Set up : 091066 49361 👉Explore more apps: https://bit.ly/3oFIOCF 👉Want to know more : 🌐 https://www.axistechnolabs.com/ #odoo #odoo18 #odoo17 #odoo16 #odoo15 #odooapps #dashboards #dashboardsoftware #odooerp #odooimplementation #odoodashboardapp #bestodoodashboard #dashboardapp #odoodashboard #dashboardmodule #interactivedashboard #bestdashboard #dashboard #odootag #odooservices #odoonewfeatures #newappfeatures #odoodashboardapp #dynamicdashboard #odooapp #odooappstore #TopOdooApps #odooapp #odooexperience #odoodevelopment #businessdashboard #allinonedashboard #odooproducts

Download YouTube By Click 2025 Free Full Activatedsaniamalik72555

Avast Premium Security Crack FREE Latest Version 2025mu394968

🌍📱👉COPY LINK & PASTE ON GOOGLE https://dr-kain-geera.info/👈🌍 Avast Premium Security is a paid subscription service that provides comprehensive online security and privacy protection for multiple devices. It includes features like antivirus, firewall, ransomware protection, and website scanning, all designed to safeguard against a wide range of online threats, according to Avast. Key features of Avast Premium Security: Antivirus: Protects against viruses, malware, and other malicious software, according to Avast. Firewall: Controls network traffic and blocks unauthorized access to your devices, as noted by All About Cookies. Ransomware protection: Helps prevent ransomware attacks, which can encrypt your files and hold them hostage. Website scanning: Checks websites for malicious content before you visit them, according to Avast. Email Guardian: Scans your emails for suspicious attachments and phishing attempts. Multi-device protection: Covers up to 10 devices, including Windows, Mac, Android, and iOS, as stated by 2GO Software. Privacy features: Helps protect your personal data and online privacy. In essence, Avast Premium Security provides a robust suite of tools to keep your devices and online activity safe and secure, according to Avast.

Maxon CINEMA 4D 2025 Crack FREE Download LINKyounisnoman75

⭕️➡️ FOR DOWNLOAD LINK : http://drfiles.net/ ⬅️⭕️ Maxon Cinema 4D 2025 is the latest version of the Maxon's 3D software, released in September 2024, and it builds upon previous versions with new tools for procedural modeling and animation, as well as enhancements to particle, Pyro, and rigid body simulations. CG Channel also mentions that Cinema 4D 2025.2, released in April 2025, focuses on spline tools and unified simulation enhancements. Key improvements and features of Cinema 4D 2025 include: Procedural Modeling: New tools and workflows for creating models procedurally, including fabric weave and constellation generators. Procedural Animation: Field Driver tag for procedural animation. Simulation Enhancements: Improved particle, Pyro, and rigid body simulations. Spline Tools: Enhanced spline tools for motion graphics and animation, including spline modifiers from Rocket Lasso now included for all subscribers. Unified Simulation & Particles: Refined physics-based effects and improved particle systems. Boolean System: Modernized boolean system for precise 3D modeling. Particle Node Modifier: New particle node modifier for creating particle scenes. Learning Panel: Intuitive learning panel for new users. Redshift Integration: Maxon now includes access to the full power of Redshift rendering for all new subscriptions. In essence, Cinema 4D 2025 is a major update that provides artists with more powerful tools and workflows for creating 3D content, particularly in the fields of motion graphics, VFX, and visualization.

Who Watches the Watchmen (SciFiDevCon 2025)Allon Mureinik

Tests, especially unit tests, are the developers’ superheroes. They allow us to mess around with our code and keep us safe. We often trust them with the safety of our codebase, but how do we know that we should? How do we know that this trust is well-deserved? Enter mutation testing – by intentionally injecting harmful mutations into our code and seeing if they are caught by the tests, we can evaluate the quality of the safety net they provide. By watching the watchmen, we can make sure our tests really protect us, and we aren’t just green-washing our IDEs to a false sense of security. Talk from SciFiDevCon 2025 https://www.scifidevcon.com/courses/2025-scifidevcon/contents/680efa43ae4f5

F-Secure Freedome VPN 2025 Crack Plus Activation New Versionsaimabibi60507

Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.Dele Amefo

Societal challenges of AI: biases, multilinguism and sustainabilityJordi Cabot

Adobe After Effects Crack FREE FRESH version 2025kashifyounis067

🌍📱👉COPY LINK & PASTE ON GOOGLE http://drfiles.net/ 👈🌍 Adobe After Effects is a software application used for creating motion graphics, special effects, and video compositing. It's widely used in TV and film post-production, as well as for creating visuals for online content, presentations, and more. While it can be used to create basic animations and designs, its primary strength lies in adding visual effects and motion to videos and graphics after they have been edited. Here's a more detailed breakdown: Motion Graphics: . After Effects is powerful for creating animated titles, transitions, and other visual elements to enhance the look of videos and presentations. Visual Effects: . It's used extensively in film and television for creating special effects like green screen compositing, object manipulation, and other visual enhancements. Video Compositing: . After Effects allows users to combine multiple video clips, images, and graphics to create a final, cohesive visual. Animation: . It uses keyframes to create smooth, animated sequences, allowing for precise control over the movement and appearance of objects. Integration with Adobe Creative Cloud: . After Effects is part of the Adobe Creative Cloud, a suite of software that includes other popular applications like Photoshop and Premiere Pro. Post-Production Tool: . After Effects is primarily used in the post-production phase, meaning it's used to enhance the visuals after the initial editing of footage has been completed.

WinRAR Crack for Windows (100% Working 2025)sh607827

Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...Eric D. Schabell

It's time you stopped letting your telemetry data pressure your budgets and get in the way of solving issues with agility! No more I say! Take back control of your telemetry data as we guide you through the open source project Fluent Bit. Learn how to manage your telemetry data from source to destination using the pipeline phases covering collection, parsing, aggregation, transformation, and forwarding from any source to any destination. Buckle up for a fun ride as you learn by exploring how telemetry pipelines work, how to set up your first pipeline, and exploring several common use cases that Fluent Bit helps solve. All this backed by a self-paced, hands-on workshop that attendees can pursue at home after this session (https://o11y-workshops.gitlab.io/workshop-fluentbit).

Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Versionsaimabibi60507

Copy & Past Link👉👉 https://dr-up-community.info/ Pixologic ZBrush, now developed by Maxon, is a premier digital sculpting and painting software renowned for its ability to create highly detailed 3D models. Utilizing a unique "pixol" technology, ZBrush stores depth, lighting, and material information for each point on the screen, allowing artists to sculpt and paint with remarkable precision .

Get & Download Wondershare Filmora Crack Latest [2025]saniaaftab72555

Copy & Past Link 👉👉 https://dr-up-community.info/ Wondershare Filmora is a video editing software and app designed for both beginners and experienced users. It's known for its user-friendly interface, drag-and-drop functionality, and a wide range of tools and features for creating and editing videos. Filmora is available on Windows, macOS, iOS (iPhone/iPad), and Android platforms.

Meet the Agents: How AI Is Learning to Think, Plan, and CollaborateMaxim Salnikov

Designing AI-Powered APIs on Azure: Best Practices& ConsiderationsDinusha Kumarasiri

AI is transforming APIs, enabling smarter automation, enhanced decision-making, and seamless integrations. This presentation explores key design principles for AI-infused APIs on Azure, covering performance optimization, security best practices, scalability strategies, and responsible AI governance. Learn how to leverage Azure API Management, machine learning models, and cloud-native architectures to build robust, efficient, and intelligent API solutions

LEARN SEO AND INCREASE YOUR KNOWLDGE IN SOFTWARE INDUSTRYNidaFarooq10

Explaining GitHub Actions Failures with Large Language Models Challenges, In...ssuserb14185

GitHub Actions (GA) has become the de facto tool that developers use to automate software workflows, seamlessly building, testing, and deploying code. Yet when GA fails, it disrupts development, causing delays and driving up costs. Diagnosing failures becomes especially challenging because error logs are often long, complex and unstructured. Given these difficulties, this study explores the potential of large language models (LLMs) to generate correct, clear, concise, and actionable contextual descriptions (or summaries) for GA failures, focusing on developers’ perceptions of their feasibility and usefulness. Our results show that over 80% of developers rated LLM explanations positively in terms of correctness for simpler/small logs. Overall, our findings suggest that LLMs can feasibly assist developers in understanding common GA errors, thus, potentially reducing manual analysis. However, we also found that improved reasoning abilities are needed to support more complex CI/CD scenarios. For instance, less experienced developers tend to be more positive on the described context, while seasoned developers prefer concise summaries. Overall, our work offers key insights for researchers enhancing LLM reasoning, particularly in adapting explanations to user expertise. https://arxiv.org/abs/2501.16495

Revolutionizing Residential Wi-Fi PPT.pptxnidhisingh691197

Secure Test Infrastructure: The Backbone of Trustworthy Software DevelopmentShubham Joshi

Not So Common Memory Leaks in Java WebinarTier1 app

Interactive odoo dashboards for sales, CRM , Inventory, Invoice, Purchase, Pr...AxisTechnolabs

Download YouTube By Click 2025 Free Full Activatedsaniamalik72555

Avast Premium Security Crack FREE Latest Version 2025mu394968

Maxon CINEMA 4D 2025 Crack FREE Download LINKyounisnoman75

Who Watches the Watchmen (SciFiDevCon 2025)Allon Mureinik

F-Secure Freedome VPN 2025 Crack Plus Activation New Versionsaimabibi60507

Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.Dele Amefo

Societal challenges of AI: biases, multilinguism and sustainabilityJordi Cabot

Adobe After Effects Crack FREE FRESH version 2025kashifyounis067

WinRAR Crack for Windows (100% Working 2025)sh607827

Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...Eric D. Schabell

Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Versionsaimabibi60507

Get & Download Wondershare Filmora Crack Latest [2025]saniaaftab72555

Meet the Agents: How AI Is Learning to Think, Plan, and CollaborateMaxim Salnikov

Designing AI-Powered APIs on Azure: Best Practices& ConsiderationsDinusha Kumarasiri

LEARN SEO AND INCREASE YOUR KNOWLDGE IN SOFTWARE INDUSTRYNidaFarooq10

Explaining GitHub Actions Failures with Large Language Models Challenges, In...ssuserb14185

Revolutionizing Residential Wi-Fi PPT.pptxnidhisingh691197

Secure Test Infrastructure: The Backbone of Trustworthy Software DevelopmentShubham Joshi

Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack

1. Implement Real-time Centralized logging System by Elastic Stack Len Chang, WeMo Scooter 1

2. Who am I ● 軟體工程師, 資料庫架構師, 資料科學家, HadoopCon 2015 Speaker ● 喜歡發呆, 做瑜珈, 跳舞, 和各種球類運動。當然，寫程式已經是生活的一部分 ● 目前較關注和熟悉的技術為: C#, Python, Elastic Stack, PostgreSQL, Spark ● 目前任職於 WeMo Scooter, 我們是一間很有趣的小新創公司, 目標是希望推廣城市內的電動機車租貸服務，進而減少廢氣排放與機車數量。 ● Linkedin: https://tw.linkedin.com/in/huailunchang 2

3. Agenda ● Elastic stack ○ Arch. ■ Open Source ■ License ● How to establish a simple Elastic Stack ? ○ filebeat ○ elasticsearch ○ kibana ● Use Case: How to convert “log timestamp” to be “sort standard” ○ logstash ● Q & A ● WeMo Scooter 3

4. Elastic stack 4

5. Overview ● Elastic's open source solutions solve a growing list of search and log analysis. ● Helps you take data from any source, any format and search, analyze, and visualize it in real time. 5

6. Architecture DB Application sys log sys log Web Console Restful API 6

7. Open Source Hadoop Ecosystem 7

8. License 8

9. How to establish a simple Elastic Stack ? 9

10. Beats DB Application sys log sys log 10

11. https://www.elastic.co/downloads/beats Beats - filebeat 11

12. filebeat - core 12

13. filebeat - Index Template Setting https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html 13

14. Logs filebeat - Index Template Description hostname: string (No Analyze) msg: string (Analyze) msg_count: number ... template.json 14

15. filebeat - Index Template Content 15

16. filebeat - Index Template Demo 16

17. filebeat - tab && space 17 Remember to use “space”

18. Elasticsearch DB Application sys log sys log 18

19. Elasticsearch - Deploy by RPM Category Explanation Destination conf Configuration files elasticsearch.yml and logging.yml. /etc/elasticsearch conf Environment variables including heap size, file descriptors. /etc/sysconfig/elasticsearch 19

20. Elasticsearch - Heap Tuning https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html ● Give (less than) Half Your Memory to Lucene ○ Lucene need memory to interact with the OS. ● Don’t Cross 32 GB! ○ Compressed oops(Ordinary object pointers) have a upper boundary (~ 32 GB) ■ 32-bit pointer can reference four billion objects, rather than four billion bytes 20

21. Kibana DB Application sys log sys log 21

22. <Use Case> How to convert “log timestamp” to be “sort standard” 22

23. Kibana - Search 23

24. Kibana - log sample 24

25. Why we can’t use Beats to do it ? DB Application sys log sys log 25

26. Filebeat config 26

27. Logstash DB Application sys log sys log 27

28. Logstash - Filter plugins 28 grok ● https://www.elastic.co/guide/en/logstash/current/p lugins-filters-grok.html ● Parse arbitrary text and structure it. ● Grok is currently the best way in logstash to parse crappy unstructured log data into something structured and queryable. date ● https://www.elastic.co/guide/en/logstash/current/p lugins-filters-date.html ● The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.

29. Logstash - Filter of Config 29 PREFIX_TIMESTAMP ^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}

30. Logstash - Result 30

31. Q & A 31

32. WeMo Scooter 32 Official Website ● http://www.wemoscooter.com/ Video Introduction ● https://www.youtube.com/watch?v=Ne1kg3KeoRs If you want to be a software engineer with us…. ● [email protected] ○ Assistant software engineer / software engineer ■ Django / Python ■ ASP.NET MVC 5 / C# ■ Others...

33. Thanks 33

Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack (20)

More from Len Chang (8)

Recently uploaded (20)

Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack