Netflow Analysis using Elastic Stack - 조인중

Aug 9, 20164 likes1,106 views

NetFlow Analysis with Elastic Stack summarizes using NetFlow data with the Elastic Stack, including Logstash, Elasticsearch, and Kibana. It discusses what NetFlow is, the typical NetFlow architecture with flow exporters, collectors, and analysis applications. It then covers how to use Logstash for NetFlow collection, Elasticsearch for storage and analysis, and Kibana for visualization. The presentation provides examples of Logstash configuration for NetFlow input, filtering, and output, as well as templates for indexing NetFlow data in Elasticsearch.

NetFlow Analysis
with Elastic Stack
Elasticsearch Seoul Meetup
me@madkoala.net

Presenter
manager at SK broadband
me@madkoala.net
I don’t have any f**king Btv coupons!

What is NetFlow?
• Cisco router feature that provides the ability
to collect IP network traffic information on
interfaces
• Main components
– Flow Exporter
– Flow collector (logstash)
– Analysis application (elasticsearch)
– Visualization (kibana or whatever works)
Source: Wikipedia

NetFlow Protocol (v5 header)
Source: https://www.plixer.com/support/netflow_v5.html

NetFlow Protocol (v5 record)
Source: https://www.plixer.com/support/netflow_v5.html

What we can do with NetFlow
• Application usage analysis
• Troubleshoot network problem
• Track down bandwidth hogs
• Detect abnormal traffic usage
• Detect hosts infected by malwares
• …

Elastic Stack - NetFlow
• Logstash: NetFlow collector
• Elasticsearch: NetFlow storage & analysis
• Kibana: Visualization

Logstash Configuration (filter)
 Enrichment with geolocation info
 Calculate estimated traffic volume
(need to consider sampling rate)

The document summarizes the new features and improvements in Elastic Stack v5.0.0, including updates to Kibana, Elasticsearch, Logstash, and Beats. Key highlights include a redesigned Kibana interface, improved indexing performance in Elasticsearch, easier plugin development in Logstash, new data shippers and filtering capabilities in Beats, and expanded subscription support offerings. The Elastic Stack aims to help users build distributed applications and solve real problems through its integrated search, analytics, and data pipeline capabilities.

Kibana + timelion: time series with the elastic stackSylvain Wallez

The document discusses Kibana and Timelion, which are tools for visualizing and analyzing time series data in the Elastic Stack. It provides an overview of Kibana's evolution and capabilities for creating dashboards. Timelion is introduced as a scripting language that allows users to transform, aggregate, and calculate on time series data from multiple sources to create visualizations. The document demonstrates Timelion's expression language, which includes functions, combinations, filtering, and attributes to process and render time series graphs.

OpenContrail ImplementationsJakub Pavlik

How to Create a Modern IIoT Monitoring Solution On iOS Using Swift, MQTT and ...InfluxData

MOXIE IoT uses industry standards and cutting-edge technology to create a monitoring solution which provides their manufacturing customers with a single source of truth. Their iOS iPad app equips their customers to visualize and analyze industrial factory data in real time. Discover how they use a time series database, MQTT and Ultra-Wideband to accurately measure and track any object, device, personnel or machinery. In this webinar, Dr. Austin Gurley will dive into: MOXIE’s approach to app-centric asset monitoring Their ability to help customers streamline operations while bridging the gap between hardware and software How a time series database improves customer experience by enabling them to query, filter and parse their data Techniques for performing Flux queries in iOS Swift

OSMC 2018 | Distributed Tracing FAQ by Gianluca ArbezzanoNETWAYS

Microservices, containers and more in general distributed systems have opened a different point of view on our system and applications. We need to understand how a single event or requests cross our app jumping over networks, containers, virtual machines and sometime clod provider. There is a specific practice called distributed tracing to increase observability of systems like that. After this talk, you will have a solid idea around what tracing means, how you can instrument your applications and you will be ready to trace your application across many languages using open source technologies like OpenTracing, OpenCensus, Zipkin, Jaeger, InfluxDB. You will ask yourself how you survived until today!

OpenStack Ousts vCenter for DevOps and Unites IT Silos at AVG Technologies Jakub Pavlik

tcp cloud & AVG User Story. Does your IT department’s left hand talk to the right hand? Finally ours does at http://www.avg.com/eu-en/homepage! This is the story of OpenStack as our salvation, and important lessons learned in technology and IT politics. Our appsdev team’s devops abilities were being held ransom on vCenter, so we wanted public cloud agility for dev/test/staging. With the help of our IT partner…Full session details here: http://awe.sm/r9Ekr

OSMC 2018 | Current State of Icinga by Bernd ErkNETWAYS

- Bernd Erk introduced the current state of Icinga including the Icinga stack, core components like Icinga2 and Director, and new products like the x509 and vSphere modules. - Recent updates to Icinga2 focus on stability, namespaces, and syntax highlighting while Director added support for multiple instances and configuration baskets. - Icinga DB is targeted for Q1 2019 and will replace IDO with improved performance, flexibility and a new schema. - The presentation demonstrated the Icinga modules for vSphere and x509 as well as integrating Icinga with Node-RED for IoT use cases.

OpenContrail Experience tcp cloud OpenStack Summit TokyoJakub Pavlik

1) The document summarizes tcp cloud's work with OpenContrail since the last meetup, including releasing version Mk.20 of their enterprise IT solution based on OpenContrail 2.21 and OpenStack Kilo. 2) It describes tcp cloud's reference architecture and automated deployment of dev clusters using SaltStack formulas and Heat templates. 3) The document discusses tcp cloud's experience upgrading AVG Technologies' OpenContrail deployment from version 2.1 to 2.21 and issues encountered upgrading OpenStack releases from Juno to Kilo and Kilo to Liberty.

Kubernetes meetup bangalore december 2017 - v02Kumar Gaurav

The document summarizes key announcements and discussions from the Kubernetes Meetup in Bangalore on December 16th 2017. Over 4,100 people attended the event where it was announced that CNCF now hosts 14 open source projects and 31 new members joined at the Silver level. Some of the discussions focused on the growing adoption of Kubernetes for building cloud-native applications, new projects being developed for Kubernetes like Metaparticle and Kata Containers, and the rise of serverless architectures and service meshes like Istio. The 2018 Europe Conference for Kubernetes and Cloud Native was also announced.

Open Tracing, to order and understand your mess. - ApiConf 2017Gianluca Arbezzano

This about how many api calls your applications were doing 3-4 years ago, and think about how many integration and difference services your requests is crossing before to come back to the final destination. How do you know this step of your pipeline is taking too much time? What is taking 2 seconds to answer? Is it the authentication service? Maybe it's the invoice generation service or the notification platform. Open Tracing is a distributed tracing cross vendor and open source that help you to understand bottleneck and to profile the requests from where they arrive at the final user. In an ecosystem where microservices and as a service concept are growing this can be a real challenge. During this presentation, we will see how it works from a general point of view to land in some real implementation, examples, and demo.

2015 03-16-elk at-bsidesJeremy Cohoe

Jeremy Cohoe presented on using the ELK (Elasticsearch, Logstash, Kibana) stack for log analysis. He began with an overview of what ELK is and its components - Logstash parses logs, Elasticsearch is the database, and Kibana provides the GUI. Cohoe then demonstrated using ELK to monitor 802.11 client probes with a software defined radio and parse Flex pager signals. Finally, he discussed implementing ELK in production for a Linux central syslog system, including scaling out with Redis, common plugins, and cluster monitoring tools.

Operators experience and perspective on SDN with VLANs and L3 NetworksJakub Pavlik

利用K8S實現高可靠應用inwin stack

This document discusses using Kubernetes to implement highly reliable applications. It begins with an introduction to microservices and containers. It then provides an overview of Kubernetes, including Kubernetes clusters, concepts like deployments and pods. It concludes by demonstrating running a NodeRed chatbot application on Kubernetes, comparing the Kubernetes architecture to a traditional VM architecture. The benefits of the Kubernetes approach for this application are around resource utilization, scalability, load balancing and self-healing.

IoT Data Connector Fluent BitToru Takahashi

Time Series Tech Stack for the IoT EdgeInfluxData

No matter what type of IoT devices you have, or what your use case is for them, you’re going to end up producing a lot of time series data. What you use to handle it is going to be as important to your success as the code you write yourself. This talk will evaluate the available open source tools for the collection, activation, transmission and visualization of time series data on the IoT Edge, and demonstrate how to use them, together with InfluxDB, to solve various use cases for the Internet of Things.

THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.ioDevOpsDays Tel Aviv

Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors. In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.

Sahara Updates - Liberty EditionOpenStack Foundation

Sahara provides scalable data processing by provisioning and operating data processing clusters and scheduling jobs. In the Kilo release, Sahara added support for new plugins like Apache Storm, improved the Sahara UI for guided cluster creation and job execution, and added features like indirect VM access, an event log, and default templates via the CLI. Looking ahead, upcoming Liberty releases will focus on high availability for CDH and HDP, support for Spark, improved data sources handling, editing existing objects, and enhanced testing.

InfluxDB Community Office Hours September 2020 InfluxData

The document provides information about InfluxDB 2.0 going to general availability (GA) and the migration path for existing users. It discusses that the next release will be the first release candidate with bug fixes and no new features. It also outlines reasons users may want to hold off upgrading, including continuous queries and authorization changes. The document asks existing users to help with testing the upcoming release candidate.

InfluxData Architecture for IoT | Noah Crowley | InfluxDataInfluxData

Glance Updates - Liberty EditionOpenStack Foundation

The document summarizes updates to OpenStack Glance from the Kilo to Liberty releases. In Kilo, Glance added features like artifact repository, catalog indexing, image conversion/introspection, and support for multiple datastores in storage drivers. Liberty priorities included additional image listing filters, store refactor/cleaner API, encrypted/authenticated image support, and tag metadata CLI support. It also focused on Glance v3 API evolution and increasing adoption of the v2 API within OpenStack. The presenter invites questions by IRC, email, the OpenStack mailing list tagged [Glance], or the weekly Glance meeting.

Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...InfluxData

The document discusses InfluxDB 2.0 and its journey to becoming generally available as open source software. It provides an overview of where InfluxDB 2.0 is currently in development, outlines the challenges of delivering it as open source versus cloud software, and outlines the remaining work needed and timeline to achieve a 2.0 open source general availability release in early fall 2020. It also discusses the vision for InfluxDB 2.0 as a platform for working with time series data beyond just a database.

OpenNebulaConf2017EU: Elastic Clusters for Data Analysis by Carlos de Alfonso...OpenNebula Project

The document discusses the EUBra-BIGSEA project which aims to provide elastic data analysis clusters in OpenNebula. The project will deploy big data analytics services on cloud infrastructure using TOSCA specifications. It will define a security framework and use the Infrastructure Manager to deploy virtual machine images and configure instances. The Elastic Clusters for Cloud Computing (EC3) will leverage CLUES to dynamically scale the clusters based on resource usage, deploying new VMs when needed and undeploying idle ones.

Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M...InfluxData

Mist is an open source multicloud management platform. Mist.io's goal is to make multicloud simple and offer a single interface from where you can manage everything. To help users make informed decisions about their infrastructure, Mist is integrating with Telegraf and InfluxDB for collecting and storing monitoring metrics. This is an evolution of a previous stack based on collectd and Graphite. This session will go over why Mist.io moved and the implementation details of its current stack. The session will also analyze the challenges faced and the solutions built. This session is a follow-up to the webinar presentation at https://www.influxdata.com/resources/how-to-gain-visibility-into-containers-vms-and-multi-cloud-environments-using-telegraf-influxdb-and-mist/ which focuses more on the technical details.

Opentelemetry - From frontend to backendSebastian Poxhofer

Distributed Tracing with OpenTracing, ZipKin and KubernetesContainer Solutions

The document discusses distributed tracing with Zipkin and Kubernetes. It defines microservices and their benefits and costs. Distributed tracing allows tracing requests across multiple services and systems. This helps identify bottlenecks and failures. The document discusses OpenTracing and Zipkin libraries for instrumenting services. It also covers Zipkin's architecture, performance considerations like sampling, and how Linkerd can help with legacy systems. Finally, it mentions answering questions about slow parts, optimizations, and failures using distributed tracing before cautioning about data volumes and interpretation.

Why Architecting for Disaster Recovery is Important for Your Time Series Data...InfluxData

Time Series data at Capital One consists of Infrastructure, Application, and Business Process Metrics. The combination of these metrics are what the internal stakeholders rely on for observability which allows them to deliver better service and uptime for their customers, so protecting this critical data with a proven and tested recovery plan is not a “nice to have” but a “must have.” In this talk, the members of IT staff, Saravanan Krisharaju, Rajeev Tomer, and Karl Daman will share how they built a fault-tolerant solution based on InfluxEnterprise and AWS that collects and stores metrics and events. They added to this, Machine Learning, which uses the collected time series to model predictions which are then brought back into InfluxDB time series database for real-time access. This Capital One team shares the journey they took to architect and build this solution as well as plan and execute on their disaster recovery plan.

Trove Updates - Liberty EditionOpenStack Foundation

Trove provides database services and improved in several areas for the Kilo and Liberty releases. Key improvements included adding new database engines like CouchDB and Vertica, improving replication functionality for MySQL and Redis, and enhancing clustering support. Testing and CI were moved to OpenStack infrastructure. For Liberty, backup/restore was added for MongoDB and Redis, and limitations on flavors per datastore were introduced. Community involvement also grew significantly over this period.

7 - Monitoring Kubernetes with ElasticKangaroot

This document introduces using Elastic Stack to monitor Kubernetes clusters managed by Rancher. It discusses the challenges of monitoring dynamic container environments and how Elastic Stack provides solutions through Beats, Logstash, Elasticsearch, and Kibana. Specifically, it recommends deploying Filebeat and Metricbeat on Kubernetes clusters using Helm or YAML, with Elasticsearch and Kibana running outside the clusters. It also provides resources for integrating Elastic in Rancher and configuring Beats to ship logs and metrics to Elasticsearch.

DIY Netflow Data Analytic with ELK Stack by CL LeeMyNOG

This document discusses using the ELK stack (Elasticsearch, Logstash, Kibana) to analyze netflow data. It describes IP ServerOne's infrastructure managing over 5000 servers across multiple data centers. Netflow data is collected and sent to Logstash for processing, then stored in Elasticsearch for querying and visualization in Kibana. Examples are given of how the data can be used, such as identifying top talkers, traffic profiling by ASN, and troubleshooting with IP conversation history. The ELK stack is concluded to be a powerful yet not difficult tool for analyzing netflow traffic.

SDN/NFV: Service Chaining Odinot Stanislas

Le SDN et NFV sont très à la mode en ce moment car en passant des appliance physiques aux équipement réseau massivement logiciel, celà devrait offrir une grande flexibilité et agilité aux entreprises (et telco en particulier). Néanmoins chainer des services réseau est un exercice encore très complexe et ce document vous explique ce qu'il est déjà possible de faire sur OpenStack en couplant par exemple : un load balancer (BigIP), un Firewall (BigIP), un réseau virtuel WAN (RiverBed) ou encore un routeur virtuel (Brocade).

More Related Content

What's hot (20)

Kubernetes meetup bangalore december 2017 - v02Kumar Gaurav

Open Tracing, to order and understand your mess. - ApiConf 2017Gianluca Arbezzano

2015 03-16-elk at-bsidesJeremy Cohoe

Operators experience and perspective on SDN with VLANs and L3 NetworksJakub Pavlik

利用K8S實現高可靠應用inwin stack

IoT Data Connector Fluent BitToru Takahashi

Time Series Tech Stack for the IoT EdgeInfluxData

THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.ioDevOpsDays Tel Aviv

Sahara Updates - Liberty EditionOpenStack Foundation

InfluxDB Community Office Hours September 2020 InfluxData

InfluxData Architecture for IoT | Noah Crowley | InfluxDataInfluxData

Glance Updates - Liberty EditionOpenStack Foundation

Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...InfluxData

OpenNebulaConf2017EU: Elastic Clusters for Data Analysis by Carlos de Alfonso...OpenNebula Project

Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M...InfluxData

Opentelemetry - From frontend to backendSebastian Poxhofer

Distributed Tracing with OpenTracing, ZipKin and KubernetesContainer Solutions

Why Architecting for Disaster Recovery is Important for Your Time Series Data...InfluxData

Trove Updates - Liberty EditionOpenStack Foundation

7 - Monitoring Kubernetes with ElasticKangaroot

Kubernetes meetup bangalore december 2017 - v02Kumar Gaurav

Open Tracing, to order and understand your mess. - ApiConf 2017Gianluca Arbezzano

2015 03-16-elk at-bsidesJeremy Cohoe

Operators experience and perspective on SDN with VLANs and L3 NetworksJakub Pavlik

利用K8S實現高可靠應用inwin stack

IoT Data Connector Fluent BitToru Takahashi

Time Series Tech Stack for the IoT EdgeInfluxData

THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.ioDevOpsDays Tel Aviv

Sahara Updates - Liberty EditionOpenStack Foundation

InfluxDB Community Office Hours September 2020 InfluxData

InfluxData Architecture for IoT | Noah Crowley | InfluxDataInfluxData

Glance Updates - Liberty EditionOpenStack Foundation

Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...InfluxData

OpenNebulaConf2017EU: Elastic Clusters for Data Analysis by Carlos de Alfonso...OpenNebula Project

Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M...InfluxData

Opentelemetry - From frontend to backendSebastian Poxhofer

Distributed Tracing with OpenTracing, ZipKin and KubernetesContainer Solutions

Why Architecting for Disaster Recovery is Important for Your Time Series Data...InfluxData

Trove Updates - Liberty EditionOpenStack Foundation

7 - Monitoring Kubernetes with ElasticKangaroot

Similar to Netflow Analysis using Elastic Stack - 조인중 (20)

DIY Netflow Data Analytic with ELK Stack by CL LeeMyNOG

SDN/NFV: Service Chaining Odinot Stanislas

Elk ruminating on logsMathew Beane

How to Implement SDN Technology in ITBSDNRG ITB

Affan Basalamah outlines a plan to implement SDN technology at Institut Teknologi Bandung (ITB) without disrupting the production network. He discusses upgrading ITB's core, datacenter, edge, access and wireless networks to support both production and experimental SDN networks. This will allow SDN research and development activities to be conducted using the campus network infrastructure. Basalamah also describes potential SDN/NFV labs, testbeds and collaboration opportunities between universities in Indonesia.

SDN and metrics from the SDOsOpen Networking Summit

This document discusses SDN and metrics from various standards development organizations (SDOs). It notes that many SDOs are developing YANG data models for network automation using model-driven APIs. There is growth in YANG models at the IETF and other SDOs like BBF, IEEE, MEF, and OpenConfig. However, coordination is needed between SDOs to reduce costs as the data models need to work together to create network services. A YANG catalog is proposed to help organize the industry and provide a common inventory of all YANG modules across SDOs and vendors.

Orion NTA Customer TrainingSolarWinds

PLNOG14: Service orchestration in provider network, Tail-f - Przemysław BorekPROIDEA

Web Services for the Internet of ThingsMarkku Laine

ELK Ruminating on Logs (Zendcon 2016)Mathew Beane

We're talking about serious log crunching and intelligence gathering with Elastic, Logstash, and Kibana. ELK is an end-to-end stack for gathering structured and unstructured data from servers. It delivers insights in real time using the Kibana dashboard giving unprecedented horizontal visibility. The visualization and search tools will make your day-to-day hunting a breeze. During this brief walkthrough of the setup, configuration, and use of the toolset, we will show you how to find the trees from the forest in today's modern cloud environments and beyond.

Data models-and-automation-jpMiya Kohno

This document discusses YANG data models and automation. It provides background on the speaker and their focus on manageability. It describes how automation is required for today's large, dynamic networks. YANG is introduced as the standard data modeling language that can be used to define management information for protocols like NETCONF and RESTCONF. Examples of organizations developing YANG models are provided. The document emphasizes that data model-driven APIs are key to enabling automation.

Network Infrastructure Monitoring @ LinkedInAshish Gite

OSDC 2016 - Unifying Logs and Metrics Data with Elastic Beats by Monica SarbuNETWAYS

The Beats are a friendly army of lightweight agents that installed on your servers capture operational data and ship it to Elasticsearch for analysis. They are open source, written in Golang, and maintained by Elastic, the company behind Elasticsearch, Logstash, and Kibana. This talk will present the first three Beats: Topbeat for system level metrics, Filebeat for log files and Packetbeat for wire data. It will also demonstrate how to combine them with Logstash and Kibana in one advanced monitoring solution, unifying log management, metrics monitoring and system stats. Finally, you will learn how to create a new Beat from scratch using Golang and the libbeat framework to capture any type of information and ship it to Elasticsearch.

Auto id-labs-kaist-research-2014Daeyoung Kim

The document describes an Internet of Things (IoT) application framework that allows for dynamic service composition on mobile devices. The framework addresses issues with traditional IoT applications by allowing services to be composed at runtime rather than requiring explicit binding during implementation. This enables interaction between devices and applications without pre-defined bindings. The framework uses a service registry to discover IoT services and a composition engine to dynamically combine services into applications on mobile devices based on user needs and context.

Load Balancing in the Cloud using Nginx & KubernetesLee Calcote

Data Ingest Self Service and Management using Nifi and KafkaDataWorks Summit

We’re feeling the growing pains of maintaining a large data platform. Last year we went from 50 to 150 unique data feeds by adding them all by hand. In this talk we will share the best practices developed to handle our 300% increase in feeds through self service. Having self-service capabilities will increase your teams velocity and decrease your time to value and insight. * Self service data feed design and ingest * configuration management * automatic debugging * light weight data governance

LinkedIn OpenFabric Project - Interop 2017Shawn Zandi

The document discusses LinkedIn's OpenFabric project which aims to simplify their data center network architecture. Some key points: - The architecture is based on the needs of LinkedIn's applications which require high intra- and inter-DC bandwidth. - It uses a simplified design with single switch hardware and software SKUs, no overlays or LAGs, and moves complexity from switches to software. - The control plane was redesigned for simplicity with a custom routing protocol instead of BGP to provide full topology visibility without configuration. - The goal is to treat the entire fabric as code and enable applications to directly interact with and control the infrastructure. Telemetry and machine learning are used for monitoring.

KubeCon USA 2017 brief Overview - from Kubernetes meetup BangaloreKrishna-Kumar

PLNOG16: Automatyzacja kreaowania usług operatorskich w separacji od rodzaju ...PROIDEA

IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, K...Kai Wähner

IoT Sensor Analytics with Apache Kafka, KSQL, TensorFlow and MQTT => Kafka-Native End-to-End IoT Data Integration and Processing. Large numbers of IoT devices lead to big data and the need for further processing and analysis. Apache Kafka is a highly scalable and distributed open source streaming platform, which can connect to MQTT and other IoT standards. Kafka ingests, stores, processes and forwards high volumes of data from thousands of IoT devices. The rapidly expanding world of stream processing can be daunting, with new concepts such as various types of time semantics, windowed aggregates, changelogs, and programming frameworks to master. KSQL is the streaming SQL engine on top of Apache Kafka which simplifies all this and make stream processing available to everyone without the need to write source code. This talk shows how to leverage Kafka and KSQL in an IoT sensor analytics scenario for predictive maintenance and integration with real time monitoring systems. A live demo shows how to embed and deploy Machine Learning models - built with frameworks like TensorFlow, DeepLearning4J or H2O - into mission-critical and scalable real time applications.

Programmability and Automation in Data Center Networks: A talk on Hot Air Bal...Joel W. King

The document discusses a talk given on programmability and automation in data center networks. It covers topics like using Ansible, Jinja templates, and Excel spreadsheets to configure and automate Cisco Nexus and ACI networks. Specific points include using Ansible to deploy ACI fabrics using XML/JSON playbooks, understanding REST APIs and markup languages, and thinking like a programmer to improve network automation. The talk emphasizes that automation tools allow network engineers to focus on applications rather than infrastructure and help adopt a DevOps mindset for networking.

DIY Netflow Data Analytic with ELK Stack by CL LeeMyNOG

SDN/NFV: Service Chaining Odinot Stanislas

Elk ruminating on logsMathew Beane

How to Implement SDN Technology in ITBSDNRG ITB

SDN and metrics from the SDOsOpen Networking Summit

Orion NTA Customer TrainingSolarWinds

PLNOG14: Service orchestration in provider network, Tail-f - Przemysław BorekPROIDEA

Web Services for the Internet of ThingsMarkku Laine

ELK Ruminating on Logs (Zendcon 2016)Mathew Beane

Data models-and-automation-jpMiya Kohno

Network Infrastructure Monitoring @ LinkedInAshish Gite

OSDC 2016 - Unifying Logs and Metrics Data with Elastic Beats by Monica SarbuNETWAYS

Auto id-labs-kaist-research-2014Daeyoung Kim

Load Balancing in the Cloud using Nginx & KubernetesLee Calcote

Data Ingest Self Service and Management using Nifi and KafkaDataWorks Summit

LinkedIn OpenFabric Project - Interop 2017Shawn Zandi

KubeCon USA 2017 brief Overview - from Kubernetes meetup BangaloreKrishna-Kumar

PLNOG16: Automatyzacja kreaowania usług operatorskich w separacji od rodzaju ...PROIDEA

IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, K...Kai Wähner

Programmability and Automation in Data Center Networks: A talk on Hot Air Bal...Joel W. King

More from NAVER D2 (20)

[211] 인공지능이 인공지능 챗봇을 만든다NAVER D2

[233] 대형 컨테이너 클러스터에서의 고가용성 Network Load Balancing: Maglev Hashing Scheduler i...NAVER D2

[215] Druid로 쉽고 빠르게 데이터 분석하기NAVER D2

[245]Papago Internals: 모델분석과 응용기술 개발NAVER D2

[236] 스트림 저장소 최적화 이야기: 아파치 드루이드로부터 얻은 교훈NAVER D2

[235]Wikipedia-scale Q&ANAVER D2

[244]로봇이 현실 세계에 대해 학습하도록 만들기NAVER D2

The document discusses challenges with using reinforcement learning for robotics. While simulations allow fast training of agents, there is often a "reality gap" when transferring learning to real robots. Other approaches like imitation learning and self-supervised learning can be safer alternatives that don't require trial-and-error. To better apply reinforcement learning, robots may need model-based approaches that learn forward models of the world, as well as techniques like active localization that allow robots to gather targeted information through interactive perception. Closing the reality gap will require finding ways to better match simulations to reality or allow robots to learn from real-world experiences.

[243] Deep Learning to help student’s Deep LearningNAVER D2

This document describes research on using deep learning to predict student performance in massive open online courses (MOOCs). It introduces GritNet, a model that takes raw student activity data as input and predicts outcomes like course graduation without feature engineering. GritNet outperforms baselines by more than 5% in predicting graduation. The document also describes how GritNet can be adapted in an unsupervised way to new courses using pseudo-labels, improving predictions in the first few weeks. Overall, GritNet is presented as the state-of-the-art for student prediction and can be transferred across courses without labels.

[234]Fast & Accurate Data Annotation Pipeline for AI applicationsNAVER D2

Old version: [233]대형 컨테이너 클러스터에서의 고가용성 Network Load BalancingNAVER D2

[226]NAVER 광고 deep click prediction: 모델링부터 서빙까지NAVER D2

This document presents a formula for calculating the loss function J(θ) in machine learning models. The formula averages the negative log likelihood of the predicted probabilities being correct over all samples S, and includes a regularization term λ that penalizes predicted embeddings being dissimilar from actual embeddings. It also defines the cosine similarity term used in the regularization.

[225]NSML: 머신러닝 플랫폼 서비스하기 & 모델 튜닝 자동화하기NAVER D2

[224]네이버 검색과 개인화NAVER D2

[216]Search Reliability Engineering (부제: 지진에도 흔들리지 않는 네이버 검색시스템)NAVER D2

[214] Ai Serving Platform: 하루 수 억 건의 인퍼런스를 처리하기 위한 고군분투기NAVER D2

[213] Fashion Visual SearchNAVER D2

The document discusses linear algebra concepts including: - Representing a system of linear equations as a matrix equation Ax = b where A is a coefficient matrix, x is a vector of unknowns, and b is a vector of constants. - Solving for the vector x that satisfies the matrix equation using linear algebra techniques such as row reduction. - Examples of matrix equations and their component vectors are shown.

[232] TensorRT를 활용한 딥러닝 Inference 최적화NAVER D2

[242]컴퓨터 비전을 이용한 실내 지도 자동 업데이트 방법: 딥러닝을 통한 POI 변화 탐지NAVER D2

[212]C3, 데이터 처리에서 서빙까지 가능한 하둡 클러스터NAVER D2

[223]기계독해 QA: 검색인가, NLP인가?NAVER D2

The document discusses machine reading comprehension (MRC) techniques for question answering (QA) systems, comparing search-based and natural language processing (NLP)-based approaches. It covers key milestones in the development of extractive QA models using NLP, from early sentence-level models to current state-of-the-art techniques like cross-attention, self-attention, and transfer learning. It notes the speed and scalability benefits of combining search and reading methods for QA.

[211] 인공지능이 인공지능 챗봇을 만든다NAVER D2

[233] 대형 컨테이너 클러스터에서의 고가용성 Network Load Balancing: Maglev Hashing Scheduler i...NAVER D2

[215] Druid로 쉽고 빠르게 데이터 분석하기NAVER D2

[245]Papago Internals: 모델분석과 응용기술 개발NAVER D2

[236] 스트림 저장소 최적화 이야기: 아파치 드루이드로부터 얻은 교훈NAVER D2

[235]Wikipedia-scale Q&ANAVER D2

[244]로봇이 현실 세계에 대해 학습하도록 만들기NAVER D2

[243] Deep Learning to help student’s Deep LearningNAVER D2

[234]Fast & Accurate Data Annotation Pipeline for AI applicationsNAVER D2

Old version: [233]대형 컨테이너 클러스터에서의 고가용성 Network Load BalancingNAVER D2

[226]NAVER 광고 deep click prediction: 모델링부터 서빙까지NAVER D2

[225]NSML: 머신러닝 플랫폼 서비스하기 & 모델 튜닝 자동화하기NAVER D2

[224]네이버 검색과 개인화NAVER D2

[216]Search Reliability Engineering (부제: 지진에도 흔들리지 않는 네이버 검색시스템)NAVER D2

[214] Ai Serving Platform: 하루 수 억 건의 인퍼런스를 처리하기 위한 고군분투기NAVER D2

[213] Fashion Visual SearchNAVER D2

[232] TensorRT를 활용한 딥러닝 Inference 최적화NAVER D2

[242]컴퓨터 비전을 이용한 실내 지도 자동 업데이트 방법: 딥러닝을 통한 POI 변화 탐지NAVER D2

[212]C3, 데이터 처리에서 서빙까지 가능한 하둡 클러스터NAVER D2

[223]기계독해 QA: 검색인가, NLP인가?NAVER D2

Recently uploaded (20)

Introduction to Zoomlion Earthmoving.pptxAS1920

Smart_Storage_Systems_Production_Engineering.pptxrushikeshnavghare94

RICS Membership-(The Royal Institution of Chartered Surveyors).pdfMohamedAbdelkader115

DSP and MV the Color image processing.pptHafizAhamed8

Lidar for Autonomous Driving, LiDAR Mapping for Driverless Cars.pptxRishavKumar530754

some basics electrical and electronics knowledgenguyentrungdo88

Degree_of_Automation.pdf for Instrumentation and industrial specialistshreyabhosale19

Development of MLR, ANN and ANFIS Models for Estimation of PCUs at Different ...Journal of Soft Computing in Civil Engineering

Passenger car unit (PCU) of a vehicle type depends on vehicular characteristics, stream characteristics, roadway characteristics, environmental factors, climate conditions and control conditions. Keeping in view various factors affecting PCU, a model was developed taking a volume to capacity ratio and percentage share of particular vehicle type as independent parameters. A microscopic traffic simulation model VISSIM has been used in present study for generating traffic flow data which some time very difficult to obtain from field survey. A comparison study was carried out with the purpose of verifying when the adaptive neuro-fuzzy inference system (ANFIS), artificial neural network (ANN) and multiple linear regression (MLR) models are appropriate for prediction of PCUs of different vehicle types. From the results observed that ANFIS model estimates were closer to the corresponding simulated PCU values compared to MLR and ANN models. It is concluded that the ANFIS model showed greater potential in predicting PCUs from v/c ratio and proportional share for all type of vehicles whereas MLR and ANN models did not perform well.

Value Stream Mapping Worskshops for Intelligent Continuous SecurityMarc Hornbeek

Explainable-Artificial-Intelligence-XAI-A-Deep-Dive (1).pptxMahaveerVPandit

Artificial Intelligence (AI) basics.pptxaditichinar

International Journal of Distributed and Parallel systems (IJDPS)samueljackson3773

The growth of Internet and other web technologies requires the development of new algorithms and architectures for parallel and distributed computing. International journal of Distributed and parallel systems is a bimonthly open access peer-reviewed journal aims to publish high quality scientific papers arising from original research and development from the international community in the areas of parallel and distributed systems. IJDPS serves as a platform for engineers and researchers to present new ideas and system technology, with an interactive and friendly, but strongly professional atmosphere.

Metal alkyne complexes.pptx in chemistrymee23nu

Structural Response of Reinforced Self-Compacting Concrete Deep Beam Using Fi...Journal of Soft Computing in Civil Engineering

Analysis of reinforced concrete deep beam is based on simplified approximate method due to the complexity of the exact analysis. The complexity is due to a number of parameters affecting its response. To evaluate some of this parameters, finite element study of the structural behavior of the reinforced self-compacting concrete deep beam was carried out using Abaqus finite element modeling tool. The model was validated against experimental data from the literature. The parametric effects of varied concrete compressive strength, vertical web reinforcement ratio and horizontal web reinforcement ratio on the beam were tested on eight (8) different specimens under four points loads. The results of the validation work showed good agreement with the experimental studies. The parametric study revealed that the concrete compressive strength most significantly influenced the specimens’ response with the average of 41.1% and 49 % increment in the diagonal cracking and ultimate load respectively due to doubling of concrete compressive strength. Although the increase in horizontal web reinforcement ratio from 0.31 % to 0.63 % lead to average of 6.24 % increment on the diagonal cracking load, it does not influence the ultimate strength and the load-deflection response of the beams. Similar variation in vertical web reinforcement ratio leads to an average of 2.4 % and 15 % increment in cracking and ultimate load respectively with no appreciable effect on the load-deflection response.

DATA-DRIVEN SHOULDER INVERSE KINEMATICS YoungBeom Kim1 , Byung-Ha Park1 , Kwa...charlesdick1345

five-year-soluhhhhhhhhhhhhhhhhhtions.pdfAdityaSharma944496

Compiler Design Unit1 PPT Phases of Compiler.pptxRushaliDeshmukh2

introduction to machine learining for beginersJoydebSheet

QA/QC Manager (Quality management Expert)rccbatchplant

Machine learning project on employee attrition detection using (2).pptxrajeswari89780

Introduction to Zoomlion Earthmoving.pptxAS1920

Smart_Storage_Systems_Production_Engineering.pptxrushikeshnavghare94

RICS Membership-(The Royal Institution of Chartered Surveyors).pdfMohamedAbdelkader115

DSP and MV the Color image processing.pptHafizAhamed8

Lidar for Autonomous Driving, LiDAR Mapping for Driverless Cars.pptxRishavKumar530754

some basics electrical and electronics knowledgenguyentrungdo88

Degree_of_Automation.pdf for Instrumentation and industrial specialistshreyabhosale19

Development of MLR, ANN and ANFIS Models for Estimation of PCUs at Different ...Journal of Soft Computing in Civil Engineering

Value Stream Mapping Worskshops for Intelligent Continuous SecurityMarc Hornbeek

Explainable-Artificial-Intelligence-XAI-A-Deep-Dive (1).pptxMahaveerVPandit

Artificial Intelligence (AI) basics.pptxaditichinar

International Journal of Distributed and Parallel systems (IJDPS)samueljackson3773

Metal alkyne complexes.pptx in chemistrymee23nu

Structural Response of Reinforced Self-Compacting Concrete Deep Beam Using Fi...Journal of Soft Computing in Civil Engineering

DATA-DRIVEN SHOULDER INVERSE KINEMATICS YoungBeom Kim1 , Byung-Ha Park1 , Kwa...charlesdick1345

five-year-soluhhhhhhhhhhhhhhhhhtions.pdfAdityaSharma944496

Compiler Design Unit1 PPT Phases of Compiler.pptxRushaliDeshmukh2

introduction to machine learining for beginersJoydebSheet

QA/QC Manager (Quality management Expert)rccbatchplant

Machine learning project on employee attrition detection using (2).pptxrajeswari89780

Netflow Analysis using Elastic Stack - 조인중

1. NetFlow Analysis with Elastic Stack Elasticsearch Seoul Meetup [email protected]

2. Presenter manager at SK broadband [email protected] I don’t have any f**king Btv coupons!

3. What is NetFlow? • Cisco router feature that provides the ability to collect IP network traffic information on interfaces • Main components – Flow Exporter – Flow collector (logstash) – Analysis application (elasticsearch) – Visualization (kibana or whatever works) Source: Wikipedia

4. NetFlow Architecture Source: Wikipedia

5. NetFlow Protocol (v5 header) Source: https://www.plixer.com/support/netflow_v5.html

6. NetFlow Protocol (v5 record) Source: https://www.plixer.com/support/netflow_v5.html

7. What we can do with NetFlow • Application usage analysis • Troubleshoot network problem • Track down bandwidth hogs • Detect abnormal traffic usage • Detect hosts infected by malwares • …

8. Elastic Stack - NetFlow • Logstash: NetFlow collector • Elasticsearch: NetFlow storage & analysis • Kibana: Visualization

9. System Architecture 이보다 더 간단할 수 없다.

10. Logstash Configuration (input)

11. Logstash Configuration (filter)  Enrichment with geolocation info  Calculate estimated traffic volume (need to consider sampling rate)

12. Logstash Configuration (output)

13. Template for NetFlow data

14. Collected data sample

15. Demo! Demo!! Demo!!!

16. Thank you.