![Sumo Logic Confidential
Scheduled Views
Copies of subsets of data, similar to a relation DB materialized view.
Use Cases
Pre-aggregated data (e.g. for long-term trends)
Find the needle in the haystack….
Best Practices
We recommend selectivity of > 1:10000
How They Work
View is updated by service ~once a minute
Allows for backfilling
Search view using _view=[viewname]
Data does count against ingest volume](https://image.slidesharecdn.com/settingsumologicsep2017-170820210544/85/Setting-Up-Sumo-Logic-Sep-2017-24-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Setting Up Sumo Logic - Sep 2017 (20)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
Ad
Setting Up Sumo Logic - Sep 2017
- 1. Sumo Logic Confidential Setting up Sumo Logic Data Collection and System Optimization Welcome! Note you are currently muted. We will get started shortly. Mario Sánchez September 2017
- 2. Sumo Logic Confidential At the completion of this webinar, you will be able to… Deploy a data collection strategy that best fits your environment Implement best practices around data collection Develop a robust naming convention for your metadata Learn to utilize optimization tools to enhance search performance
- 3. Sumo Logic Confidential Unified Logs and Metrics – Troubleshooting Demo ALERT notifies of a critical event METRICS to identify what’s going on LOGS to identify why it’s happening
- 4. Sumo Logic Confidential High-Level Data Flow
- 5. Sumo Logic Confidential Sumo Logic Data Flow Data Collection Search & Analyze Visualize & Monitor Alerts Dashboards Collectors Sources Operators Detect 1 2 3
- 6. Sumo Logic Confidential Data Collection Strategy
- 7. Sumo Logic Confidential Enterprise Logs are Everywhere Custom App Code Server / OS Virtual Databases Network Open Source Middleware Content Delivery IaaS, PaaS SaaS Security
- 8. Sumo Logic Confidential Designing Your Deployment • Sumo Logic Data Collection is infinitely flexible. • Design a Sumo Logic deployment that's right for your organization. • Installed versus Hosted Collectors.
- 9. Sumo Logic ConfidentialSumo Logic Confidential Collector and Deployment Options Collector Cloud Data Collection Centralized Data Collection Local Data Collection Collector CollectorCollector Collector Hosted Collectors Installed Collectors Best Practices on Designing Your Deployment
- 10. Sumo Logic Confidential Collector Considerations Consider having an Installed Collector on a dedicated machine if: • You are running a very high-bandwidth network with high logging levels. • You want a central collection point for many Sources. Consider having more than one Installed Collector if: • You expect the combined number of files coming into one Collector to exceed 500. • Your hardware has memory or CPU limitations. • You expect combined logging traffic for one Collector to be higher than 15,000 events per second. • Your network clusters or regions are geographically separated. • You prefer to install many Collectors, for example, one per machine to collect local files. For system requirement details, see Installed Collector Requirements.
- 11. Sumo Logic Confidential Local Data Collection The Sumo Logic Collector is installed on all target Hosts and, where possible, sends log data produced on those target Hosts directly to Sumo Logic Backend via https connection. Source Types Local Files Operating Systems, Middleware, Custom Apps, etc. Windows Events Local Windows Events Docker Logs and Stats Syslog (dedicated Collector) Network Devices, Snare, etc Script (dedicated Collector) Cloud API’s, Database Content, binary data Typical Scenarios Customers with large amounts of (similar) servers, using orchestration/automation, mostly OS and application logs - On Premise Datacenters - Cloud Instances Benefits/Drawbacks + No Hardware Requirement + Automation (Chef/Puppet/Scripting) - Outbound Internet Access Required - Resource Usage on Target
- 12. Sumo Logic Confidential Source Types Syslog Operating Systems, Middleware, Custom Applications, etc Windows Events Remote Windows Events Script Cloud API’s, Database Content, binary data Typical Scenarios Customers with mostly Windows Environments or existing logging infrastructure (syslog/logstash) - On Premise Datacenters Benefits/Drawbacks + No Outbound Internet Access + Leverage existing logging Infrastructure - Scale - Dedicated Hardware - Complexity (Failover, syslog rules) Centralized Data Collection The Sumo Logic Collector is installed on a set of dedicated machines, these collect log data from the target Hosts via various remote mechanisms and forward the data to the Sumo Logic Backend. This can be accomplished by either using Sumo Logic syslog source type or by running Syslog Servers (syslog-ng, rsyslog), write to file, and collect from there.
- 13. Sumo Logic Confidential Source Types S3 Bucket Any data written to S3 buckets (AWS Audit or other) HTTPS Lambda Scripts, Akamai, One Login, Log Appender Libraries, etc. Google / O365 Google API and O365 API Typical Scenarios Customers using Cloud Infrastructure, while it's possible to rely on Cloud Data Collection entirely, this is not typical. These source types are normally just part of the overall collection strategies Benefits/Drawbacks + No Software Installation - S3 Latency issues - Https Post Caching Need Cloud Data Collection Most Data is generated in the Cloud and by Cloud Services and is collected via Sumo Logics Cloud Integrations.
- 14. Sumo Logic Confidential Metadata Design
- 15. Sumo Logic Confidential What is Metadata? Tag Description _collector Name of the collector (defaults to hostname) _source Name of the source this data came through _sourceHost Hostname of the server (defaults to hostname) _sourceName Name and Path of the log file _sourceCategory Can be freely configured. Main metadata tag Metadata tags are associated with each log message that is collected. Values are set through collector and source configuration.
- 16. Sumo Logic ConfidentialSumo Logic Confidential Source Category Best Practices Recommended nomenclature for Source Categories Component1/Component2/Component3… From least descriptive to most descriptive * Note: Not all types of logs need to have the same amount of levels. Best Practices: Good Source Category, Bad Source Category Prod/MyApp1/Apache/Access Prod/MyApp1/Apache/Error Prod/MyApp1/CloudTrail Dev/MyApp1/Apache/Access Dev/MyApp1/Apache/Error Dev/MyApp1/CloudTrail Prod/MyApp2/Nginx/Access Prod/MyApp2/Tomcat/Access Prod/MyApp2/Tomcat/Catalina/Out Prod/MyApp2/MySQL/SlowQueries Dev/MyApp2/Nginx/Access Dev/MyApp2/Tomcat/Access Dev/MyApp2/Tomcat/Catalina/Out Dev/MyApp2/MySQL/SlowQueries
- 17. Sumo Logic ConfidentialSumo Logic Confidential Metadata: Source Category Best Practices and Benefits Simple Search Scoping _sourceCategory=Prod/MyApp1/Apache* (All Apache Logs for Prod) _sourceCategory=*/MyApp1/Apache* (All Apache Logs for all environments) Simple, Intuitive and Self-maintaining Partitions/Indexes _sourceCategory=Prod/MyApp1* _sourceCategory=Prod/MyApp2* Note: First or first and second component are used for partitioning Simple and Self-maintaining RBAC Roles _sourceCategory=Prod/MyApp1*
- 18. Sumo Logic ConfidentialSumo Logic Confidential Metadata: Source Category Best Practices Common components (and any combination of): – Environment (Prod/UAT/DEV) – Application Name – Geographic Information (East vs West datacenter, office location, etc.) – AWS Region – Business Unit Highest level components should group the data how it is most often searched together: Prod/Web/Apache/Access Dev/Web/Apache/Access Prod/DB/MySQL/Error Dev/DB/MySQL/Error Web/Apache/Access/Prod Web/Apache/Access/Dev DB/MySQL/Error/Prod DB/MySQL/Error/Dev
- 19. Sumo Logic Confidential Ingesting Metrics Host AWS Graphite-Compatible AWS ECS Webinar: Setting up Host Metrics Webinar: Setting up AWS Metrics Webinar: Setting up Graphite Metrics
- 20. Sumo Logic ConfidentialSumo Logic Confidential Sending Metrics to Sumo Logic Collector Custom Code StatsD Server Collector Server/Device/Container OS/Container Metrics Library StatsD CollectD Host Metrics Graphite Graphite 3 2 1
- 21. Sumo Logic Confidential Optimization Tools
- 22. Sumo Logic Confidential Partitions Indexes for subsets of your data. Segregate your data into smaller, logical chunks, that are mostly searched in isolation of other Partitions. Best Practices No overlap < 20 Partitions Ideally between 1% and 30% of total volume Group data that is searched together most often About Partitions Examples: _sourceCategory=Prod/MyApp1* _sourceCategory=Prod/MyApp2* or _sourceCategory=Prod/* _sourceCategory=Dev/*
- 23. Sumo Logic Confidential Field Extraction Rules Apply parse logic for a dataset at time of ingest, as opposed to at search time. Benefits Better Performance Standardized field names Simplified Searches Best Practices Build simple, specific Rules Test Parse and other operations thoroughly (use nodrop and isEmpty for testing) Limitations 50 rules/200 fields (Will be removed soon) Not all operators supported
- 24. Sumo Logic Confidential Scheduled Views Copies of subsets of data, similar to a relation DB materialized view. Use Cases Pre-aggregated data (e.g. for long-term trends) Find the needle in the haystack…. Best Practices We recommend selectivity of > 1:10000 How They Work View is updated by service ~once a minute Allows for backfilling Search view using _view=[viewname] Data does count against ingest volume
- 25. Sumo Logic Confidential Review: Search Optimization Tools What I want to do is Partition Scheduled View Field Extraction Run queries against a certain set of data Choose if the amount of data is between 1-30% Choose if the amount of data you’d like to segregate is 1% or less Choose if you want to pre-extract fields that you are searching against frequently Extract fields from logs and make available to all users ✔ Use data to identify long- term trends ✔ Segregate data by Metadata ✔ Pre-computed or aggregate data ready to query ✔ Use RBAC to deny or grant access to the data ✔ ✔
- 26. Sumo Logic Confidential In Summary, you can… Ingest any type of logs (structured and non-structured) Select a deployment option that best fits your sources Develop a robust naming convention for your metadata Take advantage of Optimization Tools Call to Action: Set up deployment option or (hybrid option) that best fits your environment Ensure you have a robust _SourceCategory naming convention At the very least, set up Field Extraction Rules for your popular data sources
- 28. Sumo Logic Confidential Technical Resources Learn Explore the tutorials Reference technical docs Attend or review training webinars Find answers or post questions to Community Open a Support case Log a Feature Request Find out What’s New
- 29. Sumo Logic Confidential Thank you!
Editor's Notes
- #4: Sumo Logic helps you gain insights into the growing pool of data within your complex environment.
- #5: Most of you are using the Sumo Logic service for at least one of the 3 following use cases: For DevOps –allows DevOps teams to monitor KPI’s to deliver quality software; less time troubleshooting and more time developing code. For IT Ops – Extract valuable information such as latencies, performance metrics, trends and any critical events tied with core systems. For Compliance and Security – Sumo Logic helps organizations simplify and automate compliance & security monitoring across their entire stack, using predictive analytics
- #6: Today I’m going to walk you through a demo which will show you how a unified Logs and Metrics solution can reduce your troubleshooting time. 1. First, we’ll use our Alerting capabilities to notify a user of a critical event 2. The Alert will direct us to a Metrics dashboard that helps us identify WHAT is going on 3. Lastly, I’ll dive into the relevant logs to Identify WHY this is happening Let’s jump right into it.
- #8: Sumo Logic Data Flow is broken into 3 main areas: Data Collection through configurable Collectors and Sources. Collectors collect, compress, cache and encrypt the data for secure transfer. Search and Analyze – Users can run searches and correlate events in real-time across the entire application stack. We will be spending most of our time in this area during this webinar, as this is most likely what you will first be doing as a new user. Visualize and Monitor- Users have the ability to create custom dashboards to help you easily monitor your data in real-time. Custom alerts notify you when specific events are identified across your stack. I will cover Data Collection at a high-level, and cover the next 2 areas through a demo.
- #10: What data can we ingest? We can ingest data from just about any source you can imagine - structured or unstructured. Here are just a few of the devices, applications and frameworks you may be using - all of which produce log data that Sumo Logic can ingest and analyze. The left hands side can present you technology stack – from custom application code all the way down to your network devices. The right can represent your infrastructure.
- #11: Sumo Logic Installed and Hosted Collectors are infinitely flexible. Design a Sumo Logic deployment that's right for your organization. <Review slide citing some examples>
- #12: Hosted Collectors Allow for seamless collection from Amazon S3 buckets and HTTP Sources. Hosted Collectors don't require installation or activation, nor do Hosted Collectors have physical requirements, since they're hosted in AWS. Because there are no performance issues to consider, you can configure as many S3 and HTTP Sources as you'd like for a single Hosted Collector. Installed Collectors Sumo Logic Installed Collectors are lightweight and efficient. You can choose to install a small number of Collectors to minimize maintenance or just because you want to keep your topology simple (Centralized). Alternatively, you can choose to install many Collectors on many machines (Local) to distribute the bandwidth impact across your network. Installed Collectors are deployed in your environment, either on a local machine, a machine in your organization, or even an Amazon Machine Image (AMI). Installed Collectors require a software download and installation. Upgrades to Collector software are released regularly. A few things to consider: Consider having an Installed Collector on a dedicated machine if: You are running a very high-bandwidth network with high logging levels. You want a central collection point for many Sources. Consider having more than one Installed Collector if: You expect the combined number of files coming into one Collector to exceed 500. Your hardware has memory or CPU limitations. You expect combined logging traffic for one Collector to be higher than 15,000 events per second. Your network clusters or regions are geographically separated. You prefer to install many Collectors, for example, one per machine to collect local files. IMPORTANT: For system requirement details, see Installed Collector Requirements.
- #13: A few things to consider: Consider having an Installed Collector on a dedicated machine if: You are running a very high-bandwidth network with high logging levels. You want a central collection point for many Sources. Consider having more than one Installed Collector if: You expect the combined number of files coming into one Collector to exceed 500. Your hardware has memory or CPU limitations. You expect combined logging traffic for one Collector to be higher than 15,000 events per second. Your network clusters or regions are geographically separated. You prefer to install many Collectors, for example, one per machine to collect local files. IMPORTANT: For system requirement details, see Installed Collector Requirements.
- #14: The Sumo Logic Collector is installed on all target Hosts and, where possible, sends log data produced on those target Hosts directly to Sumo Logic Backend via https connection.
- #15: The Sumo Logic Collector is installed on a set of dedicated machines, these collect log data from the target Hosts via various remote mechanisms and forward the data to the Sumo Logic Backend. This can be accomplished by either using Sumo Logic syslog source type or by running Syslog Servers (syslog-ng, rsyslog), write to file, and collect from there.
- #18: Great, data is ingested into the Sumo Logic service, but something else is also happening in the background. Every single message ingested gets tagged with metadata that makes it much easier to search for related messages. This table shows the 5 main tags (review them all) In particular, I want to point out the source Category metadata field, as choosing the right naming convention can make a big impact on your searching capabilities and performance.
- #22: From a metrics perspective, Sumo Logic can ingest metrics from 3 main sources: Host metrics from any of your host servers, regardless of OS. You can watch a video of this at Video Library Your AWS metrics, which we collect via CloudWatch Any Graphite-compatible metrics
- #23: Here’s how the landscape will look when you start sending Graphite-compatible metrics to Sumo. CollectD, DropWizard, Metrics Libraries can all send their metrics straight to an Installed Collector which can forward to Sumo Logic. Additionally, this Collector do double-duty and can also ingest Host Metrics from your host server. In the case of StatsD, the StatsD server can send to a Sumo Logic Collector.