Splunk - Verwandeln Sie Datensilos in Operational Intelligence

Editor's Notes

• #2: The Splunk Enterprise Technical Overview
• #3: As a company our mission is to make machine data accessible, usable and valuable to everyone. This overarching mission is what drives our company and product priorities.
• #4: Splunk is the platform for machine data, it digests all machine data and allow users to quickly analyze their data and rapidly obtain insight. The platform was designed around the premise of being able to consume any machine data even if the format changes. A relational database would cannot effectively support constantly changing underlying schemas. Splunk solves this by creating a schema on the fly… Splunk Cloud is only available in the U.S. and Canada.
• #5: It can do this because it has no back-end RDBMS. This allows users to ask new questions of their data without having to worry if their schema supports it. Splunk ingests files regardless of their format, structure, or type. It provides integrated, end-to-end capabilities to collect, store, analyze and manage machine data - real-time and historical - in a single solution. This also means you don’t have to filter the data and you can forward data from anything.
• #6: Getting data into Splunk is designed to be as flexible and easy as possible. Because the indexing engine is so flexible and doesn’t generally require configuration for most machine data, all that remains is how to collect and ship the data to your Splunk. There are many options. When Splunk is running locally as an indexer or lightweight forwarder, you have additional options and greater control. Splunk can directly monitor hundreds or thousands of local files, index them and detect changes. Additionally, many customers use our out-of-the-box scripts and tools to generate data – common examples include performance polling scripts on *nix hosts, API and more.
• #8: One of the great things about Splunk is that you don’t have to start big. Many organizations start using Splunk to solve a single problem. From there, they quickly see the value and begin to expand within the organization. Let’s take a look at how customer deployments often mature. The point I want you to see here is that it’s not unusual to start small, but make sure you plan to go big. Let’s start with Search and investigation, which is where many customers start. Using Splunk, organizations identify and resolve issues up to 70% faster and reduce costly escalations by up to 90%. Splunk is one place to find and fix problems, and investigate incidents across all your IT systems and infrastructure. In the Proactive monitoring stage we begin to monitor IT systems in real time to identify issues, problems and attacks before they impact your customers, services and revenue. Splunk keeps watch of specific patterns, trends and thresholds in your machine data so you don't have to. We trigger notifications in real-time via email or RSS, execute a script to take remedial actions. We can also send an SNMP trap to your system management console or generate a service desk ticket. From here we get into Operational visibility. We can now see the whole picture, track performance and make better decisions. We can visualize usage trends to better plan for capacity; spot SLA infractions, track how you are being measured by the business. We do all of this using your existing machine data without spending millions of dollars instrumenting your IT infrastructure. And finally we reach Real-time business insight. We can now make better-informed business decisions by understanding trends, patterns and gaining Operational Intelligence from your machine data. See the success of new online services by channel or demographic, reconcile 3rd-party service provider fees against actual use, find your heaviest users and heaviest abusers, and more. Because machine data captures every behavior, the possibilities are game changing. You'll find the lead times to get to this intelligence dramatically less than other solutions - measured in minutes/hours instead of months.
• #9: These are the five logical roles, a Splunk instance can be one or more of the roles. The search head is what most users interact with. It is the webserver and app interpreting engine that provides the primary, web-based user interface. Since most of the data interpretation happens as-needed at search time, the role of the search head is to translate user and app requests into actionable searches for it’s indexer(s) and display the results. The Splunk web UI is highly customizable, either through our own view and app system, or by embedding Splunk searches in your own web apps or our API. Additional search heads can be deployed to scale with user or search load. The core of the Splunk infrastructure is indexing. An indexer does two things – it accepts and processes new data, adding it to the index and compressing it on disk. The indexer also services search requests, looking through the data it has via it’s indices and returning the appropriate results to the searcher over a secure compressed communication channel. Indexers scale out almost limitlessly and with almost no degradation in overall performance, allowing Splunk to scale from single-instance small deployments to truly massive Big Data challenges. The Splunk forwarder is an optional component that can be installed to forward data from servers, desktops, mainframes, and even ARM based devices. There are two types of forwarders; the full Splunk distribution or a dedicated “Universal Forwarder”. The full Splunk distribution can be configured to filter data before transmitting, execute scripts locally, or run SplunkWeb. This gives you several options depending on the footprint size your endpoints can tolerate. The universal forwarder is an ultra-lightweight agent designed to collect data in the smallest possible footprint. Both flavors of forwarder come with automatic load balancing, SSL encryption and data compression, and the ability to route data to multiple Splunk instances or third party systems. The Cluster Master coordinates which indexers have copies of which buckets to ensure we have met the proper number of replication and searchable copies of each bucket. All clustered Indexers check in with the Master to alert them of their status, and the status of each of their replicated indexes and buckets. We will talk more about buckets later. And at the bottom there is the there is the Deployment Server, which can be used to manage your distributed Splunk environment. Deployment server helps you synchronize the configuration of your search heads during distributed searching, as well as your forwarders to centrally manage your distributed data collection. Of course, Splunk has a simple flat-file configuration system, so feel free to use your own config management tools if your more comfortable with what you already have.
• #10: A sourcetype is simply a default field that identifies the source type of the event -- the data format type from which the event originates, such as access_combined or cisco_syslog. It's important that you assign the right source type to your data. That way, the indexed version of the data will look the way you expect it to, with appropriate timestamps and event breaks. This will make it a lot easier to search your data later on. For the most part, it's pretty easy to assign the right source type to your data. Splunk Enterprise comes with a large number of predefined source types. When consuming data, Splunk Enterprise will usually select the correct source type automatically. Sometimes, though, Splunk Enterprise needs your help. In such a case, the data preview capability, as shown in the images here, can be used to set or configure a source type.
• #11: To build an index on the data, Splunk Enterprise will first take the raw data, compress it, and put it in “buckets”. It will also perform a number of actions, including: Configuring character set encoding. Identifying line termination using linebreaking rules. While many events are short and only take up a line or two, others can be long. Identifying timestamps or creating them if they don't exist. At the same time that it processes timestamps, Splunk identifies event boundaries. Extracting a set of default fields for each event, including host, source, and sourcetype.
• #12: Allows you to search all your data in one place in real time. The search interface operates very similar to doing a search on any web search engine. Any user can become powerful very quickly with preexisting knowledge of using a search engine; however, Splunk has created over 100 commands (135 published) to make analyzing the data quicker and easier. <This is a great time to start a demo and show the search language if giving a demo> The schema on the fly approach is a key differentiator with Splunk.
• #13: Applying a schema at the last possible moment allows for the greatest flexibility when asking questions of your data. Splunk enterprise will atomically extract the values from the fields in events. If the data source is updated and new fields are added or the format of the events change Splunk does not need to re-index the data. Sometimes the raw event doesn’t contain useful enough information and it needs to be enriched.
• #14: The data for example may have a userid but you want to search on a name. Splunk’s lookup capability can enrich the raw data by adding additional fields at search time by. Some common use cases including event and error code description fields. Think “Page not Found” instead of “404”. Enriching your data can lead to entirely new insight. In the example shown, Splunk took the userid and looked up the name and role of the user from an HR database. Similarly, it determined the location of the failed log in attempt by correlating the IP address. Even though these fields don’t exist in the raw data, Splunk allows you to search or pivot on them at any time. You can also mask data. For example, you may want social security numbers to be replaced with all X’s for regular users but not masked for others. Removing data can also be useful, such as filtering PII, before writing it to an index in Splunk.
• #15: When running a search, the search head will fetch the events from disk which could be from multiple indexers, then it will sort and summarize the events, and format into the final results as requested before displaying it to the user. Splunk Enterprise search is incredibly powerful because it….