SplunkLive! What's New in Splunk 6 Session

Editor's Notes

• #4: Splunk now has more than 850 employees worldwide, with headquarters in San Francisco and 14 offices around the world.Since first shipping its software in 2006, Splunk now has over 6,000 customers in 90+ countries. These organizations are using Splunk software to improve service levels, reduce operations costs, mitigate security risks, enable compliance, enhance DevOps collaboration and create new product and service offerings. Please always refer to latest company data found here: http://www.splunk.com/company.
• #6: Data is growing and embodies new characteristics not found in traditional structured data:Volume, Velocity, Variety, Variability/Veracity.Machine data is one of the fastest, growing, most complex and most valuable segments of big data.All the webservers, applications, network devices – all of the technology infrastructure running an enterprise or organization – generates massive streams of data, in an array of unpredictable formats that are difficult to process and analyze by traditional methods or in a timely manner. Why is this “machine data” valuable? Because it contains a trace - a categorical record - of user behavior, cyber-security risks, application behavior, service levels, fraudulent activity and customer experience.
• #7: Unlike traditional structured data or multi-dimensional data– for example data stored in a traditional relational database for batch reporting – machine data is non-standard, highly diverse, dynamic and high volume. You will notice that machine data events are also typically time-stamped – it is time-series data. Take the example of purchasing a product on your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data generated by the different systems supporting these different interactions. Each of the underlying systems can generate millions of machine data events daily. Here we see small excerpts from just some of them.
• #8: When we look more closely at the data we see that it contains valuable information – customer id, order id, time waiting on hold, twitter id … what was tweeted. What’s important is first of all the ability to actually see across all these disparate data sources, but then to correlate related events across disparate sources, to deliver meaningful insight.
• #9: If you can correlate and visualize related events across these disparate sources, you can build a picture of activity, behavior and experience. And what if you can do all of this in real-time? You can respond more quickly to events that matter. For example, if an organizations captured the customers twitter ID in their customer profile this correlation would be possible. Where that didn’t exist, they could at least group by demographic with the tweets.You can extrapolate this example to a wide range of use cases – security and fraud, transaction monitoring and analysis, web analytics, IT operations and so on.
• #10: At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. Andthis overarching mission is what drives our company and product priorities.
• #13: Here's how using Splunk and your machine data can drive significant benefits for your organization.Search and investigation. Using Splunk, organizations identify and resolve issues up to 70% faster and reduce costly escalations by up to 90%. Splunk is one place to find and fix problems, and investigate incidents across all your IT systems and infrastructure.Proactive monitoring.Monitor IT systems in real time to identify issues, problems and attacks before they impact your customers, services and revenue. Splunk keeps watch of specific patterns, trends and thresholds in your machine data so you don't have to. Trigger notifications in real-time via email or RSS, execute a script to take remedial actions, send an SNMP trap to your system management console or generate a service desk ticket.Operational visibility.See the whole picture, track performance and make better decisions. Visualize usage trends to better plan for capacity; spot SLA infractions, track how you are being measured by the business. Do all of this using your existing machine data without spending millions of dollars instrumenting your IT infrastructure.Real-time business insight. Make better-informed business decisions by understanding trends, patterns and gaining Operational Intelligence from your machine data. See the success of new online services by channel or demographic, reconcile 3rd-party service provider fees against actual use, find your heaviest users and heaviest abusers, and more. Because machine data captures every behavior, the possibilities are game changing. You'll find the lead times to get to this intelligence dramatically less than other solutions - measured in minutes/hours instead of months.
• #14: Both IT and business professionals can analyze machine data to get real-time visibility and operational intelligence.With our data engine and our customers' machine data, organizations can meaningfully improve their performance in a wide range of areas e.g. meet service levels, reduce costs, mitigate security risks, maintain compliance and gain insights.
• #16: Splunk Enterprise is the industry leading software for machine data analytics and has been driving innovation and setting the standard for Operational Intelligence since 2006.In the beginning, we were first to introduce the paradigm of ‘search’ to IT – to troubleshoot IT operations and application management issues muchfaster than ever before and to find the proverbial “needle in the haystack”. When asking customers, they often referred to it as “google for the datacenter”.As the product evolved, Splunk 4 - the engine for machine data - introduced enterprise-class features – dashboards and apps, real-time search and alerts, universal collection and indexing, enterprise controls and map-reduce for horizontal scalability on commodity servers.And then in 2012 we introduced Splunk 5 – this release represented the evolution of Splunk as an Enterprise Platform for Operational Intelligence. It introduced breakthrough innovations and platform features that included:  A new reporting architecture and transparent summarization technology delivering dramatically faster reportsA new high availability architecture delivering enterprise-class scale and resilience, even while scaling on commodity servers and storageA robust developer API and SDKs available in mainstream programming languages to enable enterprise developers to leverage Splunk softwareBig data ecosystem integrations that includedSplunk Hadoop Connect, Splunk DB Connect and the Splunk App for HadoopOps
• #17: So what next? What do organizations need?They need operational intelligence from their data to be accessible, available and usable to a wider group of users in their organizations.
• #18: Organizations want to drive value from their data across the enterprise. Using Splunk software, organizations have successfully deployed multiple use cases – using their data to run IT better, to secure and audit IT and to drive deeper insights for IT and the business. As the value from the data is recognized and realized, the potential use of operational intelligence in the organization grows and therefore the opportunity exists to drive faster, more powerful analytics across a wider user base, deliver improved ease of use and enable better systems manageability.Let’s look at each of these in turn:Deliver Operational Intelligence that anyone can use – Operational Intelligence is about harnessing key insights from the machine data generated by the technology and devices that organizations rely on. And as organizations understand their data better, many have realized that the best business data is found in machine data.Today this data is highly under-leveraged and they therefore need a data analytics platform that more users can get value from – for example, users outside of core IT, such as data analysts and other lines of business users. Give individual users faster access to their relevant data and content – Provide users with immediate access to data and content relevant to their roles and needs.Simplify management of enterprise Splunk deployments – As the use of Splunk software proliferates across the enterprise, administering Splunk should be made simpler and more visual, improving manageability for the Splunk administrator.Accelerate the development of enterprise apps using Splunk – Every business is unique and so are the solutions they require. Organizations need a solution that allows them to accelerate the development process – remove barriers like custom languages for one platform only. This sets the developer free and enables them to focus on creating solutions for the business – not learn new skills.These are all important requirements. Requirements we have met head on in the latest version of Splunk software.
• #20: Splunk 6 takes large-scalemachine data analytics to the next level by introducing three breakthrough innovations:Pivot – opens up the power of Splunk search to non-technical users with an easy-to-use drag and drop interface to explore, manipulate and visualize data Data Model – defines meaningful relationships in underlying machine data and making the data more useful to broader base of non-technical usersAnalytics Store – patent pending technology that accelerates data models by delivering extremely high performance data retrieval for analytical operations, up to 1000x faster than Splunk 5Let’s dig into each of these new features in more detail.
• #21: Many organizations have realized that there is valuable business data to be found in machine data: machine data holds a categorical record of activity and behavior – of customers, users, transactions, applications, devices, sensors, machines. Splunk software enables organizations to gain new insights from this data and a key focus for Splunk 6 is to empower a broader base of users in the organization with this insight – users that extend beyond core IT users.The new Pivot interface enables non-technical and technical users alike to quickly generate sophisticated charts, visualizations and dashboards using simple drag and drop and without learning the Search Processing Language (SPL). Users can access different chart types from the Splunk toolbox to easily visualize their data different ways. The Pivot interface opens up the power of operational intelligence to a broader base of non-technical users. It enables them to rapidly and interactively manipulate, analyze and visualize data, without requiring them to understand the format and semantics of the underlying raw machine data. Queries using the Pivot interface are powered by underlying data models, which are usually designed and implemented by users who understand the format and semantics of their indexed data, and who are familiar with the Splunk Search Processing Language (SPL). Data models are covered in the next slide.Unlike traditional BI visualization tools focused on structured data analytics, the new Pivot interface enables both non-technical and technical users to easily explore, manipulate and visualize machine data. It complements existing BI technologies by providing relevant business insights from a rapidly exploding new class of data. Data from Splunk software can also be leveraged directly using the Splunk API and SDKs and integrated into existing business analytics tools.
• #22: Data models power the new Pivot interface by defining an abstract model of the underlying machine data and meaningfulrelationships in that data. Data Models are created using the Data Model Builder and are usually designed and implemented by users who understand the format and semantics of their indexed data, and who are familiar with the Splunk Search Processing Language (SPL). They define meaningful relationships and can even include structured data from external relational databases integrated via Splunk DB Connect.Once defined, data models are used and accessed via the Pivot interface to rapidly create visualizations and dashboards without needing to understand the underlying raw data. Data models can also be accelerated by leveraging the Analytics Store – an optimized store for analytical operations covered on the next slide.Unlike data models in the traditional structured world, Splunk Data Models focus on machine data and data mashups between machine data and structured data. Splunk software is founded on the ability to flexibly search and analyze highly diverse machine data employing late-binding or search-time techniques for schematization (“schema-on-the-fly”). And Data Models are no exception. They define relationships in the underlying data, while leaving the raw machine data intact, and map these relationships at search time. They are therefore highly flexible and designed to enable users to rapidly iterate. Security is also a key consideration and data models are fully permissionable in Splunk 6.
• #23: Data Models are accelerated using the High Performance Analytics Store, new in Splunk 6. The High Performance Analytics Store represents a breakthrough innovation from Splunk that dramatically accelerates analytical operations across massive data sets by up to 1000x over Splunk 5.The Analytics Store contain a separate store of pre-extracted values derived from the underlying Splunk index. This data is organized in columns for rapid retrieval and powers dramatic improvements in the performance of analytical operations. Once created the Analytics Store is used seamlessly by Data Models and in turn the Pivot interface.For users more comfortable with the Splunk Search Processing Language (SPL), The Analytics Store can also be used directly in the search language.The Splunk Analytics Store is different from traditional Columnar databases – it is based on the Splunk lexicon and optimized for data retrieval (versus updates) by the Splunk Data Model or directly from the Splunk Search Processing Language. With the Analytics Store, Splunk Enterprise now uniquely optimizes data retrieval for both rare term searches and now analytical operations all in the same software platform.The new Pivot interface, combined with Data Models and the Analytics Store makes it dramatically easier for non-technical users and technical users alike to analyze and visualize data in Splunk and represent an important step towards Splunk’s mission of making machine data accessible, usable and valuable to everyone.
• #25: How does theAnalytics Store, Data Model and Pivot benefit users across the enterprise?Lets start with the IT Professional – this includes the Splunk Administrator or an advanced Splunk user that is familiar with SPL.Using Splunk 6 they can:Create data modelsShare data models with other users – delivering a consistent view of the dataAccelerate data models using the Analytics StoreCreate reports using Pivot (although being power users, they may prefer using SPL directly!)Next we have the enterprise developer.Using Splunk 6 they can:Leverage data models built by IT, making searches more portable (using common Data Models ensures predictability of results)Leverage the Pivot interface in custom enterprise appsFinally, there are additional users that can now benefit – for example, the business or data analyst. Using Splunk 6 they can:Create reports, dashboards, charts and other visualizations using the Pivot interface and based on data models that provide an abstracted view of the raw data. Splunk 6 is not meant to replace existing BI and Business Analytics tools, but it does provide new visibility, insights and intelligence from operational data that can be used by business analysts to augment these tools. Data from Splunk software can also be leveraged directly using the Splunk API and SDKs and integrated into existing business analytics tools. For example, the recently announced Pentaho Business Analytics for Splunk® Enterprise (http://apps.splunk.com/app/1554), enables business users to utilize Pentaho to rapidly visualize and gain additional insights from Splunk’s machine data platform using existing in-house skills.
• #26: We have already covered the Analytics Store, Data Model and Pivot – all significant new features in Splunk 6. But that’s not all – with this release we are also introducing Maps and Predictive Analytics.MapsNative Splunk Maps enables users to display geographic data and summaries on maps directly within Splunk software, without relying on an external mapping object such as Google MapsBenefits of native mapping functionality include not requiring an internet connection in order to render location information, configurability via Simple XML and integration to PDF printing instead of requiring complex Advanced XML, and integrated GeoIP translationPredictive AnalysisThe Predict command uses historical data as a baseline to forecast future patterns, thresholds and tolerancesBenefits include the ability to identify the future needs of critical system resources, no prior knowledge in predictive modeling algorithms required to use this functionality, and the ability to easily interpret and customize the resultsExamples of using predictive analysis with data in Splunk software include:Determining future hardware requirements for virtual environments and forecasting energy consumptionEnhancing root cause analysis to detect abnormal patterns in events and prevent security attacksEnhancing monitoring of key components to detect system failures and prevent outages before they occur
• #28: A core focus for Splunk 6 is to make analytics in Splunk software faster, easier and more accessible to a broader base of users beyond core IT. But we didn’t stop there. In addition to powerful new analytics features, Splunk 6 also delivers: A more intuitive user experience – all-new user experience that delivers instant access to apps and content tailored to the individual userSimplified management– new visual management capabilities to simplify management of enterprise Splunk deploymentsRich developer environment – new web framework that empowers developers to integrate Splunk capabilities into enterprise applications using standard programming languagesLet’s dive into each of these in more detail.
• #29: When first using Splunk 6, the very first thing users will notice is a new home screen. The new home screen is designed to deliver a cleaner, more useful and meaningful experience that enables rapid navigation to data and content relevant to them.Before Splunk 6, users had to find their Splunk App, wait for it to load and then scroll the menus to find a specific dashboard. Now in Splunk 6, users just need to select the menu item from the new home screen and in seconds have access to the data and Apps relevant to them.The new home screen:Exposes functionality – especially helpful for new or novice usersIs easier to navigate to the most common tasksAllows users to search any Splunk App from the landing pageThe home screen is also easily to customize, so that any user now has immediate access to the data relevant to them and the information they need.
• #30: Splunk 6 delivers an altogether more intuitive and fluid, interactive search experience that both seasoned and new Splunk users will love.The Enhanced Search Experience in Splunk 6 delivers the ability to search and report from a single interface, without jumping across different screens. Prior to Splunk 6, users had to navigate across multiple screens to create a search, save the report, view the report and then, if required, edit the report. Now they can create, save, view and edit reports through a single interface.The new user experience also brings together a collection of UI enhancements designed to improve the usability of the search interface and to enable simpler report authoring and editing. This includes a new field exploration experience and new navigation to build visualizations, tables, searches and dashboard panels from a single UI.Existing Splunk Apps will work with Splunk 6 and can also be modified to fully leverage the new Splunk 6 functionality to package prebuilt reports.
• #31: Splunk 5 introduced Index Replication to provide data resiliency while using commodity servers and storage. In Splunk 6, we’ve made administering mission-critical Splunk deployments easier with the new Simplified Cluster Management.This provides a centralized dashboard giving IT admins a global view of their critical Splunk infrastructure, including:What is the replication status?What is the health of the data?Is the data searchable?Is the data available?Simplified Cluster Management delivers quicker recovery from failures using new performance optimizations and easier management of Splunk Apps in a distributed clustered environment – no longer does a Splunk administrator need to visit every clusternode to deploy an app.With Splunk 6, it’s now easier to protect your organization’s valuable machine data.
• #32: Along with Simplified Cluster Management, Forwarder Management supports the increasing number of customers deploying and managing very large Splunk installations spanning thousands of servers and multiple geographies.  As Splunk deployments become more business critical, ensuring systems health and performance becomes a key priority. Prior to Splunk 6, configuring forwarders was complex and required updating deployment server config files. The new Splunk 6 Forwarder Management provides:A new visual management interface to deploy and monitor thousands of configurationsThe ability to track the status of rollouts and track down errorsAllows management of Splunk Forwarder configuration via the UI
• #33: With Splunk 5 we released powerful new developer features, including the REST API and SDKs in Java, JavaScript and Python. This enabled developers to develop custom enterprise applications that leveraged Splunk software.Splunk 6 introduces two new features that empower enterprise developers further in rapidly building innovative apps. The first is the Enhanced Dashboard Editor that provides the ability to add custom styling, behavior and visualizations without the need for write Advanced XML. Prior to Splunk 6, users had to write Advanced XML to manipulate dashboards, such as adding a Time Range Picker, additional panels, etc. Now this can all be driven visually though the new enhanced Dashboard Editor.Developers can also easily “escape” to HTML for more powerful customizations and integration that leverages the new Splunk Web Framework (next slide).
• #34: Splunk 6 enables developers looking for more advanced functionality and capabilities to now build Splunk Apps using popular, standards-based web technologies, such as JavaScript and Django.Building a Splunk App now looks and feels like building any modern web application, making Splunk development accessible to millions of professional developers around the world. The new Splunk Web Framework enables developers to quickly create Splunk Apps by using the modern web programming paradigm including pre-built components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk App using Simple XML,JavaScript or Django (or any combination thereof).
• #35: Splunk 6 is our most advanced version of Splunk software ever – the industry-leading machine data platform.Powerful Analytics:Splunk Enterprise 6 takes large-scale machine data analytics to the next level by introducing three breakthrough innovations:Pivot – opens up the power of analytics to non-technical users with an easy-to-use drag and drop interface to explore, manipulate and visualize data Data Model – defines meaningful relationships in underlying machine data and makes this data more useful to a broader base of users, in particular non-technical usersAnalytics Store – patent-pending technology that accelerates data models by delivering extremely high performance data retrieval for analytical processing, up to 1000x faster than Splunk Enterprise 5 The new Pivot interface, combined with Data Models and Analytics Store makes it dramatically easier for non-technical users and technical users alike to analyze and visualize data in Splunk. Now more users than ever are empowered by Splunk software to get insights from their machine data.  Intuitive User Experience:Splunk Enterprise 6 includes powerful productivity features for users with a more intuitive user experience: The new Home Experience – gives users instant access to the data, apps and content they care about The Enhanced Search Experience – brings search and reporting together– so users can author rich – dynamic reports - build visualizations – tables – and custom searches – faster than ever beforeSimplified ManagementWe’ve made Splunk Enterprise 6 easier to deploy, configure and manage – even as customers expand their Splunk Enterprise deployments to the multi-terabyte scaleSimplified Cluster Management – deliver easier management of mission-critical Splunk software deployments providing everything the Splunk admin needs to monitor high availability on a centralized dashboardForwarder Management – support big data scale with easy configuration and management of thousands of forwarders across multiple geographies Rich Developer EnvironmentAnd now Splunk Enterprise 6 provides a more powerful developer environment with the integrated Web Framework. Developers can build custom Splunk Apps, customize dashboards, or add advanced functionality - using standard web technologies, such as JavaScript and Django.Splunk 6 represents a significant milestone in our mission to make machine data accessible, usable and valuable by everyone. Find out more at www.splunk.com/6