90% of Enterprises are Using DataOps. Why Aren’t You?

Editor's Notes

• #4: The circle marker can be dragged up or down the vertical bar to designate position in the Agenda, and bolding should be changed accordingly. Make sure circle marker is centered in front of bolded Agenda text item.
• #6: 451 Research’s DataOps Survey was conducted in October 2018 with 150 representatives of North American organizations with more than 1,000 employees and a minimum of 2PBs of data under management who had a solid understanding of their organization’s data management strategy.  We deliberately targeted companies that are already engaged with DataOps. 21% declared themselves Optimized (with DataOps ingrained in company culture), while 37% saw themselves as Accelerated (with DataOps technologies and initiatives across multiple departments), and 34% Emerging (in the process of defining a DataOps strategy). Only 3% considered their level of DataOps maturity to be Nascent (piloting DataOps technologies and processes), while 5% admitted to being of Low Maturity (with no DataOps strategy in place). 
• #7: Almost all respondents – 92% – agree to some extent with the statement that improved DataOps would have a positive impact on their organization’s success. 
• #8:  89% of respondents expect to significantly or moderately change their processes, organizational structure or training/consulting to improve DataOps in the next 12 months. 
• #9:  86% of respondents plan to increase spending, investment or development effort on DataOps technologies in the next 12 months.
• #10: DataOps is about reducing the complexity involved in data provisioning and enabling self-service access to data in order to accelerate the development of data-driven applications and data-driven decision-making, as well as supporting business agility in response to rapidly changing business requirements  A key element of this is reducing ‘data friction,’ which arises when the demands of data consumers (such as data analysts, developers and senior decision-makers) are not met by data operators (e.g., data management and IT professionals).  As an umbrella term, DataOps can refer to a range of related technologies and processes, and is as much about cultural and organizational change as it is about the adoption of emerging data management products and services.  For the purposes of this paper and its associated survey, DataOps is defined as: 
• #12: 451 Research believes that an agile approach to data management is critical if enterprises are to realize the benefits of investing in data processing and analytics technologies to develop new applications that deliver improvements in operational efficiencies and competitive advantage to drive transformational change. Being data-driven is a goal for most enterprises, but is more difficult to achieve in practice given the need to balance the requirements of various users (data analysts, data scientists, developers, IT operations and business executives for example) and sometimes competing requirements (such as agility, security, privacy and compliance  One of the most significant challenges that organizations face as they seek to become more data-driven relates to the data itself: how to manage and govern it in an efficient manner to deliver the desired benefits. This is an age-old challenge, but one that is made more complicated by the increasing pace and volume at which data is generated. 
• #13: Our DataOps Survey results indicate that data management is not a single challenge, per se, but encapsulates a range of challenges (see Figure 2). The most significant of these, according to our survey respondents, is security, which was cited by 68% of respondents as a data management challenge, followed by data quality (60%) and the management/governance of distributed data (51%). Individually, each of these factors is a significant challenge. Data security is a perennial concern, of course, while addressing data quality is essential if organizations are to ensure they are making decision based on accurate data.  However, these factors do not occur in isolation. Survey respondents selected, on average, 4.7 different options as the primary data management challenges faced by their organizations. It is the confluence of this range of factors – including not just data security, quality and distributed management, but also data access, data provisioning, data privacy and data sprawl, along with reliability and data format complexity – that makes data management so challenging. 
• #14: One of the primary trends shaping the data processing and analytics market in recent years has been the growth of data. This has driven the adoption of emerging data platforms and analytics techniques, and it has also been accelerated by the use of more economical data storage and processing products and services – which has enabled enterprises to store and process data that was previously ignored.  Our DataOps Survey results illustrate the continuing challenge in relation to the growth of data, the growing number of data sources, and a growing range of data processing locations (see Figure 3). In terms of data growth: 29% of respondents stated that their organization’s data is growing at between 100-500GB per day, with 19% reporting growth between 500GB-1TB per day, 12% between 1TB-2TB per day, and 13% reporting data growth in excess of 2TB per day.  While traditionally enterprises had to deal with multiple relational database providers, perhaps with a few specialist nonrelational databases, the growth of NoSQL and Apache Hadoop means that many enterprises are now having to deal with a plethora of data platforms  In addition to the growing number of data platforms, the growing range of data processing locations is also a challenge, and one that is likely to grow as a greater volume of data processing and storage occurs in cloud computing platforms. 
• #15: While storing, processing and analyzing greater volumes of data brings potential advantages in terms of greater business intelligence, more data also means more data management challenges.  This is particularly true in relation to provisioning and managing data management environments: while agile development and DevOps practices mean that enterprises are able to develop and provision new application environments at a much faster rate than ever before, the provisioning of data to support those applications remains a bottleneck to operational efficiencies.  Two-thirds of survey respondents currently have four or more individuals involved in the data provisioning process, and those with the greater data growth require more participants: 42% of respondents with data growing at more than 1TB per day have more than five people involved in the data provisioning process, compared with 26% of all respondents. 
• #16: Additionally, 78% of survey respondents indicated that it takes their organization four days or more to provision a new data environment, and those with the greatest data growth face greater delays: 21% of respondents with data growing more than 1TB per day take more than a week to provision a new data environment, compared to 14% of all respondents 
• #17: Complexity in the provisioning of data isn’t just a data management headache: it also has significant implications in terms of business efficiency and productivity, as well as data security. According to the survey results, 65% of respondents agree (37%) or strongly agree (28%) that lack of access to the right data negatively impacts organizational efficiency, productivity and/or speed. 
• #18: In particular this can lead to challenges with regard to sharing data – either internally with employees, or externally with customers and partners – and 68% of respondents agree (39%) or strongly agree (29%) that securing data shared with internal and external users is a concern within their organization. 
• #20: While DataOps is closely associated with agility and operational efficiency, the survey results also indicate that DataOps has the potential to help alleviate security and compliance as cross-functional concerns related to data management. In fact, concern about security and compliance is actually the number one driver for focusing on DataOps, cited by 65% of respondents.  While this is in keeping with security being the number one data management challenge (see ‘Impact of DataOps’ above), it is still interesting that security and compliance concerns score higher than improved operational efficiencies (63%), cost reduction/ROI (55%), the introduction of innovative, application-driven business capabilities (52%), and the need for greater business agility and speed (51%).  The fact that all of these responses were cited by more than half of survey respondents illustrates that DataOps is a multifaceted initiative that is by no means solely focused on agility. The 
• #21: The list of priorities in terms of DataOps culture and processes tells a similar story. Again, security topped the list, specifically more secure and compliant processes (61%), while the smooth introduction of new tools and methodologies was cited by 60% of respondents, followed by greater collaboration and alignment across functions (57%), as well as organizational responsiveness and support for development (also 57%). 
• #22: In relation to perceived business benefits of DataOps, security is again top of mind with two-thirds of respondents citing greater security and compliance, followed by increased business agility and faster time to market (63%), as well as greater business insight and improved decision-making (55%), and the introduction of innovative, application-driven business capabilities (54%).
• #23: respondents perceive that there are a range of business activities that could benefit from DataOps.  These include cloud migration (cited by 65% of respondents), business intelligence/analytics (63%), and backup and recovery (63%), as well as compliance and risk management (59%), development and testing (59%), and also artificial intelligence and machine learning (63%). 
• #32: Organizations ‘Optimized’ for DataOps Are more advanced in terms of digital transformation: 94% are executing digital transformation (versus 67% of all respondents)