Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud

Editor's Notes

• #2: Thanks John Good morning everyone, thanks for choosing to listen to our presentation. As John said, I am the business development director for IBT, or other words, SALES, so it’s a sobering thought then that I am probably the least intelligent person in the room. So hopefully you’ll cut me some slack when we get to the techie part. It’s ok, I’m a father of 4, so I’m used to being told it on a daily basis.
• #3: No doubt you’ve heard this before but I find it incredible to know that my 7 year old has already been exposed to as much information, if not more, than my grand parents had throughout the whole of their lives. In fact more than 90% of all the data there has ever been has been produced in the last few years. But only 0.5% of it is ever analysed. And the value of that data, so I’m told, decreases by almost 90% within the first 5 minutes!
• #4: So the stakes are high and it’s vitally important to manage that data as effectively as possible. The need is real. Companies that can’t keep pace will no longer be able to compete. They will not see or anticipate the disruption in their industry. They MUST respond rapidly, they MUST make fact-based decisions, and they MUST leverage their data assets to remain viable. The stakes are too high not to.
• #5: So it’s no wonder that CIO’s rank analytics and cloud adoption as their 2 foremost priorities. Now this survey was taken back in 2015 but it still rings true today, particularly in Australia.
• #6: But more than that, the trend for self service and automation is turning IT into BT – and I’m not talking about British Telecom, for those of you who’ve been to the UK, for that would be truly awful. Business Technology, meaning business users want access to data not just the IT departments. They want to agility and veracity to mine data for themselves. Find links, try things, fail and try again.
• #7: Because for the first time real value is being extracted from it. The author of this report had been surveying Fortune 1000 companies since 2012 and for the first time nearly 50% reported achieving measurable results from their big data investments, with 80.7% of executives characterizing their big data investments as “successful.” And this is why we are here today. To talk about how we can help you achieve similar results. Using the power of todays technology - Hadoop and Attunity and others – so that your business can keep ahead of its competition.
• #8: So who is IBT? Well we’re Attunity’s authorised partner for Australia. We sell, implement and support all Attunity products. So we are the face you see and the people you deal with for support and assistance.
• #9: But we are here to talk about Attunity. The No1 independent provider of data integration in the market. With 2,500 customers globally, 44 of the top Fortune 100 included, recognised as a challenger on Gartner’s Magic Quadrant, and a partner of choice for most of major software vendors, data integration and data management is their business … and only their business.
• #10: Flagship products are Attunity Replicate, the product we’ll be mostly talking about today, moves you data from almost any source to almost any target, in fact the largest supported end points in the market, without any need to code. As it is so simple to set up, you see an almost immediate return on investment. Attunity Compose – enables you to automate the process of creating data warehouses, data marts, Hadoop and now HIVE data lakes with no manual ETL coding, saving time, ensuring compliance and lessening the burden on your budgets. Attunity Visibility – provides insight how data is used by the business and impacting your systems - enabling you to make intelligent and informed data management decisions, saving some customers millions of dollars
• #11: These are some examples of what customers are using the technology for. As we are here to talk about SAP and Hadoop, the two that jump out at me are : The international food processor that needed to merge data from multiple sources, including SAP, into a centralized data lake for analytics. To improve efficiency and reduce costs. Their challenge was to decode that data from the complex underlying structures. They chose Attunity Replicate to continuously load data from the first sales order ERP module into a Hadoop data lake. Business analysts were then able to select and add custom Z tables, including indexed columns. Attunity Replicate accelerated data integration — saving significant time and labor costs. The global telco (Verizon) found getting real time data for financial reporting time consuming & labour intensive. Using a multi-tier process to pull data from 2 SAP and 1 PeopleSoft system and integrating that into its Hadoop data lake was extremely challenging. They had to wait a whole day to see data changes which was not sustainable. With Replicate, all divisional and corporate financial reporting teams now have access to ERP data in one place. As data changes, it is automatically uploaded to the data lake in less than an hour rather than overnight so that the most up-to-date data is available for daily reports. The solution also eliminated the middle-tier database, reducing maintenance time and cost, and easing the strain on valuable developers. They now gain new insights, and make more strategic decisions for different parts of the business, accelerating key business initiatives.
• #12: And here in Australia, we are little behind the curve when it comes to Hadoop. But it is finding its way into our customer sites. The Healthcare company needed a real time CDC solution for its customer transformation program and to build its data lake in SQL Server in Azure, and simultaneously to Event Hubs for streaming. And the state owned Energy business needed to replicate and capture changes from its source systems into Vertica. Speed was essential as some tables were huge, 4.3 billion rows (that’s since grown to over 6bn) – was taking their incumbent solution 3 days. Replicate did it in 5 hours. So the idea being we are streaming real time data so that you can analyse it downstream, NOW, not tomorrow.
• #13: So with this in mind, to stay competitive, Enterprises must be prepared to: Analyze data from 100s if not 1000s of sources of data, Analyze data anywhere — on-premises or in the cloud from databases, data warehouses, Hadoop systems, in-memory structures, etc., and Analyze data in real-time to capture new and changing data and process it as a stream. Your most valuable data resides in your operational systems. It’s what’s running your organisation. It’s great that you can pull data from social media, and other external data, but if you can’t get at that transactional data that is running the business, what are you enriching it with? It’s great you can see who loves a good cat meme, but if it isn’t linked to the data that is driving the business where is the real dollar value? So for true insight, you need all the data from all your sources. As an example, a global auto manufacturer has taken this real-time integration approach for 4500 applications, from SAP, DB2, Oracle, and streaming it into its Hortonworks data lake. Using Replicate’s CDC process  they maintain true real-time analytics with less overhead. So the ability to deliver this is where Attunity help organisations today.
• #14: So how do we do it? How do we help you analyse, everything, anywhere in real time? So as an example, what is the most efficient way of getting data out of an Oracle system say? It is off platform, meaning it is stand-alone with nothing installed on the source. You need to read the change logs in memory and push them into the target. In other words turn the database into a stream. This is critical for building new analytics. If you’re living in a batch world with end of day processing and you want to see a bank balance say at midnight when it’s quiet, you take an ETL snap shot and all you’re seeing is a point in time, you don’t get to see the full transactional flow as it accrues in real time out of operational systems. So by turning the database into a stream your analytics team can build out a fuller picture for risk, anti-money laundering, credit card fraud etc, and then build micro services out of Kafka, as an example, to deliver changes to the business. And you still have your historical data in Hadoop, if someone has a question. This is what Attunity Replicate does. It provides high speed data replication – for batch and full loads whilst at the same time running CDC or change data capture. It supports the widest end points on the market, and optimizes the data transfer using parallel loading and in-memory streaming. The end-to-end process is completely automated with no manual coding and there are no agents needed on your source systems - meaning lower IT costs, less maintenance and a faster return on investment. And you can access the information through its web based user interface to configure, control and monitor your replication tasks, whether on-premise, across a WAN, in the cloud and back again, if need be. It provides an immutable store of all the transactional history, so you can go back and replay the changes on your analytical platform.
• #15: So its highlights are that it is EASY. It’s written for DBA’s, not developers, so no scripting is involved. There’s no footprint on your production systems, so nothing to maintain - imagine having to maintain 1000’s of agents on all your sources? How much time and cost would that involve? It supports over 45 different end points, including SAP, Kafka, all the versions of Hadoop and it’s fast, secure using AES 256 encryption and its scalable, up to thousands of applications and sources.
• #16: From legacy systems to the major data warehouses, the cloud and more
• #17: But let’s look more deeply into SAP. It’s not your typical database system is it. It brings more challenges than most with its pooled and clustered tables, its stored blobs and clobs that have no meaning at the database layer. It only means anything if you talk at the ABAP level of the SAP structure to understand what the data is. So accessing and unpacking that data is critical. Replicate does a query based read initially then takes the data from the transactions logs to stream changes in real time. The Flexibility is the choice you have. You can merge that transactional data with all your other sources to find new insights in the cloud, Hadoop and other external targets. And you don’t have to be dependent on SAP modules. The data in BW may already have gone through the ETL process, so you might want data straight from the operational system, your CRM say, and stream that into a Kafka platform or directly into Hadoop. Then you can build out your analytics. You can then move it all back into HANA if necessary. That said, we do also work with SAP to enable users to move external data to HANA. And its because of this we are Gold Certified and you won’t face a hefty legal bill by using an external tool. So you’re not locked into the SAP world and dependent on BW only. It’s because Attunity is seen as being technology agnostic, that they work with all the major software providers, that Gartner lists them as the leading independent vendor for data integration. Attunity decodes SAP by using a subset of its SAP certified add-on product called Gold Client, a tool that has been part of the SAP ecosphere for 20 years. It creates QA and test / dev environments so that you can replicate between SAP instances and its this tool that gives us the ability to demystify the data. And its EASY. The whole process is automated, you don’t even need to know how to code in ABAP. Imagine that, take the whole mundane ETL process, that’s fraught with fat fingers and human error, that takes time and eats into your budget, and replace it with an end-to-end solution that is up and working in as little as 2 hours! That development budget can be put to better use finding insights in the analytical layer. And you’d have happier staff. And maybe a CFO Attunity can do this because it’s all we do. Attunity focuses only on making data available – whatever the system - and after 20 years they’ve become pretty good at it. Its something I tell my kids to do. Focus and Practice. If only I had when I was younger. I could have been a contender
• #18: So Replicate for SAP moves application data in bulk or real-time for Big Data Analytics. All your Documents, transactions and business data. All core and industry-specific SAP modules. The topics, your table structures, the schema definition, all automated and predefined so you can .. Integrate with all major targets - Databases, data warehouse and Hadoop, On premises or in Cloud and move external data into SAP HANA As well as streaming services like Apache Kafka, and others like Event Hubs or MapR Streams Why is this important? Because Attunity’s CDC engine is a natural fit. If you plan on building applications and business processes out of Kafka and using it as the core ingest system for multiple business units, Attunity streams the data directly into it, all automated, so that it is just another end point. It just works
• #19: And these are the use case we are seeing globally. Building data lakes within Hadoop and using streaming tools like Kafka. Building analytics platform in the cloud and setting up operational data stores outside of SAP to merge data from other ERP sources so that corporate financial reporting and BI teams have access to all ERP data in one place. In near real-time. More value is generated by delivering real-time insights and analytics in combining various data sets with the financials, and new insights can be discovered by bringing additional data sets to the data lake.
• #20: You can control it all via Replicate’s web based interface. It’s Point and click, that alerts you to issues, and makes it easy to configure and manage your replication tasks. And if, like the car builder I mentioned earlier, you have hundreds of tasks across multiple data centres running at the same time ..
• #21: … Then Enterprise Manager module that sits on top of it lets you centralize design, management, & control all from one pane of glass. It requires only network connections to the Replicate servers to automatically discover the environment, and does not require any additional agents to be installed on the database server.
• #22: Traditionally the Replicate product looks at the database level, but with SAP you need to talk directly to the ABAP level. You want the business objects so you can see at the SAP layer what it is you want to replicate. Then we decode those pools and cluster tables to give you exactly what you want in the target. So the business object level is being met, but in the target you get valuable information – product ID, customer, discount, price etc – rather than the structure that at the database level is just a blob that has little meaning if you just transferred it straight to the target.
• #23: Not all CDC engines are the same. Some systems use flags and triggers and need agents on the source, all of which puts a lot of pressure on those production systems, which can impact them negatively and cause instability. Attunity CDC has no agents to install, it is log based only and the most optimized way to unlock your production systems and make that data available to the business.
• #24: So the enablers I have to stress is: Attunity is SAP certified. You are not breaking any of your SAP license agreements. Attunity is seen as a good partner of SAP as their products help to move data around and into the SAP platforms. The boffins at SAP are not daft, they know they can’t turn their back on things like Hadoop and Kafka. They need to show their value with Hana and real time engines whilst understanding the benefits of the Hadoop platforms around that. So we help the flow of moving data to where it needs to be AS WELL AS moving that data to the SAP systems.
• #25: So here is the slightly altered architecture when it comes to SAP. Again there are no agents at the database level, but we do need the gold client subset add on within SAP to talk to the ABAP engine. So that we can understand the data. The Replicate server makes API calls to the gold client add-on to connect to the SAP system as well as at the database layer. Replicate reduces the impact on the source and makes RFC calls if the data can be found at the database layer. From then on, Replicate works as normal so that you get meaningful data on the target side. The data has the structure around it so you can build analytics quicker and efficiently, rather than having to go back and unlock what all that data means.
• #26: Replicate for SAP supports ECC 6, but it can also support older versions. Anything before 4.6C and we have a challenge on our hands but hopefully we won’t be seeing too many of those cases.
• #27: Because Gold Client has been playing in the SAP space for a long time, Attunity has over 150 customers using the product for moving data within the SAP environment SO we do have insight into specific customisations for the core modules within SAP. We don’t want to take data from BW because the data has probably already gone through its clean up process, and that would be like taking data out of the data warehouse. We want to move the operational data and merge that with others on to newer external platforms for even more insight
• #28: So the KEY Values then for opening flexible access to real time SAP data are: Better decisions support with real-time BI analysis of SAP data Agility and flexibility to use SAP data in new and valuable ways Lower IT costs and reduce IT\Dev dependency for access to SAP data Faster time to value by realizing new analytics value for SAP Data
• #29: So why Hortonworks in particular? This is an older slide as they’ve added cloud in the hexagon, but it’s the HDP and HDF layer that is so important – the actionable insights – both historical and perishable, that makes the Hortonworks and Attunity partnership so important. It’s how Attunity streams data in real time so you can get more than just historical insight, you get the perishable insight as well. If you just want to know what you’ve sold this year compared to last year, great, pop it into HDP. That’s something you can do with BW, SQL Server, Teradata etc. But what if you have a piece of equipment that is found in all your stores, depots, warehouses, perhaps globally, and a fault has been detected. You can pick that up in real time, and you now know you have a unique code or part number that you can do predictive analytics across that so you can alert the whole business to the problem. You can flag it immediately so that all parts of your business knows this perishable insight and can do something about it before it becomes a huge issue. So with HDF you are driving the change to business using your most valuable data.
• #30: So Attunity is an enabling platform to bring all your critical data to the newer platforms available today. Attunity Replicate supports all versions of Hadoop, both as a source and a target, and works with all major software vendors
• #31: And before I move on, I just wanted to mention a recently released product that works specifically with Hortonworks 2.6 and utilises its ACID Merge operation. Attunity Compose, the data warehouse automation tool I mentioned earlier, is now available for HIVE. So for the first time you can easily set up your historical data store as well as an Operational data store with no need to code. This means that you’ve got the historical data including various views, but now you have an Operational store that you can run real time insight from without having to go back to the source. It really is the fastest way to build an analytics ready data lake.
• #32: So to conclude, Attunity is trusted by almost half of the Fortune 100 companies, many of whom have been customers for years ….
• #33: And is trusted by all the major software vendors. Microsoft, Oracle and IBM OEM Attunity technology to help theirs work. SQL Server 2012 onwards, for instance, uses Attunity technology. And Attunity has close partnerships with data warehouse vendors, like Teradata & HP as well as all the Hadoop solutions. So we may be small in comparison, but we carry some clout.
• #34: So that’s it, thanks again for your time. If you’d like to know more come swing by the stand, or better still let’s go for a beer, god knows I need one. Oh and I am available for children’s parties.