![[DevDay 2016] OpenStack and approaches for new users - Speaker: Chi Le – Head...](https://cdn.slidesharecdn.com/ss_thumbnails/slideopenstack-devday2016-160414042927-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Spectrum Scale - Cognitive (20)
Ad
Recently uploaded (20)
Spectrum Scale - Cognitive
- 1. #ibmedge© 2016 IBM Corporation Leveraging Watson Cognitive Computing with Object Storage to Derive Insights and Relationships Sandeep Patil, STSM, IBM Spectrum Scale Smita Raut , Object Development, IBM Spectrum Scale Acknowledgement: Bill Owen, Pushkar Thorat, Nisarga Lolage
- 2. #ibmedge Introduction to SWIFT Object Store • Object storage is highly available, distributed, eventually consistent storage. • Data is stored as individual objects with unique identifier • Flat addressing scheme that allows for greater scalability • Has simpler data management and access • REST-based data access • Simple atomic operations: – PUT, POST, GET, DELETE • Usually software based that runs on commodity hardware • Capable of scaling to 100s of petabytes • Uses replication and/or erasure coding for availability instead of RAID • Access over RESTful API over HTTP, which is a great fit for cloud and mobile applications • Amazon S3, Swift, CDMI API 1
- 3. #ibmedge Spectrum Scale –The Complete Data Management Solution 2 Spectrum Scale Object (based on SWIFT)
- 4. #ibmedge Spectrum Scale Object Store – Additional Features • Unified file and object support with Hadoop connectors • Support for Encryption • Support for Compression • Only Object Store with Tape support for Backup • Object store with integrated transparent cloud tiering Support • Multi Region support • AD/LDAP support for authentication • ILM support for Object • Movement of Object across storage tiers based on access heat • Spectrum Scale Object with IBM DeepFlash becomes object store over all flash array for newer faster workloads. • Spectrum Scale Object with WAN caching support (AFM) 3
- 5. #ibmedge Introduction to Object Store Metadata • Data is stored as individual objects with unique identifier • Typically, Objects consist of an object identifier (OID), data and metadata • Object data is unstructured – images, text, audio, video • Metadata consists on system metadata and user defined custom metadata that can be extensive 4 System Metadata Filename: taj1234.jpg Created: 01 Aug 2016 Last Modified: 03 Aug 2016 Custom Metadata Subject: Taj Mahal Place taken: India Category: Travel Allow Sharing: yes Data Custom Metadata Data + Metadata = Object System Metadata Object type = image
- 6. #ibmedge Object Metadata – Usage One can assign Object metadata such as • Tags indicating the contents of the object and type of application the object is associated with. • The level of data protection / ACLs, Replication / Deletion controlling of object, Movement of object to a different tier of storage/geography; • The possibilities are limitless. Indexing/Searching • Metadata tags are used to categorize data • Example: Object repository for Sports – e.g. sport, indoor sport, chess • Objects are then searched based on category – e.g. • Search All articles related to outdoor sports • Search Images of all indoor sports • Search videos related to Chess 5 Sports Indoor Outdoor Chess Soccer Object Metadata for Categorizing
- 7. #ibmedge Object Metadata – Usage (continued…) Smart Tiering • Objects are placed in different tiers of storage pools based on metadata tags • Allow objects of different categories to be placed in different tiers based on needs – Example: Its time for the soccer world cup where objects related to soccer will be potentially highly accessed. – Place all the soccer related objects on faster tier. • Allow independent tiering of objects within the same category – Sports Analysts wants to run analytics over only Indoor game. This means one needs to run an haddop job on this data. – Within “Sport” Category tier only objects tagged as “Indoor” to faster storage pool for analytics. 6 Tier-0 (SSD) Tier 1 (HDD) Tier 2 (Tape) Trending Objects like Soccer world cup event Photos, Videos Non Trending event data
- 8. #ibmedge Now we know Object Metadata is very valuable and it’s usages are limitless…. But then, Where is the Problem? 7
- 9. #ibmedge Meaningful Tagging of Objects with Metadata ? For Leveraging the power of User Defined Metadata associated with object, the object has to be appropriately tagged, else it is of less use. Following are Inhibitors of meaningful tagging of object metadata • Typical metadata generation processes are • Device-based (e.g.: Camera tagging basic info to pics) – Most of the times these attributes are primitive and low level in nature and provides raw data about that object. – Since the tags are Limited or specific they are of less or constrained value for analytics • Manual – given by user or applications – User / applications defined metadata is sometimes looked as unnecessary overhead by the end user at time of object generation and do not tag the data. – User might add unnecessary or misleading metadata attributes, which are not adding value for further processing. • There could be many dimensions of a object, and not all may be added when the object is generated. This too constraints its value. e.g. • Image may have many faces, but user might tag only few. • A song might be fusion of two genres but in metadata only one is captured. Need for Objects to be accurately auto tagged by Object Stores ! 8
- 10. #ibmedge First of a Kind We need a provision to Cognitively Auto-tag Heterogeneous Unstructured data in form of Object to Leverage its benefits…. 9 Solution : Integration of Cognitive Computing Services with Object Storage for auto-tagging of unstructured data in the form of objects. Object Storage
- 11. #ibmedge What is IBM Watson Cognitive Computing Services • Cognitive services are based on a technology that encompasses machine learning, reasoning, natural language processing, speech and vision and more • IBM Watson Developer Cloud enables cognitive computing features in your app using IBM Watson’s Language, Vision, Speech and Data APIs. Watson Services • Alchemy Language – Text Analysis to give Sentiments of the Document • Language Translation – Translate and publish content in multiple languages • Tone Analyzer – Discover, understand, and revise the language tones in text • Visual Recognition – Understand the contents of images. • Personality insights – Uncover a deeper understanding of people's personality • Retrieve and Rank – Enhance information retrieval with machine learning • Natural language Classifier – Interpret and classify natural language with confidence • And Many More … 10
- 12. #ibmedge Example of IBM Watson Cognitive Service • Visual Recognition Service • Allows users to understand the contents of an image or video frame. • Provides answer to "What is in this image?" • Result is in scores for relevant classifiers representing things such as objects, events and settings. – e.g. dog (relevance 0.7), mountain (relevance 0.5) 11 Input Image Watson Results
- 13. #ibmedge Example of IBM Watson Cognitive Service (cont.) Speech to Text The Speech to Text service converts the human voice/audio into the written word. 12 Tone Analyzer This service uses linguistic analysis to detect and interpret emotions, social tendencies, and language style cues found in text. Input Audio / Voice Watson Results Speech to Text Tone Analyzer
- 14. #ibmedge How Does Cognitive Insights solve the Problem of lack of meaningful Object metadata tags. • Cognitive Services post deep analysis gives insights of unstructured data (images, audio, video, text, etc.) • These insights directly relate to the content of the unstructured data and can be used for: • Further Analytics of the data • Categorization of the data and subsequent use of the categorization in different use cases. • Index & Search of data, etc. • When unstructured data is stored in form of objects, these cognitive insights can be defined and stored as user defined metadata (tags) for the objects: • Helps address the problem of meaningful tagging of objects. • Opens a realm of newer possibility and opportunity for analytics. 13 Object Storage
- 15. #ibmedge Deriving Object Tags Using Cognitive Services Cognitive services are asynchronously run by the Object Store to auto-tag the objects. Example: • Visual recognition service is used for images to tag and categorize the images • Alchemy API's entities and concept tagging for text objects like blogs and text feeds • Speech-to-text conversion in conjunction with Alchemy API for tagging audio/video objects • Tone Analyzer can be used over audio files to tag likewise them Cognitive services categorize the objects into specific group or categories. • Example: – Animal, canine, tiger – Sport, outdoor sport, football Based on the categories, object relations can be derived, such as • All articles on indoor sports • All images of animals • All nature songs 14
- 16. #ibmedge High Level Flow 15 IBM Spectrum Scale Object Storage Clicks Photo From Phone Service to feed Object toWatson { “WatsonTags” : “Eiffel“: 94.78%, “Tower“ : 85.81%, “Tour“: 66.82% }
- 17. #ibmedge What Makes it Possible: OpenStack Swift Middleware Framework • OpenStack Swift is based on WSGI specifications and Middleware is a WSGI feature which extends functionality of any WSGI application. • Middlewares are heavily used in swift, for purposes such as logging, tempurl, tempauth, quotas etc. • If there is a middleware in an application pipeline, every request and response is passed through the middleware when a request is served. • Can I write custom middleware for Spectrum Scale Object ? • Yes (needs to be reviewed by development before deployment) 16
- 18. #ibmedge How it Works • Spectrum Scale Unfied File and Object Data Ingest Request Response Spectrum Scale Object (OpenStack Swift) Cognitive Middleware Send Objects Receive Cognitive Tags and updates Objects Associated CognitiveTags to objects as user defined metadata for applications to leverage.
- 19. #ibmedge How OpenStack Swift is integrated with Spectrum Scale • At very high level Spectrum Scale cluster contains nodes which share common filesystem namespace which is cross mounted on all of its nodes. • OpenStack Swift cluster consist of multiple type of processes, of which object, container and account servers are backend and Proxy server acts as interface for the cluster. • There are few designated nodes in Spectrum Scale cluster which are known as ‘Protocol Nodes’ which hosts protocol stack. OpenStack Swift is installed on these protocol nodes. • All the OpenStack Swift processes are installed on every protocol node of the cluster. • Depending on the request type i.e. Object or Container or Account, Proxy server chooses the responsible backend server which will serve depending on distributed circular hash, called as Ring. • So a proxy server can contact any backend server running on other protocol node to fulfill the request. 18
- 20. #ibmedge Approach chosen Developed Custom Watson Cognitive Middleware for Spectrum Scale Object • User will upload the object header – X-Visual_Insights_Enabled to enable it for cognitive analysis. • Watson Cognitive middleware present in proxy server pipeline will intercept the request in return path, if successful. • Middleware appends the object path and object timestamp in local queue, where the queue contains a list of object that needs to be tagged by cognitive computing. Developed a Watson Background Service for Spectrum Scale • Is running on every protocol node, and monitors the queue configuration provided. • When a STOMP (Simple Text Oriented Messaging Protocol) message is added in queue by middleware. Watson service checks the timestamp of the message with the file’s. If the queue timestamp is older than the file, it ignores the request. • If the message timestamp matches with the file, it processes the object with Watson Cognitive service and updates the metadata thru file interface. 19
- 21. #ibmedge 20 IBM Spectrum Scale Proxy Server Object Server Proxy Server Object Server Node 1 Node 2 MessageQ MessageQ Watson integration Service Watson integration Service Swift Client 1 3 4 5 6 8 9 1 0 2 1 1 7 Image Upload Request Delegates Req. to Object Server Object Write Message to the queueImage Upload to Watson Service Watson Generated Tags of Image Update Metadata Of the Object with WatsonTags How Does it Work ?
- 22. #ibmedge Use Case1: Smart Object Tiering based on Cognitive Tags. • Objects are tagged by cognitive services based on its content. • Tags can be accessed as xattr by Spectrum Scale placement and migration polices. • Admins can write placement /movement rules for object based on cognitive tags associated with objects as user defined metadata. • Example: A sports media portal host sports images and videos for its end users. Assuming Soccer world cup is coming in a month , its end users will access more soccer related content which the portal would like to serve with better response time. With cognitive object tagging on spectrum scale, one can move all images tagged as soccer by the cognitive service to fast tier for better response time for end users. 21 Spectrum Scale Unified File and Object Auto Tagging of Objects as user defined metadata Tiering of Objects based on Business Rules leveraging conginitive Tags Silver pool ( SATA) Gold pool (SAS)
- 23. #ibmedge • These tags helps categorize data based on type and genre. • One can then run in-place analytics leveraging spectrum scale Hadoop connectors on the same data to derive more insights. • Example: Run analytic over all objects marked as “anger emotion chat” to derive the time-line and product being discussed and present a word chart showing which product is generating anger emotion and what were the timelines when it happened. Use case 2: Chat Center Interaction Analysis22 Results returned In place over the good /bad chats Faster Tier Auto Tagging of Objects (Chat-logs With emtions Objects to be analyzed are identified via the auto tagging and moved to faster storage Spectrum Scale Unified File and Object Spark or Hadoop MapReduce In-Place Analytics Analytics With Unified File and Object Access Slower Tier • On-going chat Center Analysis is key to any Business where the business is required to know: • % of good chats • % of bad chats • Analysis over the chat (using Hadoop) – Demographic relation to good/bad chat – Specific Product relation to good/bad chat – etc. • Chat interaction files stored as Objects are tagged by cognitive services as good & bad interactions or tags based on emotions identified by cognitive services.
- 24. #ibmedge How to Use this Concept: Its Easy and Simple - Do It Yourself • We have provided a sample and open sourced the middleware and service code that will allow you to auto tag objects in form of images ( hosted over IBM Spectrum Scale Object). • Based on your business needs and types of objects you can re-use and develop the required middleware • Spectrum Scale Object which is based on OpenStack Swift supports customer middleware like these to be used, post review with the development. • The code and instruction are available on GitHub under Apache 3.0 license. • https://github.com/SpectrumScale/watson-spectrum-scale-object-integration 23
- 25. #ibmedge Project location on GITHUB 24
- 26. #ibmedge Demo 25
- 27. #ibmedge Spectrum Scale User Group • The Spectrum Scale User Group is free to join and open to all using, interested in using or integrating Spectrum Scale. • Join the User Group activities to meet your peers and get access to experts from partners and IBM. • Next meetings: - APAC: October 14, Melbourne - Global at SC16 : November 13 1pm to 5pm, Salt Lake City • Web page: http://www.spectrumscale.org/ • Presentations: http://www.spectrumscale.org/presentations/ • Mailing list: http://www.spectrumscale.org/join/ • Contact: http://www.spectrumscale.org/committee/ • Meet Bob Oesterlin (US Co-Principal) at Edge2016: [email protected]
- 28. #ibmedge Session : How to apply Flash benefits to big data analytics and unstructured data NDA & Customers ONLY • Who: IBM Elastic Storage Server Offering Management • Alex Chen • When: Thursday, September 22, 2016 • 1:15pm to 2:15pm • Where: Grand Garden Arena, Lower Level, MGM, Studio 10 • Contact(if any questions) • • [email protected], [email protected] 27
- 29. #ibmedge Spectrum Scale Trial VM • Download the IBM Spectrum Scale Trial VM from : • http://www-03.ibm.com/systems/storage/spectrum/scale/trial.html 28
- 30. #ibmedge Notices and Disclaimers 29 Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.” Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
- 31. #ibmedge Notices and Disclaimers Con’t. 30 Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
- 32. © 2016 IBM Corporation #ibmedge Thank You
- 33. #ibmedge How to Deploy and Use the Sample Middleware with IBM Spectrum Scale Prerequisite a) Get IBM Bluemix Visual Recognition account b) Install following packages on protocol nodes: I. Stompest https://pypi.python.org/pypi/stompest/ II. Apache Active MQ III. Watson Developer Cloud SDK https://pypi.python.org/pypi/watson-developer-cloud c) Ensure connectivity to server - gateway-a.watsonplatform.net from protocol nodes Deployment a) Install Watson middleware dist/watsonintegration-0.1-1.noarch.rpm on all protocol nodes (from GitHub) b) Update proxy-server.conf to include the middleware, and restart proxy servers. c) Start watsonintegration service on all protocol nodes d) Create a SwiftOnFile policy and create a container with it. e) Upload an image - $ swift upload -H "X-Visual_Insights_Enable:true" <container_name> <object_name> f) Check Watson metadata tags: $ swift stat <container_name> <object_name> 32